记录一次kubelet错误日志排查

线上K8s集群的一个节点总报告kubelet不健康,使用journalctl 查看日志发现大量

  • Scope libcontainer-nnnnn has no PIDs. Refusing
  • Orphaned pod found - but volume paths are still present on disk

在此记录下问题的解决方案

问题1. Scope libcontainer-nnnnn has no PIDs. Refusing

问题描述

使用journalctl查询 /var/log/messages处日志,有大量

Scope libcontainer-nnnnn has no PIDs. Refusing

问题原因

问题解法

参考出处

问题2. Orphaned pod found - but volume paths are still present on disk

问题描述

使用journalctl查询 /var/log/messages处日志,有大量报错

[kubelet_volumes.go:128] Orphaned pod "86d60ee9-9fae-11e8-8cfc-525400290b20" found, but volume paths are still present on disk. : There were a total of 100 errors similar to this.  Turn up verbosity to see them

问题原因

集群Pod停止删除了,数据文件未自动清理

问题解法

删除/var/lib/kubelet/pods对应日志ID的残留文件

参考出处