线上K8s集群的一个节点总报告kubelet不健康,使用journalctl 查看日志发现大量
Scope libcontainer-nnnnn has no PIDs. Refusing
Orphaned pod found - but volume paths are still present on disk
在此记录下问题的解决方案
问题1. Scope libcontainer-nnnnn has no PIDs. Refusing
问题描述
使用journalctl查询 /var/log/messages
处日志,有大量
Scope libcontainer-nnnnn has no PIDs. Refusing
问题原因
问题解法
参考出处
问题2. Orphaned pod found - but volume paths are still present on disk
问题描述
使用journalctl查询 /var/log/messages
处日志,有大量报错
[kubelet_volumes.go:128] Orphaned pod "86d60ee9-9fae-11e8-8cfc-525400290b20" found, but volume paths are still present on disk. : There were a total of 100 errors similar to this. Turn up verbosity to see them
问题原因
集群Pod停止删除了,数据文件未自动清理
问题解法
删除/var/lib/kubelet/pods
对应日志ID的残留文件