Container keeps running on node after Kubernetes pod deletion

Problem

Container keeps running on the node even after you deleted the pod from the Kubernetes cluster.
Kubelet logs show that an API call was made to delete the pod but the container associated with that pod keeps running on the node.
Message logs show it did try to kill the container using SIGKILL but failed to delete the container.

On node - Messages log
    
 
2022-10-05T18:42:49-07:00 daemon:info [1e] dockerd (408285): time="2022-10-05T18:42:49.951251921-07:00" level=info msg="Container d651b04e617c170115e84462b52eb42e8068aae565f0af40777fc304162d1dd0 failed to exit within 2 seconds of signal 15 - using the force"2022-10-05T18:42:59-07:00 daemon:info [1e] dockerd (408285): time="2022-10-05T18:42:59.963726137-07:00" level=info msg="Container d651b04e617c failed to exit within 10 seconds of kill - trying direct SIGKILL"
Copy

On node - Kubelet log
    
 
I1014 11:08:05.510428   45736 kubelet.go:1948] "SyncLoop DELETE" source="api" pods=[metallb-system/speaker-4pcxk]I1014 11:08:05.510569   45736 kuberuntime_container.go:661] "Killing container with a grace period override" pod="metallb-system/speaker-4pcxk" podUID=057070b8-f582-4605-87aa-798d10aca94c containerName="speaker"containerID="docker://7862332cf8f03cff83cc706031102835ed3a84f0086b38f1c526cb49fce04e10" gracePeriod=2I1014 11:08:05.517822   45736 kubelet.go:1942] "SyncLoop REMOVE" source="api" pods=[metallb-system/speaker-4pcxk]I1014 11:08:05.517897   45736 kubelet_pods.go:1285] "Killing unwanted pod" podName="speaker-4pcxk"I1014 11:08:05.517944   45736 kuberuntime_container.go:661] "Killing container with a grace period override" pod="metallb-system/speaker-4pcxk" podUID=057070b8-f582-4605-87aa-798d10aca94c containerName="speaker"
Copy

Environment

Platform9 Managed Kubernetes - v 5.4
Docker and linux kernel versions are 19.03.11 and 5.4.0-123-generic respectively.

Version output
    
xxxxxxxxxx
 
# docker versionClient: Docker Engine - Community Version:           19.03.11 API version:       1.40 Go version:        go1.13.10 Git commit:        ID Built:             Mon Jun  1 09:12:34 2020 OS/Arch:           linux/amd64 Experimental:      false Server: Docker Engine - Community Engine:  Version:          19.03.11  API version:      1.40 (minimum version 1.12)  Go version:       go1.13.10  Git commit:       ID  Built:            Mon Jun  1 09:11:07 2020  OS/Arch:          linux/amd64  Experimental:     false containerd:  Version:          1.4.12  GitCommit:        ID runc:  Version:          1.0.2  GitCommit:        v1.0.2-0-g52b36a2 docker-init:  Version:          0.18.0  GitCommit:        ID   #uname -aLinux 5.4.0-123-generic #138 SMP Sun Aug 7 21:49:28 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
Copy

Cause

This is an upstream docker bug on docker 19.03 version, which kept container running in the background.

Upstream bug
    
xxxxxxxxxx
 
https://github.com/moby/moby/issues/41579
Copy

Resolution

Manual deletion of the container can be done on the node.
Upgrade docker to possible bug fix version 20.10.

Last updated on

Was this page helpful?