Container keeps running on node after Kubernetes pod deletion

Problem

  • Container keeps running on the node even after you deleted the pod from the Kubernetes cluster.

  • Kubelet logs show that an API call was made to delete the pod but the container associated with that pod keeps running on the node.

  • Message logs show it did try to kill the container using SIGKILL but failed to delete the container.

2022-10-05T18:42:49-07:00 daemon:info [1e] dockerd (408285): time="2022-10-05T18:42:49.951251921-07:00" level=info msg="Container d651b04e617c170115e84462b52eb42e8068aae565f0af40777fc304162d1dd0 failed to exit within 2 seconds of signal 15 - using the force"
2022-10-05T18:42:59-07:00 daemon:info [1e] dockerd (408285): time="2022-10-05T18:42:59.963726137-07:00" level=info msg="Container d651b04e617c failed to exit within 10 seconds of kill - trying direct SIGKILL"
I1014 11:08:05.510428 45736 kubelet.go:1948] "SyncLoop DELETE" source="api" pods=[metallb-system/speaker-4pcxk]
I1014 11:08:05.510569 45736 kuberuntime_container.go:661] "Killing container with a grace period override" pod="metallb-system/speaker-4pcxk" podUID=057070b8-f582-4605-87aa-798d10aca94c containerName="speaker"
containerID="docker://7862332cf8f03cff83cc706031102835ed3a84f0086b38f1c526cb49fce04e10" gracePeriod=2
I1014 11:08:05.517822 45736 kubelet.go:1942] "SyncLoop REMOVE" source="api" pods=[metallb-system/speaker-4pcxk]
I1014 11:08:05.517897 45736 kubelet_pods.go:1285] "Killing unwanted pod" podName="speaker-4pcxk"
I1014 11:08:05.517944 45736 kuberuntime_container.go:661] "Killing container with a grace period override" pod="metallb-system/speaker-4pcxk" podUID=057070b8-f582-4605-87aa-798d10aca94c containerName="speaker"

Environment

  • Platform9 Managed Kubernetes - v 5.4

  • Docker and linux kernel versions are 19.03.11 and 5.4.0-123-generic respectively.

# docker version
Client: Docker Engine - Community
Version: 19.03.11
API version: 1.40
Go version: go1.13.10
Git commit: ID
Built: Mon Jun 1 09:12:34 2020
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.11
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: ID
Built: Mon Jun 1 09:11:07 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: ID
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2

Cause

This is an upstream docker bug on docker 19.03 version, which kept container running in the background.

https://github.com/moby/moby/issues/41579

Resolution

  • Manual deletion of the container can be done on the node.

  • Upgrade docker to possible bug fix version 20.10.


© 2025 Platform9. All Rights Reserved

On This Page
Container keeps running on node after Kubernetes pod deletionProblemEnvironmentCauseResolution