Pods Stuck in ContainerCreating with Error: "no space left on device"

Problem

Pods are stuck in container crating state with error "No space left on device"

Environment

• Platform9 Managed Kubernetes - All versions

• Docker

Cause

• By default, the Docker does not remove unused objects such as containers, images, volumes, and networks unless you explicitly tell it to do so.

• The error can be observed due to the filesystem mounted on directory /var/lib/docker do not have free disk space. Also, this issue could be observed if the inodes are exhausted on the filesystem.

• The below command will show disk space usage for the host.

For checking the inode usage for each filesystem run the below command.

• The below command will show the disk usage by docker objects along with the amount of space that can be reclaimed.

Example

Resolution

1. Run the below command to delete all the unused docker objects like images, stopped containers, all networks not used by at least one container, and build cache.