Pods Stuck in ContainerCreating with Error: "no space left on device"

Problem

Pods are stuck in container crating state with error "No space left on device"

Environment

• Platform9 Managed Kubernetes - All versions
• Docker

Cause

• By default, the Docker does not remove unused objects such as containers, images, volumes, and networks unless you explicitly tell it to do so.
• The error can be observed due to the filesystem mounted on directory /var/lib/docker do not have free disk space. Also, this issue could be observed if the inodes are exhausted on the filesystem.
• The below command will show disk space usage for the host.

xxxxxxxxxx

# df -h

For checking the inode usage for each filesystem run the below command.

xxxxxxxxxx

# df --inodes

• The below command will show the disk usage by docker objects along with the amount of space that can be reclaimed.

Example

xxxxxxxxxx

# docker system df -h

TYPE TOTAL ACTIVE SIZE RECLAIMABLE

Images 209 31 61.04GB 52.82GB (86%)

Containers 76 68 13.15MB 118B (0%)

Local Volumes 18 18 23.17MB 0B (0%)

Build Cache 0 0 0B 0B

Resolution

1. Run the below command to delete all the unused docker objects like images, stopped containers, all networks not used by at least one container, and build cache.

xxxxxxxxxx

# docker system prune -a