How To Prevent a Run Away Pod or Container From Using All Resources on the Node

Problem

Looking to prevent a run-away pod or container from using all resources on the system. For instance, one pod uses up all of the resources on one machine causing it to timeout and fail, then jumping to the next node and causing the same issue until the entire cluster goes into a pending state.

Environment

• Platform9 Managed Kubernetes - All Versions

Procedure

1. Resource requests and limits are ways in Kubernetes by which you can restrict CPU and Memory usage of your containers/pods as per your requirement.
2. Once implemented within your application specifications, the node where the Pod is running if has enough resource available, it's possible (and allowed) for the container to use more resources than it's request specified value. However, a container is not allowed to use more than its resource limitspecified value.
3. From a Scheduling perspective, when one specifies the resource request for Containers in a Pod spec, the scheduler uses this information to decide which node to place the Pod on. However, when one specifies a resource limit for a Container, kubelet will enforce those limits so that the running container is not allowed to use more of that resource than the limit one sets.

Additional Information

1. You can read more about this in detail here Managing Resources for Containers.
2. For further illustrations with examples for both resource types details, see Assign Memory Resources and Assign CPU Resources.
3. Consider reserving some CPU and memory resources for system processes by referring to How-To Implement CPU and Memory System Reservation for Kubernetes Nodes