How To Enable fail-on-swap=false Option as a Parameter Value in Kubelet on a Worker Node

Problem

How to Enable fail-on-swap=false option as a parameter value in Kubelet one a worker node.

Environment

• Platform9 Managed Kubernetes - All Versions

Procedure

Steps to enable swap on a worker node.

1. Start kubectl proxy in the background.

xxxxxxxxxx

root@test-master1:~# kubectl proxy --port=8001 &

[1] 22409

root@test-master1:~# Starting to serve on 127.0.0.1:8001

2. Current Configmap associated with the worker node.

xxxxxxxxxx

root@test-master1:~# kubectl get node 172.20.4.210 -o yaml

...

spec:

configSource:

  configMap:

    kubeletConfigKey: kubelet

name: worker-default-kubelet-config

namespace: kube-system

...

3. Download and unpackage the worker node configuration.

root@test-master1:~# curl -sSL "http://localhost:8001/api/v1/nodes/172.20.4.210/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet

4. Edit the config file named kubelet and change the failSwapOn parameter value to false and saved the file.

xxxxxxxxxx

root@test-master1:~# cat kubelet | grep failSwapOn

  "failSwapOn": false,

5. Push the config file by creating a new ConfigMap. In this case, the configmap "new-worker-config-swapon" is created.

xxxxxxxxxx

root@test-master1:~# kubectl create configmap new-worker-config-swapon -n kube-system --from-file=/root/kubelet

configmap/new-worker-config-swapon created

xxxxxxxxxx

root@test-master1:~# kubectl get configmap -n kube-system | grep swap

new-worker-config-swapon            1      18s

6. Edit the worker node reference to point to the new ConfigMap.

xxxxxxxxxx

root@test-master1:~# kubectl edit node 172.20.4.210

node/172.20.4.210 edited

xxxxxxxxxx

root@test-master1:~# kubectl get node 172.20.4.210 -o yaml

...

spec:

configSource:

  configMap:

    kubeletConfigKey: kubelet

name: new-worker-config-swapon

namespace: kube-system

...

7. Post this you will observe that the pf9-kubelet service on the node will be restarted and a new checkpoint will be created in this case ca8caeaa-da7d-4d95-8fbb-6adac5795958 (Kubelet always refers to the newest created checkpoint when setting dynamic configuration). Note that initially when the service restarts you would see kubelet setting the swap value to "true" from its default configuration in kubelet logs but it will then load the newer configmap and set the value to false accordingly.

ubuntu@test-worker1:/var/log/pf9/kubelet$ less kubelet.INFO

I0111 21:56:30.148885  22192 fsstore.go:116] kubelet config controller: loading Kubelet configuration checkpoint for source /api/v1/namespaces/kube-system/configmaps/new-worker-config-swapon,

UID: ca8caeaa-da7d-4d95-8fbb-6adac5795958, ResourceVersion: 9898516

root@test-worker1:/var/opt/pf9/kube/kubelet-config/dynamic-config/store/checkpoints# ls -ltr

total 8

drwxr-xr-x 3 pf9 pf9group 4096 Dec 1 23:09 1ed02d48-73e5-448b-8b1e-163a73dac8b0

drwxr-xr-x 3 root root 4096 Jan 11 21:56 ca8caeaa-da7d-4d95-8fbb-6adac5795958

root@juniperlimit-worker1:/var/opt/pf9/kube/kubelet-config/dynamic-config/store/checkpoints/ca8caeaa-da7d-4d95-8fbb-6adac5795958/9898516# cat kubelet | grep -i swap

"failSwapOn": false,

8. In terms of memory allocation, the node's memory resource request/limit allocation will be honored but since the swap is not supported natively the way in which kubelet does the calculations might not be accurate leading to pod evictions due to memory.

Note: If the same change has to be made across all nodes, you can read more about it here.

Additional Information

• Enabling swap on a node in Kubernetes supported starting K8s v1.22. Reference: https://kubernetes.io/blog/2021/08/04/kubernetes-1-22-release-announcement/#node-system-swap-support
• From Platform9's perspective, a feature request PMK8-I-148 has been created to look into ways and options in which we can expose an option to enable the fail-on-swap=false as a feature flag in the cluster creation wizard.