How To Rotate Container Logs on The Master and Worker Nodes

Problem

Container logs are utilizing all the disk space on the master and worker nodes.

Environment

• Platform9 Managed Kubernetes - All Versions
• Docker

Cause

The docker service is not configured to rotate the container logs.

Resolution

1. Back up the /etc/docker/daemon.json file on the affected node if present. Add the log rotation config on the node in the file /etc/docker/daemon.json file as shown below.

xxxxxxxxxx

$ less /etc/docker/daemon.json

{

   "log-driver": "json-file",

   "log-opts": {

     "max-size": "50m",   

     "max-file": "5"

   }

}

2. Stop the Platform9 Managed Kubernetes stack by executing commands in the below sequence.

xxxxxxxxxx

$ sudo systemctl stop pf9-hostagent pf9-nodeletd

xxxxxxxxxx

$ sudo /opt/pf9/nodelet/nodeletd phases stop

3. Restart the docker service.

xxxxxxxxxx

$ systemctl restart docker

4. Start the Platform9 Managed Kubernetes stack by executing the commands in the below sequence.

xxxxxxxxxx

$ sudo /opt/pf9/nodelet/nodeletd phases start

xxxxxxxxxx

$ sudo systemctl start pf9-hostagent pf9-nodeletd

5. Verify the changes took effect on the container by running below command.

xxxxxxxxxx

$ sudo docker ps | grep -i <conmtainer-name>

xxxxxxxxxx

$ sudo docker inspect <container-id>|  grep -A6 -i LogConfig

           "LogConfig": {

               "Type": "json-file",

               "Config": {

                   "max-file": "5",

                   "max-size": "50m"

               }

           },