Unable to Start pf9-kube Service Due to Package Hold

Problem

The pf9-kube service does not start on one of the nodes due to hold present on the docker-ce package with the following errors observed in the nodelet and kube logs.

pf9@host:/var/log/pf9$ less nodelet.log
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
docker-ce*
The following held packages will be changed:
docker-ce
0 upgraded, 0 newly installed, 1 to remove and 1 not upgraded.

E: Held packages were changed and -y was used without --allow-change-held-packages.
" func=main.startKubeService file="/pf9/nodelet/nodeletd.go:210"time="XXXX-XX-XXTXX:XX:XX-XX:XX" level=warning msg="Trying to start pf9-kube 66. Failed task on the last run was: Configure Docker" func=main.writeExtensionData file="/pf9/nodelet/nodeletd.go:261"time="XXXX-XX-XXTXX:XX:XX-XX:XX" level=warning msg="pf9-kube restarted due to failure of status check: Check if start operation in progress" func=main.writeExtensionData file="/pf9/nodelet/nodeletd.go:269"
pf9@host:/var/log/pf9/kube$ less kube.log
The following packages have unmet dependencies:
docker-ce : Depends: docker-ce-cli but it is not going to be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.

Environment

  • Platform9 Managed Kubernetes - All Versions

  • Docker

Cause

  1. Hold is present on the packages and can be verified using the following command.

pf9@host:/var/log/pf9$ apt-mark showhold
docker-ce
docker-ce-cli

Resolution

  1. Remove the hold on the packages.

$ apt-mark unhold [PackageName]
  1. Once the hold is removed, the pf9-kube service starts successfully.

pf9@host:/var/log/pf9$ less nodelet.log
time="XXXX-XX-XXTXX:XX:XX-XX:XX" level=info msg="pf9-kube is already running..." func=main.handleKubeServiceStartState file="/pf9/nodelet/nodeletd.go:341"
time="XXXX-XX-XXTXX:XX:XX-XX:XX" level=info msg="Expected service state: &{ClusterRole:master ClusterID:UUID ServiceState:true}" func=main.handleExpectedServiceState file="/pf9/nodelet/nodeletd.go:172"
time="XXXX-XX-XXTXX:XX:XX-XX:XX" level=info msg="pf9-kube is already running..." func=main.handleKubeServiceStartState file="/pf9/nodelet/nodeletd.go:341"
time="XXXX-XX-XXTXX:XX:XX-XX:XX" level=info msg="Expected service state: &{ClusterRole:master ClusterID:UUID ServiceState:true}" func=main.handleExpectedServiceState file="/pf9/nodelet/nodeletd.go:172"