Host Reporting Offline in Clarity UI

Problem

The Clarity UI reports the host as "offline" even though it is up and running.

Environment

• Platform9 Managed OpenStack - All Versions
• Platform9 Managed Kubernetes - All Versions

Cause

One or more of the following can be the possible cause(s).

1. Disk Utilization is 100%

   1. The Hostagent daemon logs its events to /var/log/pf9/hostagent.log.

   2. In an event where the /var file system gets full, Hostagent will fail to log, thus failing to send heartbeats to the Management Plane

   3. Check the disk usage.

xxxxxxxxxx

df -h /var

2. Platform9 Hostagent and/or Comms Service may not be Running

   1. Check the status of the services.

xxxxxxxxxx

systemctl status pf9-hostagent.service pf9-comms.service

3. Host Unable to Communicate With Management Plane: "ETIMEDOUT"

4. Stalled vgs Execution

   1. The pf9-hostagent daemon collects certain system utilization information like vgs, which at times can stall if LVM filters are not set.
   2. Check whether the vgs process is running.

xxxxxxxxxx

ps -ef | grep vgs

5. System Load Average is High

   1. When all the CPUs are consumed, the pf9-hostagent fails to get a CPU cycle. As a result, pf9-hostagent is unable to send a heartbeat to the Management Plane.
   2. Check the system load average using the top or uptime command.
   3. Check the number of CPUs on the host/node using the lscpucommand.
   4. If the system load average is greater than the number of CPUs, identify the source of the load. However, if the load is genuine, increase the number of CPUs.

6. Platform9 Hostagent Daemon is Taking Too Long to Generate A Support Bundle

   1. When the command to generate a support bundle is fired, it pulls all the logs from /var/log/pf9.
   2. If the size of the directory is greater than 2GB, it takes time to copy and compress the logs.
   3. As pf9-hostagent is a single-threaded process, it will neither log events to /var/log/pf9/hostagent.log nor send heartbeats to the Management Plane, until the support bundle is generated.
   4. Check if the process is in a 'D' (defunct) state.

$ ps aux | grep hostagent 

pf9      23903  0.0  0.0 113132  1204 ?        Ss  Jan04  0:00 /bin/bash -c /opt/pf9/hostagent/bin/pf9-hostd >> /var/log/pf9/hostagent-daemon.log 2>&1 

pf9      23904  0.3  0.0 393408 67824 ?        D    Jan04 406:16 /opt/pf9/hostagent/bin/python /opt/pf9/hostagent/bin/pf9-hostd 

$

7. Hostagent Stuck While Executing an Extension

Resolution

Resolve based on the respective cause(s) stated above.