Hostagent Fails to Communicate With the Management Plane due to Wrong Credentials

Problem

  • Hostagent failing to communicate with the Platform9 Management Plane and hostagent service is logging authentication errors:

2024-10-30 02:17:19,768 - slave.py ERROR - Connection error. Retrying in 10 seconds.
Traceback (most recent call last):
File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/slave.py", line 127, in reconnect_loop
start(config, log, app_db, agent_app_db, app_cache,
File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 770, in start
dual_channel_io_loop(log,
File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbcommon/amqp.py", line 245, in dual_channel_io_loop
conn.ioloop.start()
File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 461, in start
self._poller.start()
File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 721, in start
self.poll()
File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 1114, in poll

Environment

  • Platform9 Managed OpenStack - v5.0 and Higher

Cause

  • Hostagent uses AMQP credentials to authenticate with the Management Plane. Misconfigured credentials can lead to authentication issues; making host disconnected from the Management Plane.

Resolution

Workaround steps:

  1. Retrieve username & password from /etc/pf9/hostagent.conf file present over any healthy host from the same region.

  2. Over affected host, modify the credentials for amqp section in /etc/pf9/hostagent.conf file.

[amqp]
host=example-region.platform9.net
username=<username>
password=<password>
  1. Restart pf9-hostagent service over the affected host.

# systemctl restart pf9-hostagent

Additional Information

  • During credential rotation activity (if performed earlier), the OFFLINE hosts would not retrieve updated credentials to authenticate hostagent with AMQP running on Management Plane.

  • The legacy bbslave user displayed in the username has been replaced by hostagent user now.