Hostagent Fails to Communicate With the Management Plane due to Wrong Credentials

Problem

Hostagent failing to communicate with the Platform9 Management Plane and hostagent service is logging authentication errors:

Hostagent Logs
    
 
2024-10-30 02:17:19,768 - slave.py ERROR - Connection error. Retrying in 10 seconds.Traceback (most recent call last):  File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/slave.py", line 127, in reconnect_loop    start(config, log, app_db, agent_app_db, app_cache,  File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbslave/session.py", line 770, in start    dual_channel_io_loop(log,  File "/opt/pf9/hostagent/lib/python3.9/site-packages/bbcommon/amqp.py", line 245, in dual_channel_io_loop    conn.ioloop.start()  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 461, in start    self._poller.start()  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 721, in start    self.poll()  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 1114, in poll    self._dispatch_fd_events(fd_event_map)  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/select_connection.py", line 831, in _dispatch_fd_events    handler(fileno, events)  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/base_connection.py", line 410, in _handle_events    self._handle_read()  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/adapters/base_connection.py", line 464, in _handle_read    self._on_data_available(data)  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2041, in _on_data_available    self._process_frame(frame_value)  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2162, in _process_frame    if self._process_callbacks(frame_value):  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2140, in _process_callbacks    self.callbacks.process(frame_value.channel_number,  # Prefix  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 60, in wrapper    return function(*tuple(args), **kwargs)  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 92, in wrapper    return function(*args, **kwargs)  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 236, in process    callback(*args, **keywords)  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 1838, in _on_connection_close    self._on_terminate(self.closing[0], self.closing[1])  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 2119, in _on_terminate    self.callbacks.process(0,  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 60, in wrapper    return function(*tuple(args), **kwargs)  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 92, in wrapper    return function(*args, **kwargs)  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/callback.py", line 236, in process    callback(*args, **keywords)  File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 1856, in _on_connection_error    raise exceptions.AMQPConnectionError(error_message orpika.exceptions.AMQPConnectionError: (403, 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.')
Copy

Environment

Platform9 Managed OpenStack - v5.0 and Higher

Cause

Hostagent uses AMQP credentials to authenticate with the Management Plane. Misconfigured credentials can lead to authentication issues; making host disconnected from the Management Plane.

Resolution

Workaround steps:

Retrieve username & password from /etc/pf9/hostagent.conf file present over any healthy host from the same region.
Over affected host, modify the credentials for amqp section in /etc/pf9/hostagent.conf file.

/etc/pf9/hostagent.conf
    
xxxxxxxxxx
 
[amqp]host=example-region.platform9.netusername=<username>password=<password>
Copy

Restart pf9-hostagent service over the affected host.

Bash
    
xxxxxxxxxx
 
# systemctl restart pf9-hostagent
Copy

Additional Information

During credential rotation activity (if performed earlier), the OFFLINE hosts would not retrieve updated credentials to authenticate hostagent with AMQP running on Management Plane.
The legacy bbslave user displayed in the username has been replaced by hostagent user now.

Last updated on

Was this page helpful?