Resetting RabbitMQ Database Caused Complete Host Disconnection from Management Plane

Problem

  • All hosts were disconnected from the management plane due to the absence of the bbslave user, which is required to authenticate with the RabbitMQ database.

File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 1856, in _on_connection_error
raise exceptions.AMQPConnectionError(error_message or
pika.exceptions.AMQPConnectionError: (403, 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.')
2025-04-25 05:28:50.371 [error] <0.14706.446> Error on AMQP connection <0.14706.446> (127.0.0.1:37640 -> 127.0.0.1:5672, state: starting):
PLAIN login refused: user 'bbslave' - invalid credentials

  • No users were present on rabbitmq db

[root@testbed rabbitmq]# sudo rabbitmqctl list_users
Listing users ...
user tags
guest [administrator]

Environment

  • Platform9 Managed Edge Cloud - v5.3

Cause

  • The following commands performed on rabbitmq db wiped off all the users, which makes all the hosts not able to authenticate with management plane.

sudo rabbitmqctl stop_app
sudo rabbitmqctl reset
sudo rabbitmqctl start_app
sudo rabbitmqctl restart

Resolution

  • The way out from the situation is to perform same version upgrade of the management plane.

  • The upgrade process restores default RabbitMQ users and permissions, allowing full recovery.

Additional Information


Do not make any modifications to the databases without consulting Platform9 Support or referring to the official Platform9 documentation.


© 2025 Platform9. All Rights Reserved

On This Page
Resetting RabbitMQ Database Caused Complete Host Disconnection from Management PlaneProblemEnvironmentCauseResolutionAdditional Information