Resetting RabbitMQ Database Caused Complete Host Disconnection from Management Plane

Problem

• All hosts were disconnected from the management plane due to the absence of the bbslave user, which is required to authenticate with the RabbitMQ database.

File "/opt/pf9/hostagent/lib/python3.9/site-packages/pika/connection.py", line 1856, in _on_connection_error

    raise exceptions.AMQPConnectionError(error_message or

pika.exceptions.AMQPConnectionError: (403, 'ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.')

2025-04-25 05:28:50.371 [error] <0.14706.446> Error on AMQP connection <0.14706.446> (127.0.0.1:37640 -> 127.0.0.1:5672, state: starting):

PLAIN login refused: user 'bbslave' - invalid credentials

• No users were present on rabbitmq db

xxxxxxxxxx

[root@testbed rabbitmq]# sudo rabbitmqctl list_users

Listing users ...

user    tags

guest   [administrator]

Environment

• Platform9 Managed Edge Cloud - v5.3

Cause

• The following commands performed on rabbitmq db wiped off all the users, which makes all the hosts not able to authenticate with management plane.

xxxxxxxxxx

sudo rabbitmqctl stop_app

sudo rabbitmqctl reset

sudo rabbitmqctl start_app

sudo rabbitmqctl restart

Resolution

• The way out from the situation is to perform same version upgrade of the management plane.
• The upgrade process restores default RabbitMQ users and permissions, allowing full recovery.

Additional Information

• Do not make any modifications to the databases without consulting Platform9 Support or referring to the official Platform9 documentation.