Nodelet Fails at "Configure and start pf9-bouncer" Phase

Problem

Platform9 nodelet phase fails at "Configure and start pf9-bouncer" phase with "bind address already in use" on bouncer logs.

{"log":"2024/04/10 05:35:40 serve http:listen tcp 127.0.0.1:9977: bind: address already in use\n","stream":"stderr","time":"2024-04-10T05:35:40.02933616Z"}

Nodelet logs keep showing "Waiting for "authn_webhook_listening" to evaluate to true" message.

{"L":"INFO","T":"2024-04-10T05:35:39.781Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] Stopping containers 'bouncer'"}

{"L":"INFO","T":"2024-04-10T05:35:39.807Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] No such container: bouncer"}

{"L":"INFO","T":"2024-04-10T05:35:39.810Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] Destroying containers 'bouncer'"}

{"L":"INFO","T":"2024-04-10T05:35:39.836Z","C":"command/command.go:120","M":"[2024-04-10 05:35:39] bouncer"}

{"L":"INFO","T":"2024-04-10T05:35:40.030Z","C":"command/command.go:120","M":"[2024-04-10 05:35:40] 90907c098a9b96c6c3ce40605f7d2f869fb4e67d55f86f9b6ddf279a56aba7a4"}

{"L":"INFO","T":"2024-04-10T05:35:40.053Z","C":"command/command.go:120","M":"[2024-04-10 05:35:40] Waiting for \"authn_webhook_listening\" to evaluate to true ..."}

{"L":"INFO","T":"2024-04-10T05:35:45.073Z","C":"command/command.go:120","M":"[2024-04-10 05:35:45] Waiting for \"authn_webhook_listening\" to evaluate to true ..."}

Environment

SMCP version 5.9.1.

Cause

On the underlying node stale bouncer process keeps holding port 9977.

Resolution

Verify the stale process by running the below command:

# ps -aux | grep -i "9977"

# netstat -tunlp | grep -i "9977"

Once we get the process ID follow the procedure given below:

Stop the pf9-nodeletd and pf9-hostagent service - systemctl stop pf9-hostagent pf9-nodeletd.
Force stop the nodelet phases using $ /opt/pf9/nodelet/nodeletd phases stop --force .
Kill the stale bouncerd process using $ kill -9 <bouncerd-process-id> .
Start nodelet phase using - systemctl start pf9-hostagent.
Verify the nodelet phase status using $ /opt/pf9/nodelet/nodeletd phases status .

Additional Information

The above procedure helps to recover the nodelet phase issue due to the bouncer bind address already in use error.

Last updated on

Was this page helpful?