Host disconnected due to pf9-comms proxy issue

Problem

Host is disconnected from the management plane, even though all the pf9 services [hostagent, comms and sidekick] are actively running.
Host is able to reach management plane over 443 port yet it is showing disconnected.
Pf9-hostagent logs

C
    
 
2024-03-10 16:49:39,257 - slave.py ERROR - Connection error. Retrying in 10 seconds.Traceback (most recent call last):  File "/opt/pf9/hostagent/lib/python3.6/site-packages/bbslave/slave.py", line 129, in reconnect_loop    channel_retry_period=retry_period)  File "/opt/pf9/hostagent/lib/python3.6/site-packages/bbslave/session.py", line 796, in start    socket_timeout=socket_timeout)  # in secs  File "/opt/pf9/hostagent/lib/python3.6/site-packages/bbcommon/amqp.py", line 240, in dual_channel_io_loop    conn.ioloop.start()  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 354, in start    self.poll()  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 602, in poll    self._process_fd_events(fd_event_map, write_only)  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 443, in _process_fd_events    handler(fileno, events, write_only=write_only)  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 364, in _handle_events    self._handle_read()  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 412, in _handle_read    return self._handle_disconnect()  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 288, in _handle_disconnect    self._adapter_disconnect()  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/select_connection.py", line 95, in _adapter_disconnect    super(SelectConnection, self)._adapter_disconnect()  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 154, in _adapter_disconnect    self._check_state_on_disconnect()  File "/opt/pf9/hostagent/lib/python3.6/site-packages/pika/adapters/base_connection.py", line 169, in _check_state_on_disconnect    raise exceptions.IncompatibleProtocolErrorpika.exceptions.IncompatibleProtocolError
Copy

Pf9-sidekick logs

C
    
xxxxxxxxxx
 
[2024-03-10 16:49:16.647] [ERROR] client - request or response processing failed: RequestError: Error: read ECONNRESET    at new RequestError (/opt/pf9/sidekick/node_modules/request-promise/lib/errors.js:11:15)    at Request.RP$callback [as _callback] (/opt/pf9/sidekick/node_modules/request-promise/lib/rp.js:60:32)    at self.callback (/opt/pf9/sidekick/node_modules/request/request.js:185:22)    at Request.emit (events.js:400:28)    at Request.onRequestError (/opt/pf9/sidekick/node_modules/request/request.js:877:8)    at ClientRequest.emit (events.js:412:35)    at TLSSocket.socketErrorListener (_http_client.js:475:9)    at TLSSocket.emit (events.js:400:28)    at emitErrorNT (internal/streams/destroy.js:106:8)    at emitErrorCloseNT (internal/streams/destroy.js:74:3)    at processTicksAndRejections (internal/process/task_queues.js:82:21)
Copy

Environment

Platform9 Managed OpenStack - All versions
Platform9 Managed Kubernetes - All versions

Cause

Service pf9-comms failing to establish communication with sni-broker and sni-vouch specifically. Rest of the tunnels are intact.

C
    
 
[2024-03-10 16:34:23.625] [ERROR] sni-broker-openstack.v4.<DU-FQDN>-127.0.0.1-5673-3 - TLS socket for client 118749 error: Error: read ECONNRESET[2024-03-10 16:34:23.626] [ERROR] sni-broker-openstack.v4.<DU-FQDN>-127.0.0.1-5673-3 - TLS socket for client 118750 error: Error: read ECONNRESET[2024-03-10 16:35:13.927] [ERROR] sni-vouch-private.v4.<DU-FQDN>-127.0.0.1-5673-3 - TLS socket for client 118735 error: Error: read ECONNRESET
Copy

Resolution

Proxy for pf9-comms service needs to be enabled as per KB; that allows the host to connect to Platform9 Management Plane via proxy server instead of direct connectivity.

Last updated on

Was this page helpful?