Introspection Fails During Baremetal Provisioning Due to the FMS Error

Problem

During baremetal provisioning, Introspection fails due to the FMS Error.

Environment

Platform9 Managed OpenStack - v8.0 and Higher
Ironic Regions

Cause

This is a known bug and is reported internally with ID: IAAS-10796
In ironic-inspector logs (on Management Plane), check for the below logs.
- The dnsmasq filter sync fails while fetching ironic_macs in dnsmasq driver due to 504 error.

[Timestamp] 25367 ERROR ironic_inspector.pxe_filter.base [-] The PXE filter DnsmasqFilter, state=initialized encountered an exception: HttpException: 504: Server Error for url: https://[DU-FQDN]/ironic/v1/ports?fields=address, 504 Gateway Time-out: nginx/1.18.0; resetting the filter: openstack.exceptions.HttpException: HttpException: 504: Server Error for url: https://[DU-FQDN]/ironic/v1/ports?fields=address, 504 Gateway Time-out: nginx/1.18.0

[Timestamp] 25367 DEBUG ironic_inspector.pxe_filter.base [-] Resetting the PXE filter driver DnsmasqFilter, state=initialized reset /opt/pf9/ironic-inspector/lib/python3.9/site-packages/ironic_inspector/pxe_filter/base.py:121

[Timestamp] 772 25367 WARNING ironic_inspector.pxe_filter.base [-] Filter driver DnsmasqFilter, state=uninitialized disabling periodic sync task because of an invalid state.: ironic_inspector.pxe_filter.base.InvalidFilterDriverState: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition)

The following introspection attempt after this fails due to the FSM error.

[Timestamp] 25367 INFO ironic_inspector.node_cache [-] Node [Node UUID] was created successfully

[TImestamp] 417 25367 ERROR ironic_inspector.node_cache [-] [node: [Node UUID] state processing] Processing the error event because of an exception <class 'ironic_inspector.pxe_filter.base.InvalidFilterDriverState'>: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition) raised by ironic_inspector.process._process_node: ironic_inspector.pxe_filter.base.InvalidFilterDriverState: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition)

Workaround

Workaround is to add exception handling for the ironic API call.
Refer - Do not reset FSM when ironic ports cant be queried.
Patch the controller host with below fix:

$ sudo vim /opt/pf9/ironic-inspector/lib/python3.9/site-packages/ironic_inspector/pxe_filter/dnsmasq.py +88

---

try:

# active_macs are the MACs for which introspection is active

active_macs = node_cache.active_macs()

# ironic_macs are all the MACs know to ironic (all ironic ports)

ironic_macs = set(port.address for port in

ir_utils.call_with_retries(ironic.ports, limit=None,

fields=['address']))

except Exception:

LOG.exception(

"Could not list ironic ports, can not sync dnsmasq PXE filter")

return

Restart Ironic services

$ sudo systemctl restart pf9-ironic-*

Additional Information

The issue is fixed in the Platform9 Managed Openstack v5.10.1 Release.

Last updated on

Was this page helpful?