Introspection Fails During Baremetal Provisioning Due to the FMS Error

Problem

During baremetal provisioning, Introspection fails due to the FMS Error.

Environment

Platform9 Managed OpenStack - v8.0 and Higher
Ironic Regions

Cause

This is a known bug and is reported internally with ID: IAAS-10796
In ironic-inspector logs (on Management Plane), check for the below logs.
- The dnsmasq filter sync fails while fetching ironic_macs in dnsmasq driver due to 504 error.

ironic-api.log
    
​x
 
[Timestamp] 25367 ERROR ironic_inspector.pxe_filter.base [-] The PXE filter DnsmasqFilter, state=initialized encountered an exception: HttpException: 504: Server Error for url: https://[DU-FQDN]/ironic/v1/ports?fields=address, 504 Gateway Time-out: nginx/1.18.0; resetting the filter: openstack.exceptions.HttpException: HttpException: 504: Server Error for url: https://[DU-FQDN]/ironic/v1/ports?fields=address, 504 Gateway Time-out: nginx/1.18.0 [Timestamp] 25367 DEBUG ironic_inspector.pxe_filter.base [-] Resetting the PXE filter driver DnsmasqFilter, state=initialized reset /opt/pf9/ironic-inspector/lib/python3.9/site-packages/ironic_inspector/pxe_filter/base.py:121 [Timestamp] 772 25367 WARNING ironic_inspector.pxe_filter.base [-] Filter driver DnsmasqFilter, state=uninitialized disabling periodic sync task because of an invalid state.: ironic_inspector.pxe_filter.base.InvalidFilterDriverState: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition)
Copy

The following introspection attempt after this fails due to the FSM error.

ironic-inspector.log
    
 
[Timestamp] 25367 INFO ironic_inspector.node_cache [-] Node [Node UUID] was created successfully [TImestamp] 417 25367 ERROR ironic_inspector.node_cache [-] [node: [Node UUID] state processing] Processing the error event because of an exception <class 'ironic_inspector.pxe_filter.base.InvalidFilterDriverState'>: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition) raised by ironic_inspector.process._process_node: ironic_inspector.pxe_filter.base.InvalidFilterDriverState: The PXE filter driver DnsmasqFilter, state=uninitialized: my fsm encountered an exception: Can not transition from state 'uninitialized' on event 'sync' (no defined transition)
Copy

Workaround

Workaround is to add exception handling for the ironic API call.
Refer - Do not reset FSM when ironic ports cant be queried.
Patch the controller host with below fix:

On Controller Host
    
xxxxxxxxxx
 
$ sudo vim /opt/pf9/ironic-inspector/lib/python3.9/site-packages/ironic_inspector/pxe_filter/dnsmasq.py +88 ---        try:            # active_macs are the MACs for which introspection is active            active_macs = node_cache.active_macs()            # ironic_macs are all the MACs know to ironic (all ironic ports)            ironic_macs = set(port.address for port in                              ir_utils.call_with_retries(ironic.ports, limit=None,                                                         fields=['address']))        except Exception:            LOG.exception(                "Could not list ironic ports, can not sync dnsmasq PXE filter")            return
Copy

Restart Ironic services

Command
    
xxxxxxxxxx
 
$ sudo systemctl restart pf9-ironic-*
Copy

Additional Information

The issue is fixed in the Platform9 Managed Openstack v5.10.1 Release.

Last updated on

Was this page helpful?