Instances Lose External Connectivity After Detaching Floating IP

Problem

After removing a floating IP from an instance, external connectivity ceases.

Environment

  • Platform9 Managed OpenStack - v4.5 - v5.2

  • Neutron

Cause

The neutron-l3-agent service fails to update the routing rule corresponding to the instance's fixed IP within the router namespace on the L3 agent node(s) responsible for hosting the virtual router (in a DVR configuration, this would be the instance hypervisor). Subsequently, this causes a loss of external connectivity for the instance.

Resolution

  1. Identify the fixed IP, network, and host associated with the affected instance.

Screen_Shot_2020-01-02_at_3.38.20_PM.png
  1. Identify the router associated with the network.

Screen_Shot_2020-01-02_at_3.39.56_PM.png
  1. Record the UUID of the network and router identified in the previous two steps.

Info

Note: In the UI, this may be done by navigating to the "Networks" tab, selecting either Networks or Routers and then subsequently clicking on the name of the resource. An OpenStack UUID column will be visible in the resource overview.

  1. In the qrouter namespace (corresponding to the UUID of the router) on the instance's host, inspect the routing rules.

Attention

Note: In the following examples, a unique router UUID will be used which will differ from your own. Ensure you replace the UUID when executing the commands yourself.

# ip net e qrouter-204830ee-4553-4c25-b720-646e565ee4bf ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
66502: from 192.168.0.28 lookup 16
3232235521: from 192.168.0.1/24 lookup 3232235521
  1. Inspect the route table (of which the ID may be identified based on the prior step) corresponding to the instance's fixed IP.

# ip net e qrouter-204830ee-4553-4c25-b720-646e565ee4bf ip route show table 16
default via 169.254.110.93 dev rfp-c0dd3bdd-3

The default route with the rfp interface in the rule indicates all instance traffic is still going to the floating IP namespace.

  1. Delete the stale routing rule.

# ip net e qrouter-204830ee-4553-4c25-b720-646e565ee4bf ip rule del pref 66502

Additional Information