Instance Cannot Contact Metadata Service (Ubuntu 16.04)

Problem

• An instance is unable to contact the metadata service via the metadata proxy IP (169.254.169.254) with an error similar to the following.

xxxxxxxxxx

# curl http://169.254.169.254curl: (7) Failed to connect to 169.254.169.254 port 80: No route to host

• The gateway IP for the metadata proxy route is associated with a reserved DHCP port.

1. Identify the gateway IP.

# route -nKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Iface...169.254.169.254 10.0.0.1 255.255.255.255 UGH 0 0 0 ens3

2. Track the IP down to a Neutron port.

# openstack port list --network __NETWORK_ID__ --long --fit-width | grep -w 10.0.0.1

3. Check the device ID of the port.

# openstack port show __PORT_ID__ -f value -c device_id

- The value should be reserved_dhcp_port. If not, you are likely facing a separate issue with that particular DHCP agent.

Environment

• Platform9 Managed OpenStack - All Versions
• Ubuntu 16.04

Cause

A DHCP agent failover has occurred and thus the IP/route to the Neutron metadata proxy has changed.

Resolution

1. Remove the route for the metadata proxy IP.

xxxxxxxxxx

# sudo ip route del 169.254.169.254

2. Run dhclient.

xxxxxxxxxx

# dhclient -v

3. Check that the gateway IP for the 169.254.169.254 route has been updated.

xxxxxxxxxx

# route -n

4. Ensure that the metadata service is reachable via the proxy IP.

xxxxxxxxxx

# curl http://169.254.169.254