Managed OpenStack
Latest
Frequently Asked Questions
Solution
How To
Internal Only
Templates
Powered By
Title
Message
Create new category
What is the title of your new category?
Edit page index title
What is the title of the page index?
Edit category
What is the new title of your category?
Edit link
What is the new title and URL of your link?
Live Migration Fails With Error: "Too many open files
Copy Markdown
Open in ChatGPT
Open in Claude
Problem
- Live migration of an instance fails with error "Too many open files" as see in /var/log/nova/nova-conductor.log.
04 3437 WARNING nova.scheduler.utils [req-a34cc0ab-5cc2-4332-b2e4-9c2b16c934d8 9f39d094aa9649da8fdd01f84d3a6a04-57a5eabb1acb4f8a90931897bafb19fc - - -] Failed to compute_task_migrate_server: [Errno [Errno 24] Too many open files- Below errors are seen in /var/log/pf9/ostackhost.log.
442 TRACE nova.compute.manager [instance: 31879d2c-6096-4db6-aadd-7d0201226e9d] ConnectFailure: Unable to establish connection to http://localhost:8158/keystone/v3/auth/tokens: HTTPConnectionPool(host='localhost', port=8158): Max retries exceeded with url: /keystone/v3/auth/tokens (Caused by NewConnectionError('[requests.packages.urllib3.connection.HTTPConnection object at 0x7f20b45c5550]: Failed to establish a new connection: [Errno 24] Too many open files',))Environment
- Platform9 Managed OpenStack - v3.6.0 and Higher
Cause
For the majority of operating systems, 1024 is the maximum number of open files per process. But for the pf9-ostackhost process, we can see that the threshold has been crossed and there are 1158 files opened.
$ systemctl status pf9-ostackhost● pf9-ostackhost.service - Platform9 OpenStack Compute service Loaded: loaded (/lib/systemd/system/pf9-ostackhost.service; disabled; vendor preset: enabled) Active: active (running) since Thu XXXX-XX-XX YY:YY:YY PDT; 11 months 25 days ago Main PID: 4579 (pf9-ostackhost) Tasks: 24 Memory: 433.9M CPU: 4month 2d 10h 41min 57.056s CGroup: /system.slice/pf9-ostackhost.service ├─ 4579 /opt/pf9/venv/bin/python /opt/pf9/venv/bin/pf9-ostackhost --config-dir /opt/pf9/etc/nova/conf.d/ --log-file /var/log/pf9/ostackhost.log ├─ 5468 /opt/pf9/venv/bin/python /opt/pf9/venv/bin/privsep-helper --config-dir /opt/pf9/etc/nova/conf.d --privsep_context vif_plug_ovs.privsep.vif_plug --privsep_sock_path /tmp/tmpawAsR0/privs └─418015 /opt/pf9/venv/bin/python /opt/pf9/venv/bin/privsep-helper --config-dir /opt/pf9/etc/nova/conf.d --privsep_context os_brick.privileged.default --privsep_sock_path /tmp/tmpS_M48Q/privsepxxxxxxxxxx$ cat /proc/4579/limitsLimit Soft Limit Hard Limit UnitsMax open files 1024 4096 filesxxxxxxxxxx$ ulimit -a | grep openopen files (-n) 1024xxxxxxxxxx$ lsof -p 4579 | wc -l1158This indicates that pf9-ostackhost service has 1158 open files and has crossed the limit of 1024.
Resolution
Restart pf9-ostackhost service on the affected host.
VariableType to search · ESC to discard
GlossaryType to search · ESC to discard
InsertType to search · ESC to discard
No matches
Last updated on
Was this page helpful?
Discard Changes
Do you want to discard your current changes and overwrite with the template?
Archive Synced Block
Message
Create new Template
What is this template's title?
Delete Template
Message