Host Fails to Converge With Error: "rpmdb open failed"

Problem

• A host is failing to converge with the following error printed in /var/log/pf9/hostagent.log.

246 - session.py INFO - --- Converging ---

...

188 - pf9_app.py INFO - Setting the desired service state

703 - pf9_app_db.py ERROR - Erase command failed : sudo /opt/pf9/hostagent/bin/pf9-yum erase pf9-*. Return code: 1, stdout: , stderr: error: db5 error(11) from dbenv->open: Resource tempo

rarily unavailable

error: cannot open Packages index using db5 - Resource temporarily unavailable (11)

error: cannot open Packages database in /var/lib/rpm

CRITICAL:yum.main:

Error: rpmdb open failed

Environment

• Platform9 Managed OpenStack - All Versions
• Platform9 Managed Kubernetes - All Versions
• CentOS

Cause

• One or more partitions on the root disk are full.
• The Yum database is corrupted on disk.

Resolution

1. Check the disk utilization on the host.

xxxxxxxxxx

# df -h

2. If one or more partitions are full, identify where space is being utilized and clear the necessary amount of disk space so that the Platform9 components may continue to function.

xxxxxxxxxx

# du -h /partition | sort -h

3. If there is a sufficient amount of disk space available (<97%) on all of the root disk partitions and yum commands fail with the same error, the package database may need to be rebuilt.

xxxxxxxxxx

# rpm --rebuilddb

4. Restart the pf9-hostagent service.

xxxxxxxxxx

# systemctl restart pf9-hostagent

Additional Information

How to Fix Corrupted Package Database for Yum