Host Fails to Converge With Error: "rpmdb open failed"

Problem

  • A host is failing to converge with the following error printed in /var/log/pf9/hostagent.log.

246 - session.py INFO - --- Converging ---
...
188 - pf9_app.py INFO - Setting the desired service state
703 - pf9_app_db.py ERROR - Erase command failed : sudo /opt/pf9/hostagent/bin/pf9-yum erase pf9-*. Return code: 1, stdout: , stderr: error: db5 error(11) from dbenv->open: Resource tempo
rarily unavailable
error: cannot open Packages index using db5 - Resource temporarily unavailable (11)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:


Error: rpmdb open failed

Environment

  • Platform9 Managed OpenStack - All Versions

  • Platform9 Managed Kubernetes - All Versions

  • CentOS

Cause

  • One or more partitions on the root disk are full.

  • The Yum database is corrupted on disk.

Resolution

  1. Check the disk utilization on the host.

# df -h
  1. If one or more partitions are full, identify where space is being utilized and clear the necessary amount of disk space so that the Platform9 components may continue to function.

# du -h /partition | sort -h
  1. If there is a sufficient amount of disk space available (<97%) on all of the root disk partitions and yum commands fail with the same error, the package database may need to be rebuilt.

# rpm --rebuilddb
  1. Restart the pf9-hostagent service.

# systemctl restart pf9-hostagent

Additional Information

How to Fix Corrupted Package Database for Yum