High Memory Consumption by Journald Causing Cluster Instability

Problem

Journald seems to consume more than expected amount of memory causing cluster nodes instability.

Environment

• Centos, RedHat - v7+

Cause

• The systemd journal is configured by default to store logs only in a small ring-buffer in _/run/log/journal_, which is not persistent.
• Systemd-journald on some linux distros is configured to auto-detect which storage system to use for journal files, "volatile" or "persistent". The /var/log/journal file is not created by default, so the in-memory (tmpfs) _/run/log/journal_ location is used. By default, it is configured to allow up to 4 GB of log data be stored. As a consequence, heavy journal activity can eat up memory to the default of 4 GB for the _/run_ file system. For memory constrained environments, this can be significant

Resolution

To limit the amount of in-memory logs stored in /run/log/journal, set the following in /etc/systemd/journald.conf :

xxxxxxxxxx

---

RuntimeMaxUse=500M

RuntimeMaxFileSize=10M

---