Linux Inode Monitoring: Preventing Filesystem Inode Exhaustion

Inode exhaustion is one of the most confusing failures in Linux operations: df -h shows 60% free space, the application logs "no space left on device," and ten engineers spend an hour deleting random files in /tmp with no effect. The fix is a single character: df -i. This guide explains why inodes run out, which workloads cause it, and how to monitor for the early warning signs before the disk effectively fills.

What an inode actually is

An inode is a fixed-size record on a filesystem that stores metadata for one file: ownership, permissions, timestamps, and pointers to data blocks. The number of inodes is fixed at mkfs time on ext2/3/4 and many other filesystems. Once they are exhausted, no new files can be created — even if the disk has petabytes of free space.

df -i                        # inode usage per filesystem
df -hT --output=target,fstype,size,used,avail,pcent
sudo tune2fs -l /dev/sda1 | grep -E 'Inode count|Free inodes'

Workloads that exhaust inodes

Several common patterns produce millions of tiny files:

• Mail servers — Maildir uses one file per message; a busy inbox grows fast.
• Cache directories — Squid, varnish on-disk cache, web frameworks that cache one file per fragment.
• Session stores — PHP sessions in /var/lib/php/sessions/ with a per-session file.
• Log fragments — apps that rotate every minute or write per-request files.
• Container images — overlayfs layers each contribute thousands of inodes.
• Build systems — node_modules with hundreds of thousands of small JS files.

Finding which directory is to blame

sudo find / -xdev -type d -exec sh -c \
  'echo "$(ls -1A "$1" | wc -l) $1"' _ {} \; 2>/dev/null \
  | sort -nr | head -20

This walks each filesystem (no crossing of -xdev), counts entries per directory, and returns the top 20. On a healthy server every directory should have under a few thousand entries; anything in the millions is a problem in the making.

A faster alternative on big trees:

sudo find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -rn | head

Monitoring continuously

A 10-line script run by a systemd timer every five minutes:

#!/bin/bash
THRESHOLD=85
df -i --output=source,target,ipcent | tail -n +2 | while read src tgt pct; do
  num=${pct%%%}
  if [ "$num" -ge "$THRESHOLD" ]; then
    echo "INODE CRITICAL: $tgt at ${pct} (device $src)"
  fi
done

Pair with the same logic on df -h — both signals should be in your monitoring. Prometheus node_exporter exposes node_filesystem_files_free and node_filesystem_files; the ratio is your alert.

Provisioning more inodes

You cannot add inodes to an existing ext4 filesystem without recreating it. Plan ahead at mkfs time:

sudo mkfs.ext4 -N 10000000 /dev/sdb1     # explicit inode count
sudo mkfs.ext4 -i 4096 /dev/sdb1         # one inode per 4 KB
sudo mkfs.ext4 -T news /dev/sdb1         # preset for many small files

For workloads dominated by small files, the news preset gives a much higher inode density. XFS is dynamic — it allocates inodes on demand — and is a strong choice for Maildir, news, or cache servers.

Cleanup strategies

When you are already in trouble:

sudo find /var/lib/php/sessions -type f -mtime +14 -delete
sudo find /tmp -type f -atime +7 -not -newer /tmp/.timestamp -delete
sudo find /var/cache -type f -atime +30 -delete
sudo journalctl --vacuum-size=200M
sudo docker system prune -af

Be wary of find -delete on directories you do not own — accidentally pruning /var/lib/postgresql/ ruins the day. Always run with -name or -path filters first, then add -delete.

systemd-tmpfiles

For predictable cleanup, use systemd-tmpfiles instead of cron. Drop a file in /etc/tmpfiles.d/:

d /var/cache/myapp 0750 myapp myapp 7d
e /tmp - - - 7d -

This declares "keep /var/cache/myapp with these perms; remove anything older than 7 days; clean up stale entries in /tmp after 7 days." Apply with systemd-tmpfiles --clean; runs daily by default.

Container caveats

Each running container's writable layer consumes inodes on the host filesystem. Run:

sudo du --inodes -d 1 /var/lib/docker/overlay2 | sort -rn | head
docker system df                           # Docker's view
podman system df

Set storage.options.overlay.size= per-container in your storage.conf to cap, and prune aggressively in CI/test hosts.

Common pitfalls

• Reformatting a partition after running out of inodes without specifying -N or -i — you get the same default and the problem returns in months.
• Deleting files that the application has open — the inodes are not released until the process closes them; lsof | grep deleted reveals the culprits.
• Trusting du to find the inode hog; du measures bytes, not file counts.
• Choosing ext4 with default settings for a Maildir host — you will pay for it.

Inode exhaustion is unforgiving but extremely preventable. Add df -i to your monitoring today, audit the top per-directory counts on every server you operate, and pick the right filesystem for your workload at provisioning time.