Linux Ulimit and Resource Limits: Preventing Open File and Process Exhaustion

"Too many open files." Every operations engineer has stared at that error message. The Linux ulimit machinery — and its system-wide counterparts — controls how many file descriptors, processes, and other resources each user can consume. Default limits were sized for university timeshare machines in 1990; production servers running databases, web servers, or message brokers blow past them in minutes. This guide explains the layers, the limits that actually matter, and how to set them correctly for shells, cron jobs, and systemd services.

Inspecting current limits

ulimit -a                       # all limits for the current shell
ulimit -Sa                      # soft limits
ulimit -Ha                      # hard limits
cat /proc/$$/limits             # the same, more readable
prlimit --pid 1234              # any other PID

The columns matter: the soft limit is what the kernel enforces; the hard limit is the ceiling the user may raise to. Only root can raise hard limits.

The four limits you will actually adjust

• nofile — open file descriptors. Most common bottleneck. Default 1024.
• nproc — concurrent processes/threads for the user. Default 4096 to 32768 depending on distro.
• memlock — pages that may be locked into RAM (prevents paging). Required for InfluxDB, RocksDB, MongoDB.
• stack — per-thread stack size. Lower values = more concurrent threads (Go runtimes care).

Setting limits via PAM

For interactive logins and any process started under PAM, edit /etc/security/limits.conf or — better — drop a file in /etc/security/limits.d/:

# /etc/security/limits.d/99-app.conf
appuser   soft  nofile  65535
appuser   hard  nofile  65535
appuser   soft  nproc   32768
appuser   hard  nproc   32768
*         soft  core    0           # disable core dumps for everyone
root      soft  nofile  65535
root      hard  nofile  65535

For these to apply, pam_limits.so must be in /etc/pam.d/common-session (Debian) or /etc/pam.d/system-auth (RHEL). Confirm with grep pam_limits /etc/pam.d/*.

Why your changes do not stick under systemd

Systemd services do not run through PAM by default. limits.conf is irrelevant to them. Set limits in the unit file:

# /etc/systemd/system/myapp.service.d/limits.conf
[Service]
LimitNOFILE=65535
LimitNPROC=32768
LimitMEMLOCK=infinity
LimitCORE=0

After editing, systemctl daemon-reload then systemctl restart myapp. Verify with systemctl show myapp | grep ^Limit.

System-wide ceilings

Per-user limits are bounded by global kernel limits set via sysctl:

# /etc/sysctl.d/99-fs.conf
fs.file-max = 2097152            # system-wide max open files
fs.nr_open = 1048576             # per-process max FDs
kernel.pid_max = 4194303
kernel.threads-max = 1048576

Apply with sysctl --system. Without raising fs.nr_open, no process — even root — can exceed its current value, no matter what limits.conf says.

Verifying inside the running process

The limit enforced by the kernel is the one inside /proc/PID/limits, not what your shell reports. Always check there:

pgrep -f myapp | head -1 | xargs -I{} cat /proc/{}/limits | head -8

If your application reads its own limit at start (most do, e.g. nginx's worker_rlimit_nofile), restart it after changing systemd units.

Per-user shell defaults via /etc/profile.d

For shells launched without PAM (e.g. su - or some container exec), drop into profile:

# /etc/profile.d/limits.sh
ulimit -n 65535 2>/dev/null || true

This is a fallback; PAM and systemd should be your primary mechanisms.

Container considerations

Docker and Podman support --ulimit:

docker run --ulimit nofile=65535:65535 --ulimit nproc=32768:32768 nginx

Kubernetes does not expose ulimits per pod. Set them on the host kubelet or in a privileged init container that calls prlimit --pid 1 --nofile=65535:65535.

Diagnosing FD exhaustion

cat /proc/sys/fs/file-nr             # currently allocated, free, max
ls -1 /proc/$(pgrep -f myapp | head -1)/fd | wc -l
sudo lsof -p $(pgrep -f myapp | head -1) | awk '{print $5}' | sort | uniq -c

The lsof aggregation tells you whether the leak is sockets, regular files, pipes, or anonymous inodes — different code paths cause each.

Common pitfalls

• Setting only the soft limit; the application still cannot raise it past the hard limit.
• Editing limits.conf for a systemd-managed service and wondering why nothing changed.
• Setting LimitNOFILE=infinity — older kernels treat this as 4096 because of an int-cast bug; use a concrete large value like 1048576.
• Forgetting fs.nr_open when raising LimitNOFILE above 1048576.
• Disabling all core dumps globally then losing the one core dump that would have explained a production crash; consider a per-service core configuration with coredumpctl.

Quick checklist for a new server

1. Drop 99-fs.conf with fs.file-max=2097152, fs.nr_open=1048576.
2. For PAM-launched workloads: a limits.d drop-in per service user.
3. For systemd-managed workloads: LimitNOFILE, LimitNPROC in a service drop-in.
4. Validate via /proc/PID/limits after restart.
5. Add fs.file-nr to your monitoring dashboard.

Resource limits are unglamorous plumbing, but they are the difference between a production incident at 1000 concurrent connections and a quiet server humming along at 100,000. Set them once, deliberately, per host class — and never accept the 1024 default in production again.