Auditd Production Setup: Rules, Logs, and Performance Tuning

A compliance team wants audit trails for file access, privilege escalation, and user commands. You install auditd, add rules with auditctl, then reboot and the rules vanish. Or the disk fills with audit logs in a week. Or auditd drops 20% of events under load. An auditd production setup requires persistent rules, log rotation that scales, and tuning to keep overhead under control. Below is the checklist: writing rules that survive reboots, measuring performance impact, configuring the dispatcher, and avoiding the pitfalls that break auditing or flood disks.

You will see real commands with flags that matter, config snippets with the keys you need to change, and the trade-offs for high-throughput servers. The goal is a production-ready audit system that delivers evidence when you need it without bringing down the server or filling storage in three days.

Installing and Enabling the Audit Daemon

On RHEL, Rocky, AlmaLinux or Fedora, install the audit package. On Debian and Ubuntu, it is auditd. Both include the daemon, auditctl command-line tool, and log utilities.

# RHEL/Rocky/AlmaLinux/Fedora
dnf install audit

# Debian/Ubuntu
apt install auditd

Enable and start the service so it runs on every boot:

systemctl enable --now auditd
systemctl status auditd

The status output should show active (running). The audit daemon is userspace; it reads events from the kernel audit subsystem and writes them to /var/log/audit/audit.log. Verify the kernel subsystem is enabled:

auditctl -s

Look for enabled 2 (2 means locked, immutable until reboot; 1 means enabled but mutable) and a pid greater than zero. If enabled 0, the kernel audit support is off or not compiled. Rules you add with auditctl live in kernel memory and disappear on reboot unless you make them persistent in configuration files.

Writing Persistent Audit Rules

Rules placed directly with auditctl -w ... or auditctl -a ... vanish on reboot. For production, store rules in /etc/audit/rules.d/*.rules. On RHEL, Fedora, and derivatives, augenrules --load assembles these files into /etc/audit/audit.rules at boot. On Debian and Ubuntu, the service restart reads /etc/audit/rules.d/*.rules directly. Either way, the rules load automatically when auditd starts.

The /etc/audit/rules.d/*.rules File Structure

Use a numbered naming scheme so rules load in a predictable order. The kernel processes some filter chains first-match-wins, so sequencing matters for performance and correctness. A common layout:

• 10-base.rules – buffer size, failure mode, immutable flag at the end
• 20-file-access.rules – watches on sensitive files and directories
• 30-privilege.rules – syscalls for privilege escalation, user/group changes
• 40-network.rules – network-related syscalls if needed
• 99-finalize.rules – make the ruleset immutable with -e 2

After editing, validate syntax with augenrules --check (RHEL-family) or restart the service and check systemctl status auditd for errors.

Rule Syntax: -w, -a, -S and Key Tags

File watches use -w /path -p permissions -k keyname. Permissions are r (read), w (write), x (execute), a (attribute change). The -k key is a label you search for later with ausearch -k keyname.

-w /etc/passwd -p wa -k passwd-changes
-w /var/log -p wa -k log-tampering

Syscall rules use -a action,list -F field=value -S syscall -k key. The action is usually always or never; the list is typically exit (log on syscall exit). Filters (-F) let you match architecture, UID, executable path, or success/failure.

-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=unset -k user-commands
-a always,exit -F arch=b64 -S open -S openat -F dir=/etc/ -F perm=wa -k etc-writes

The first rule logs every command run by a real user (UID ≥1000, not a system account or unset audit UID). The second logs writes to /etc/ via open or openat. Always specify -F arch=b64 on 64-bit systems; omitting it can cause the rule to fire twice (once for 32-bit syscall layer, once for 64-bit).

Common Rule Patterns for Files, Syscalls, and Users

Watch critical authentication and configuration files:

-w /etc/shadow -p wa -k shadow-changes
-w /etc/sudoers -p wa -k sudoers-changes
-w /etc/ssh/sshd_config -p wa -k sshd-config

Log privilege escalation and identity changes:

-a always,exit -F arch=b64 -S setuid -S setgid -S setreuid -S setregid -k privilege-escalation
-a always,exit -F arch=b64 -S chown -S fchown -S fchownat -k ownership-changes

Track file deletions:

-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k file-deletion

Filter by executable to reduce noise. For example, log only passwd calls, not every write to /etc/shadow by automated tools:

-a always,exit -F arch=b64 -S open -S openat -F exe=/usr/bin/passwd -k passwd-tool

After adding rules, reload with augenrules --load (or systemctl restart auditd on Debian). Verify they are active:

auditctl -l

The output lists all loaded rules. If a rule does not appear, check for syntax errors or conflicting earlier rules that match the same events. The auditctl man page documents every filter field and syscall name.

Performance Impact and Tuning

Audit rules add overhead. Every matched syscall or file access generates an event, queues it in the kernel, and writes it to disk. Broad rules on a busy server can saturate the audit subsystem, drop events, or slow I/O. Measure before and after you deploy rules, and tune the backlog and rate limits to match your workload.

Measuring Audit Overhead with auditctl -s

Run auditctl -s to see the current state of the audit subsystem:

enabled 2
failure 1
pid 1234
rate_limit 0
backlog_limit 8192
lost 0
backlog 12
backlog_wait_time 0

The lost counter is critical. If it climbs, the kernel is dropping events because the backlog filled faster than auditd could drain it. Check lost after adding rules and again under load. A non-zero lost means you need a larger backlog, a higher rate limit, or fewer rules. The backlog line shows how many events are queued right now; if it stays near backlog_limit, you are close to dropping events.

To see the impact on throughput, run a disk-heavy workload (compile, extract a tarball, run a database benchmark) with and without audit rules. Broad syscall rules (especially open, read, write on all files) can reduce throughput by 5–15% on high-I/O servers. File watches (-w) are cheaper than syscall rules because the kernel only checks the path, not every syscall invocation.

The backlog_limit, rate_limit, and Failure Modes

Set backlog_limit in /etc/audit/rules.d/10-base.rules at the top of your ruleset:

-b 16384

The default is often 64 or 8192, depending on distribution and kernel version. For a server with hundreds of processes and heavy syscall rules, 16384 or 32768 is safer. Increasing the backlog uses more kernel memory but prevents event loss during bursts.

The rate_limit caps how many messages per second the kernel will generate. The default is 0 (unlimited). Set a limit if you want to prevent a runaway process from flooding the audit log:

-r 1000

This allows 1000 messages/sec. Events beyond that are silently dropped, so use it carefully. On most production systems, leave rate_limit 0 and size the backlog instead.

The failure mode controls what happens when the audit subsystem cannot write events (disk full, auditd stopped). Set it in 10-base.rules:

-f 1

Modes: 0 = silent (drop events, no warning), 1 = printk (log to dmesg, continue), 2 = panic (halt the system). Never use -f 2 unless compliance mandates it and you have tested failover; a full /var/log/audit partition will crash the server. Mode 1 is the safest default: you see warnings in dmesg, and the system stays up. For deeper tuning and log rotation strategies, see the Linux Log Management and Analysis eBook.

Searching and Reading Audit Logs

Audit events land in /var/log/audit/audit.log as text, one event per line or split across multiple lines for complex syscalls. Each line starts with type=, followed by a timestamp and fields. The ausearch tool queries these logs by key, user, time range, or event type. The aureport tool summarizes activity.

Using ausearch for Queries

Search by the key you assigned in your rules:

ausearch -k passwd-changes

This returns all events tagged with -k passwd-changes. To find activity by a specific user, use the audit UID (auid, the original login UID, not the effective UID):

ausearch -ua 1001

For a time range, combine -ts (start) and -te (end). Formats include today, now, recent (last 10 minutes), or absolute timestamps like 01/15/2025 14:00:00:

ausearch -ts today -te now -k file-deletion

Add -i to interpret numeric fields into human-readable names (UIDs to usernames, syscall numbers to names, hex to text):

ausearch -k etc-writes -i

Without -i, you see raw UIDs and syscall numbers. The ausearch man page lists every filter option, including -f for filename, -p for process ID, and -sv for success/failure.

Generate summary reports with aureport. For a count of events by syscall:

aureport --syscall --summary

By user:

aureport --user --summary

By file:

aureport --file --summary

These summaries help you spot which rules generate the most events and whether a user or process is abnormally active.

Understanding the Raw Log Format

A typical syscall event looks like this:

type=SYSCALL msg=audit(1705334401.234:56789): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=7ffde4b2a1c0 a2=80000 a3=0 items=1 ppid=1234 pid=5678 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=3 comm="cat" exe="/usr/bin/cat" key="file-open"

The msg=audit(timestamp:serial) is a unique event identifier. The arch=c000003e is x86_64, syscall=257 is openat. The auid=1000 is the audit UID (the user who logged in), uid=0 is the effective UID (root in this case). The key="file-open" matches the -k tag in your rule.

One logical operation often generates multiple records. An openat call produces a SYSCALL record, a PATH record with the file path, and a CWD record with the current directory. All share the same timestamp and serial number. ausearch groups them together when you query by key or time. If you grep the raw log, you see each record separately, so always use ausearch for queries to get the full context.

Log Rotation, Retention, and Centralization

Audit logs grow fast. A server with broad syscall rules can write gigabytes per day. You must configure rotation limits and decide whether to keep logs locally or forward them to a central collector. Auditd handles rotation internally; logrotate is not used for audit.log.

Configuring auditd's Built-In Rotation

Edit /etc/audit/auditd.conf. The key settings are max_log_file (size in MB before rotation), num_logs (how many rotated files to keep), and max_log_file_action (what to do when max_log_file is reached):

max_log_file = 50
num_logs = 10
max_log_file_action = rotate

This rotates at 50 MB and keeps 10 files (500 MB total). When the 11th rotation occurs, the oldest file is deleted. Other max_log_file_action values: keep_logs (rotate but never delete, disk will fill), suspend (stop logging), syslog (send a syslog message and stop). Use rotate for most production setups; if compliance requires indefinite retention, forward logs to a SIEM or archive storage rather than keeping them all on the local disk.

The space_left and space_left_action settings trigger warnings when /var/log/audit approaches full. Set space_left to a value in MB (e.g., 200) and space_left_action to email or syslog so you get an alert before the partition fills. If the partition does fill, auditd will stop logging (or panic, depending on disk_full_action). Always monitor disk usage on /var/log/audit.

After changing auditd.conf, restart the service:

systemctl restart auditd

Verify the new settings took effect by checking /var/log/audit/audit.log size and waiting for a rotation to confirm num_logs is honored.

Forwarding Logs with audisp (audispd) Plugins

The audit dispatcher (audisp, also called audispd on older distributions) sends events to external programs in real time. Plugins live in /etc/audit/plugins.d/. The syslog.conf plugin forwards events to the local syslog daemon, which can then relay them to a remote syslog server or SIEM.

Edit /etc/audit/plugins.d/syslog.conf:

active = yes
direction = out
path = /sbin/audisp-syslog
type = always
args = LOG_INFO
format = string

Set active = yes to enable the plugin. The path points to the audisp-syslog binary (location varies by distribution; on Debian it may be /usr/sbin/audisp-syslog). Restart auditd to activate the plugin:

systemctl restart auditd

Now audit events appear in /var/log/syslog or /var/log/messages, tagged with audit:. Configure rsyslog or syslog-ng to forward these messages to a remote collector. For example, in /etc/rsyslog.d/audit.conf:

:programname, isequal, "audispd" @@remote-siem.example.com:514

This sends all audisp messages to remote-siem.example.com over TCP (@@). Centralized logging is common for compliance frameworks (PCI-DSS, HIPAA) that require tamper-proof log storage. For more on rsyslog and log aggregation, see the Linux Log Management and Analysis eBook.

Warning: The audit dispatcher uses an in-memory queue. If the queue fills (default q_depth is often 80), events are dropped. Increase q_depth in /etc/audit/audisp-remote.conf or the plugin config if you see dispatcher queue full messages in dmesg. A larger queue uses more memory but prevents loss during bursts.

Common Pitfalls and How to Avoid Them

Rules added directly with auditctl -w /etc/passwd -p wa or auditctl -a always,exit -S execve live in kernel memory and vanish on reboot. Always put production rules in /etc/audit/rules.d/*.rules so they reload automatically when auditd starts. After editing a .rules file, run augenrules --load (RHEL-family) or systemctl restart auditd (Debian/Ubuntu) to activate them. Verify with auditctl -l; if a rule is missing, check for syntax errors or a missing augenrules step.

Overlapping Rules and Incorrect Architecture Filters

Forgetting -F arch=b64 on x86_64 systems can cause a syscall rule to fire twice: once for the 32-bit compatibility layer, once for the native 64-bit syscall. This doubles event volume and confuses log analysis. Always specify the architecture explicitly:

-a always,exit -F arch=b64 -S open -S openat -k file-access

If you need to audit both 32-bit and 64-bit syscalls (rare on modern servers), write two rules with arch=b32 and arch=b64. Overlapping rules (two rules that match the same event with different keys) generate duplicate log entries. The kernel evaluates rules in order, and both fire. Review auditctl -l to spot duplicates and consolidate them into a single rule with multiple -S syscalls or a broader -F dir= filter.

Setting failure=2 in Test Environments

The -f 2 failure mode panics the kernel if the audit subsystem cannot write events. This is sometimes required by compliance frameworks, but it will crash the server if /var/log/audit fills or auditd stops unexpectedly. Never use -f 2 in a test environment or on a server without robust monitoring of disk space and auditd health. If compliance mandates panic-on-failure, test it on a non-critical system first, ensure space_left and space_left_action are configured to alert before the partition fills, and document the runbook for recovery (which requires a reboot to clear the immutable audit state).

Monitoring Lost Events with auditctl -s

The lost counter in auditctl -s increments silently when the kernel backlog overflows. Check it daily with a monitoring script or add it to your metrics pipeline. A rising lost count means you are missing events, which defeats the purpose of auditing. Fix it by increasing backlog_limit (e.g., from 8192 to 32768), removing noisy rules, or adding filters to reduce event volume. Do not ignore lost events; they often signal a misconfigured ruleset or undersized backlog.

Logging Every File Access System-Wide

A rule like -a always,exit -F arch=b64 -S open -S openat -k all-opens with no filters will log every file open on the server. On a busy application server or database host, this generates tens of thousands of events per minute and fills the disk in hours. Start with a minimal ruleset: watch /etc/passwd, /etc/shadow, /etc/sudoers, key binaries in /usr/bin or /usr/sbin, and a few critical directories. Test under realistic load for a day, check disk usage and lost events, then expand gradually. Use -F dir=/etc/ or -F exe=/usr/bin/sudo to narrow scope and reduce noise.

Security Hardening for the Audit System Itself

Audit logs are evidence. If an attacker can delete or modify them, the audit system is useless. Protect /var/log/audit with filesystem permissions, append-only attributes, and rules that log changes to audit configuration itself. Send logs off-box immediately so local tampering is detectable.

Making Logs Immutable with chattr +a

Set the append-only attribute on audit.log after initial setup and testing:

chattr +a /var/log/audit/audit.log

This allows auditd to append new events but prevents deletion or modification, even by root. To remove the attribute (for example, to manually rotate or delete old logs), use chattr -a. The append-only flag persists across reboots but can be cleared if an attacker has root and knows to run chattr -a. For stronger protection, remount /var/log/audit as a separate partition with the nodev and nosuid mount options, and consider full-disk encryption to prevent offline tampering.

Restricting Directory Permissions and Auditing Configuration Changes

Ensure /var/log/audit is readable only by root:

chmod 0700 /var/log/audit
chown root:root /var/log/audit

This prevents non-root users from reading audit logs, which often contain sensitive data (command arguments, file paths, user activity). Add a rule to log changes to audit configuration files:

-w /etc/audit/ -p wa -k audit-config
-w /etc/audit/rules.d/ -p wa -k audit-rules

Any modification to auditd.conf, plugin configs, or rule files will appear in the audit log. If an attacker disables a rule or changes the failure mode, you have a record. For deeper defense, forward logs to a remote syslog server or SIEM using the audisp syslog plugin (covered earlier) so local log tampering is immediately visible.

Ensuring Early Start with Systemd

On systemd-based distributions, auditd should start early in the boot sequence so no events are missed. Check /usr/lib/systemd/system/auditd.service (do not edit this file directly; use a drop-in if you need changes). It should have DefaultDependencies=no and Before=sysinit.target to start before most other services. If auditd starts late, early boot events (kernel module loads, initial network configuration) will not be logged. Most distributions configure this correctly by default, but verify it if you see gaps in the log after a reboot.

SELinux policies often protect auditd processes and log files. Do not disable SELinux to "fix" audit issues; instead, check for AVC denials in /var/log/audit/audit.log and adjust the policy or file contexts. Disabling SELinux removes a layer of protection for the audit subsystem itself.

Production Deployment Checklist

Before deploying auditd in production, verify each step:

• Rules are in /etc/audit/rules.d/*.rules, not added with auditctl directly
• augenrules --check (RHEL-family) or systemctl restart auditd (Debian/Ubuntu) runs without errors
• auditctl -l shows all expected rules with correct -F arch=b64 filters
• backlog_limit set to 16384 or higher in 10-base.rules
• failure mode set to 1 (printk) unless compliance requires 2 (panic) and you have tested failover
• max_log_file and num_logs in auditd.conf sized for your retention policy and disk space
• space_left and space_left_action configured to alert before /var/log/audit fills
• audisp syslog plugin enabled and tested if you forward logs to a SIEM
• chattr +a /var/log/audit/audit.log applied after testing
• /var/log/audit permissions set to 0700 root:root
• Monitoring script or metric for auditctl -s to track lost events
• Test under realistic load for 24 hours; check disk usage, lost counter, and log volume

Start with a minimal ruleset (authentication files, sudoers, key binaries), measure performance impact, then expand gradually. For a comprehensive guide to auditd rules, log analysis, and integration with systemd journal, see the Linux System Auditing with Auditd and Systemd Journal eBook. It covers advanced rule patterns, correlation with journal logs, and case studies from production environments.

Frequently Asked Questions

Why do my auditctl rules disappear after a reboot?

Rules added with auditctl directly live in kernel memory and are not persistent. Put them in /etc/audit/rules.d/*.rules instead. On RHEL-family systems, run augenrules --load to compile them into /etc/audit/audit.rules. On Debian and Ubuntu, restart auditd to load the files. Verify with auditctl -l after a reboot to confirm they persist.

What is the performance overhead of auditd on a busy server?

Overhead depends on the ruleset. File watches (-w) add 1–3% CPU and I/O overhead. Broad syscall rules (e.g., all open or read calls) can reduce throughput by 5–15% on high-I/O workloads. Measure with auditctl -s to check for lost events and run realistic benchmarks before and after deploying rules. Start with a minimal ruleset and expand based on actual impact.

How do I send audit logs to a remote syslog server?

Enable the audisp syslog plugin by editing /etc/audit/plugins.d/syslog.conf and setting active = yes. Restart auditd. Audit events will appear in /var/log/syslog or /var/log/messages. Configure rsyslog or syslog-ng to forward these messages to a remote server using @@remote-host:514 for TCP or @remote-host:514 for UDP. Test with logger to confirm forwarding works.

What does 'lost events' in auditctl -s mean and how do I fix it?

The lost counter increments when the kernel audit backlog overflows and events are dropped. This happens when the backlog fills faster than auditd can drain it. Fix it by increasing backlog_limit in your rules (e.g., -b 32768), reducing the number of rules, or adding filters to narrow rule scope. Check auditctl -s regularly to catch lost events early.

Should I set auditd failure mode to panic in production?

Only if compliance mandates it and you have tested failover. -f 2 panics the kernel if the audit subsystem cannot write events (disk full, auditd stopped). This crashes the server. Use -f 1 (printk) for most production environments; it logs warnings to dmesg and keeps the system running. If you must use panic mode, configure space_left alerts and monitor disk usage closely.