at/batch Scheduler Security Audit: Linux Admin Guide

The at and batch commands are the forgotten cousins of cron. They schedule one-off jobs at a specific time or whenever load drops, they run as root by default if invoked by root, and they almost never appear in security baselines — which is exactly why attackers love them. A pending at job is persistent, invisible to crontab -l, and easy to overlook during forensic review. This guide covers the audit techniques that catch hostile use of at and the access controls that prevent it.

What at and batch actually do

• at — schedules a job to run once at a specific time (at 03:00 tomorrow).
• batch — schedules a job to run once when system load drops below 0.8 (configurable).
• atd — the daemon that watches for due jobs.

Jobs live in /var/spool/cron/atjobs/ (Debian) or /var/spool/at/ (RHEL) as one file per job, owned by the queuing user.

Inventory pending jobs

atq                              # current user's pending jobs
sudo atq                         # all users' pending jobs
sudo atq | sort -k2,3            # ordered by run time
sudo find /var/spool/at* /var/spool/cron/atjobs -type f 2>/dev/null -ls

Output of sudo atq: job ID, scheduled date/time, queue letter (a=at, b=batch, =running), and queueing user. A job scheduled by an unexpected user is a red flag.

Inspect what a job will run

sudo at -c 7              # show full script for job ID 7
sudo cat /var/spool/cron/atjobs/* | less    # all jobs verbatim

The output includes the entire shell environment captured at submission time plus the actual commands. Read it line by line — attackers commonly insert curl … | sh or reverse-shell patterns here.

Removing a hostile job

sudo atrm 7                       # by job ID
sudo atq | awk '$5=="bob"{print $1}' | xargs -r sudo atrm
sudo systemctl restart atd

Access control: who can submit

Two files in /etc/ control access:

• /etc/at.allow — if it exists, only listed users may run at. Everyone else is denied.
• /etc/at.deny — if at.allow does not exist, everyone except those in at.deny may run at.

The CIS-recommended baseline is an empty allow file plus secure permissions:

sudo touch /etc/at.allow
sudo chown root:root /etc/at.allow /etc/at.deny 2>/dev/null
sudo chmod 0640 /etc/at.allow /etc/at.deny 2>/dev/null
sudo rm /etc/at.deny 2>/dev/null
echo root | sudo tee -a /etc/at.allow      # explicitly allow root only

An empty at.allow means literally no user can submit at jobs — including root, unless explicitly listed. For most production servers this is the right baseline; legitimate one-off scheduling can use systemd-run instead.

Auditing for hostile use

#!/bin/bash
echo "== Pending at jobs =="
sudo atq

echo
echo "== Job contents =="
for j in $(sudo atq | awk '{print $1}'); do
  echo "--- Job $j ---"
  sudo at -c "$j" | tail -20
done

echo
echo "== Allow/deny files =="
ls -l /etc/at.allow /etc/at.deny 2>/dev/null
echo "Allowed users:"
cat /etc/at.allow 2>/dev/null

echo
echo "== atd status =="
systemctl is-active atd

Run weekly via systemd timer; ship output to your log collector. Alert on any new pending job or any change to the allow file.

Logging at activity

By default, atd logs through syslog. Verify and harden:

grep atd /var/log/syslog | tail
sudo journalctl -u atd --since today
echo '-w /var/spool/at -p wa -k at-jobs' | sudo tee -a /etc/audit/rules.d/at.rules
echo '-w /var/spool/cron/atjobs -p wa -k at-jobs' | sudo tee -a /etc/audit/rules.d/at.rules
sudo augenrules --load

The audit watches catch any attempt to write a job file directly, bypassing at itself.

If you do need at jobs

Some legitimate uses remain: a one-off database migration, a delayed restart, an admin-issued "reboot if I do not log in within an hour" failsafe. Best practices:

• Always include the job purpose in a comment at the top of the script.
• Capture stdout/stderr to a log: echo 'cmd >> /var/log/at-job.log 2>&1' | at now + 5 minutes.
• Use absolute paths — atd's PATH is minimal.
• For "delayed restart safety net" use shutdown -r +60 or systemd-run with --on-active=1h instead — both are easier to inspect and cancel.

The systemd-run alternative

For most one-off scheduling, systemd-run is cleaner than at:

systemd-run --on-active=2h /usr/local/bin/cleanup
systemd-run --on-calendar='*-*-* 03:00:00' /usr/local/bin/job
systemctl list-timers                       # see pending units
systemctl stop run-1234.timer               # cancel

The job appears in normal systemd tooling, gets journald logging, supports resource limits, and is impossible to hide from list-timers.

Common pitfalls

• Leaving the default config (no allow file, empty deny file) — every user can submit jobs.
• Forgetting that disabling atd with systemctl disable still leaves the spool directory writable; remove the package if you do not need it: apt purge at.
• Trusting crontab -l alone for "scheduled jobs on this host" — at jobs do not appear there.
• Ignoring atd in your audit ruleset because "no one uses at" — that is exactly why an attacker would.

The at scheduler is a five-minute audit and a one-line lockdown. On any server where you do not deliberately use it, restrict at.allow to root only or remove the package entirely. On servers where it is legitimate, watch the spool and ship every job submission to your SIEM.