The first artifact every responder asks for after a suspected breach is the login history โ who connected, from where, when, and what failed. Linux records this across three log streams that have existed for thirty years: wtmp, btmp, and the auth journal. Knowing how to read them in 30 seconds is the difference between confirming and missing an intrusion.
The three log files
/var/log/wtmpโ successful logins, logouts, reboots. Read withlast./var/log/btmpโ failed logins. Read withlastb(root-only by default)./var/log/lastlogโ last login per user. Read withlastlog.
On systemd-based distros, the same events also flow into the journal. Both stores are useful โ wtmp is portable; the journal is structured and queryable.
Reading recent logins
last -n 30 # last 30 successful logins
last -F # full timestamps
last -i # show source IPs (not hostnames)
last reboot # only reboot events
last -p now # who was logged in right now
last -s yesterday -t today # specific window
Watch for patterns: a user logging in from an IP you have never seen, multiple sources for one account in a short window, or a source from a country no employee operates from.
Failed login analysis
sudo lastb -n 50 # 50 most recent failures
sudo lastb -i | awk '{print $3}' | sort | uniq -c | sort -rn | head
sudo lastb | awk '{print $1}' | sort | uniq -c | sort -rn | head
The first aggregator counts attempts per source IP; the second counts per attempted username. Brute-force campaigns reveal themselves as a single source against many usernames or a single username against many sources.
SSH-specific events from the journal
journalctl -u ssh -p err --since '24 hours ago'
journalctl _COMM=sshd --grep 'Failed password' --since '1 hour ago'
journalctl _COMM=sshd --grep 'Accepted (publickey|password)' --since today \
| awk '{print $9, $11}' | sort -u
The last command produces a list of (user, source-IP) pairs that successfully authenticated today โ perfect for a daily review email.
Detecting brute force in real time
Three families of defense:
- fail2ban reads sshd logs and bans IPs at the firewall. The default jail config triggers after 5 failures in 10 minutes.
- sshguard is similar, slightly lower memory.
- PAM tally (
pam_faillock) locks the user account after N failures, regardless of source. Edit/etc/security/faillock.conf:deny=5,unlock_time=900.
Use both: fail2ban prevents the public scan, faillock prevents an attacker who already has password spray inside the network.
Currently logged-in users
who -a # all sessions, terminal, source
w # who + current command, idle time
loginctl list-sessions # systemd-managed sessions
ss -tnp 'sport = :ssh' # raw TCP view of SSH sockets
Cross-reference with expected on-call: any session not matching the schedule deserves a question.
Suspicious patterns to alert on
- First-ever login from a new country (geoip lookup of source IP).
- Login outside business hours for an account that historically only logs in 09:00โ18:00.
- Successful login immediately after a burst of failures from the same source โ almost always indicates a successful guess.
- Login as a service account (e.g.
nginx,postgres) when the policy is "no interactive logins for service users." - Empty audit log after a login โ attackers commonly clear logs as one of their first actions.
Forensic capture
If you suspect an active intrusion, copy the log files before doing anything else:
sudo cp /var/log/wtmp /var/log/btmp /var/log/lastlog /tmp/forensics/
sudo journalctl --since '2 hours ago' > /tmp/forensics/journal.log
sudo cp -r /var/log /tmp/forensics/all-logs
sha256sum /tmp/forensics/* > /tmp/forensics/SHA256SUMS
Move the bundle off the host to a write-only location before further investigation โ attackers with root will delete and modify logs in real time.
Rotation and retention
By default logrotate keeps btmp/wtmp for one month. For compliance, extend retention:
# /etc/logrotate.d/wtmp
/var/log/wtmp {
monthly
create 0664 root utmp
minsize 1M
rotate 24
}
Twenty-four months of monthly rotation gives auditors a two-year window without taking meaningful disk space.
Common pitfalls
- Trusting
lastoutput exclusively โ root attackers tamper with wtmp; corroborate with the journal and external SSH logs. - Forgetting that key-based logins still appear in
lastbas failures during the negotiation phase; filter bypublickey acceptedevents for true success. - Running
lastbas a non-root user and assuming the empty output means no failures. - Letting
btmpgrow unbounded under a brute-force campaign โ without rotation it can fill the disk.
A weekly five-minute review of last, lastb, and the SSH journal catches the slow brute force, the credential spray, and the after-hours login by a former employee whose key was not removed. Build it into your operational rhythm and you will detect intrusions in days rather than months.