Quick Summary: Linux logs are your primary diagnostic tool for troubleshooting system issues, security incidents, and performance problems. Modern Linux uses journald (systemd journal) for structured logging alongside traditional syslog. This guide covers log locations, journalctl queries, log rotation, centralized logging, and practical analysis techniques.
Where Linux Stores Logs
| Log File | Contents | Key Use |
|---|---|---|
/var/log/syslog | General system messages | First place to check for issues |
/var/log/auth.log | Authentication events | SSH logins, sudo usage, failed attempts |
/var/log/kern.log | Kernel messages | Hardware issues, driver errors |
/var/log/nginx/ | Web server logs | Access patterns, errors |
/var/log/postgresql/ | Database logs | Slow queries, errors |
/var/log/fail2ban.log | Intrusion prevention | Banned IPs, attack patterns |
/var/log/cron | Cron job execution | Scheduled task success/failure |
journalctl: Modern Log Queries
| Command | Purpose |
|---|---|
journalctl -u nginx | Logs for a specific service |
journalctl -u nginx --since "1 hour ago" | Time-filtered logs |
journalctl -u nginx -f | Follow logs in real time |
journalctl -p err | Only error-level messages |
journalctl -b | Logs since last boot |
journalctl -b -1 | Logs from previous boot |
journalctl --since "2026-03-25" --until "2026-03-26" | Date range query |
journalctl -o json-pretty | JSON output for parsing |
journalctl --disk-usage | Check journal storage size |
Log Rotation with logrotate
logrotate prevents log files from consuming all disk space:
- Configuration:
/etc/logrotate.confand/etc/logrotate.d/ - Key directives:
daily/weekly/monthly,rotate 14(keep 14 rotations),compress,missingok,notifempty - Test:
logrotate -d /etc/logrotate.conf(debug/dry-run mode) - Force rotation:
logrotate -f /etc/logrotate.conf
Practical Log Analysis Techniques
- Find failed SSH logins:
grep "Failed password" /var/log/auth.log | tail -20 - Count requests per IP:
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10 - Find 500 errors:
grep " 500 " /var/log/nginx/access.log - Monitor disk space alerts in syslog:
grep -i "no space" /var/log/syslog - Check OOM killer activity:
grep -i "out of memory" /var/log/kern.log
Centralized Logging
| Tool | Type | Best For |
|---|---|---|
| ELK Stack (Elasticsearch, Logstash, Kibana) | Full-featured | Large deployments, search and analytics |
| Loki + Grafana | Lightweight | Kubernetes, cost-effective |
| Graylog | Enterprise | Compliance, alerting |
| rsyslog forwarding | Simple | Small setups, central syslog server |
Frequently Asked Questions
How long should I keep logs?
Security logs: minimum 90 days (many regulations require 1 year). Application logs: 30-90 days depending on debugging needs. Access logs: 30 days minimum for traffic analysis. Always balance retention with storage costs.
How do I find why my server crashed?
Check journalctl -b -1 for logs from the previous boot. Look for OOM killer messages (grep "oom" /var/log/kern.log), kernel panics, or service failures. Check dmesg for hardware-related issues.
What is the difference between syslog and journald?
journald is systemd's structured logging system with powerful querying. syslog (rsyslog) is the traditional text-based logging system. Most modern distributions run both: journald captures all logs, rsyslog writes traditional log files. They complement each other.
