Linux Log Management & Troubleshooting: Complete Guide (2026)

System logs are your window into what's happening on a Linux server. When services crash, security incidents occur, or performance degrades, logs are the first place you'll look. This guide covers everything from basic log viewing to advanced centralized logging configurations that scale across multiple servers.

Linux Log Locations

Understanding where logs live is the first step to effective troubleshooting:

Viewing Logs with journalctl

On systems with systemd (virtually all modern Linux distributions), journalctl is the primary log viewing tool:

Basic Viewing

# View all logs
journalctl

# Follow logs in real-time (like tail -f)
journalctl -f

# Show only recent entries
journalctl -n 50    # Last 50 lines
journalctl -n 100   # Last 100 lines

# Show logs since boot
journalctl -b
journalctl -b -1    # Previous boot
journalctl --list-boots  # List all boots

# Reverse order (newest first)
journalctl -r

Filtering by Unit/Service

# Logs for a specific service
journalctl -u nginx
journalctl -u sshd
journalctl -u mysql

# Follow service logs in real-time
journalctl -u nginx -f

# Multiple services
journalctl -u nginx -u php-fpm

Time-Based Filtering

# Since a specific time
journalctl --since "2026-01-15 10:00:00"

# Until a specific time
journalctl --until "2026-01-15 12:00:00"

# Time range
journalctl --since "2026-01-15" --until "2026-01-16"

# Relative time
journalctl --since "1 hour ago"
journalctl --since "30 minutes ago"
journalctl --since yesterday
journalctl --since today

Priority Filtering

# By priority level
journalctl -p err      # Errors and above
journalctl -p warning  # Warnings and above
journalctl -p crit     # Critical and above

# Priority levels: emerg(0), alert(1), crit(2), err(3),
#                  warning(4), notice(5), info(6), debug(7)

# Combine with service filter
journalctl -u nginx -p err --since "1 hour ago"

Output Formats

# JSON output (for parsing)
journalctl -u nginx -o json-pretty

# Short format with microseconds
journalctl -o short-precise

# Verbose output (all metadata)
journalctl -o verbose

# Export for analysis
journalctl -u nginx --since today -o json > /tmp/nginx-logs.json

Traditional Log Tools

# Real-time log watching
tail -f /var/log/syslog
tail -f /var/log/nginx/error.log

# Watch multiple logs
tail -f /var/log/nginx/access.log /var/log/nginx/error.log

# Search logs with grep
grep "error" /var/log/syslog
grep -i "failed" /var/log/auth.log
grep "404" /var/log/nginx/access.log | wc -l

# Search with context
grep -B 5 -A 5 "error" /var/log/syslog

# Search compressed logs
zgrep "error" /var/log/syslog.2.gz

# Last login information
last
lastb    # Failed logins
lastlog  # Last login for all users

rsyslog Configuration

rsyslog is the standard syslog daemon on most Linux distributions:

# Main config file
# /etc/rsyslog.conf

# Log all auth messages to auth.log
auth,authpriv.*    /var/log/auth.log

# Log mail messages
mail.*    /var/log/mail.log

# Log everything except auth and mail to syslog
*.*;auth,authpriv.none    /var/log/syslog

# Log kernel messages
kern.*    /var/log/kern.log

# Custom application log
local0.*    /var/log/myapp.log

# Forward logs to remote syslog server
*.* @@logserver.example.com:514   # TCP
*.* @logserver.example.com:514    # UDP

# After changes:
sudo systemctl restart rsyslog

logrotate Configuration

logrotate prevents log files from consuming all disk space:

# Global config: /etc/logrotate.conf
# App configs: /etc/logrotate.d/

# Example: /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

# Example: Custom application log
/var/log/myapp/*.log {
    weekly
    rotate 12
    compress
    dateext
    dateformat -%Y-%m-%d
    missingok
    notifempty
    create 0644 myapp myapp
}

# Test logrotate configuration
sudo logrotate -d /etc/logrotate.d/nginx   # Dry run
sudo logrotate -f /etc/logrotate.d/nginx   # Force rotate

Log Analysis Techniques

# Top 10 IP addresses in Nginx access log
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10

# Top requested URLs
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -10

# HTTP status code distribution
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

# Failed SSH login attempts
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn | head -10

# Requests per hour
awk -F'[' '{print $2}' /var/log/nginx/access.log | awk -F: '{print $1":"$2}' | sort | uniq -c

# Find large log files
find /var/log -type f -size +100M -exec ls -lh {} \;

# Disk usage by log directory
du -sh /var/log/*/ 2>/dev/null | sort -rh

Systemd Journal Persistence

# Make journal logs persistent across reboots
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

# Configure journal size limits (/etc/systemd/journald.conf)
# SystemMaxUse=500M
# RuntimeMaxUse=100M
# MaxRetentionSec=1month

# Restart journald
sudo systemctl restart systemd-journald

# Clean old journal entries
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=30d

# View journal disk usage
journalctl --disk-usage