Understanding System Logs in /var/log

Introduction to System Logging

System logs are essential files that record various system activities, errors, warnings, and informational messages on Linux and Unix-like operating systems. The /var/log directory serves as the central repository for these log files, providing administrators with crucial information for system monitoring, troubleshooting, and security auditing.

System logging follows a standardized approach using facilities and priorities to categorize messages. The logging system helps maintain system health, track user activities, monitor security events, and diagnose problems when they occur.

The /var/log Directory Structure

The /var/log directory contains numerous log files, each serving specific purposes. Understanding the structure and contents of these files is fundamental for effective system administration.

Common Log Files and Their Purposes

| Log File | Purpose | Typical Contents | |----------|---------|------------------| | /var/log/messages | General system messages | Kernel messages, system events, hardware issues | | /var/log/syslog | System-wide messages | All system messages (Debian/Ubuntu) | | /var/log/auth.log | Authentication events | Login attempts, sudo usage, SSH connections | | /var/log/secure | Security-related events | Authentication failures, security violations | | /var/log/kern.log | Kernel messages | Kernel errors, hardware detection, driver issues | | /var/log/boot.log | Boot process messages | System startup information and errors | | /var/log/dmesg | Kernel ring buffer | Hardware detection, driver loading | | /var/log/cron | Cron job execution | Scheduled task execution and errors | | /var/log/mail.log | Mail server activity | Email sending/receiving, mail server errors | | /var/log/apache2/ | Apache web server logs | HTTP requests, errors, access patterns | | /var/log/nginx/ | Nginx web server logs | Web server access and error logs | | /var/log/mysql/ | MySQL database logs | Database queries, errors, slow queries |

Distribution-Specific Variations

Different Linux distributions organize log files differently:

Red Hat/CentOS/Fedora: - Uses /var/log/messages for general system messages - Authentication logs in /var/log/secure - Mail logs in /var/log/maillog

Debian/Ubuntu: - Uses /var/log/syslog for general system messages - Authentication logs in /var/log/auth.log - Mail logs in /var/log/mail.log

Log File Formats and Structure

Most log files follow a standardized format containing timestamp, hostname, process name, process ID, and message content.

Standard Log Entry Format

` timestamp hostname process[PID]: message `

Example Log Entries

` Dec 15 10:30:45 server01 sshd[1234]: Accepted password for user from 192.168.1.100 port 22 ssh2 Dec 15 10:31:02 server01 sudo[1245]: user : TTY=pts/0 ; PWD=/home/user ; USER=root ; COMMAND=/bin/ls Dec 15 10:31:15 server01 kernel: [12345.678901] USB disconnect, address 1 `

Syslog Priority Levels

The syslog system uses priority levels to categorize message importance:

| Priority Level | Numerical Value | Description | Usage | |----------------|----------------|-------------|--------| | Emergency | 0 | System unusable | System panic conditions | | Alert | 1 | Action required immediately | Conditions requiring immediate attention | | Critical | 2 | Critical conditions | Hard device errors | | Error | 3 | Error conditions | Non-urgent failures | | Warning | 4 | Warning conditions | Warning messages | | Notice | 5 | Normal but significant | Normal but significant conditions | | Info | 6 | Informational messages | General information | | Debug | 7 | Debug-level messages | Debug information |

Syslog Facilities

Facilities categorize the source of log messages:

| Facility | Code | Description | |----------|------|-------------| | kern | 0 | Kernel messages | | user | 1 | User-level messages | | mail | 2 | Mail system | | daemon | 3 | System daemons | | auth | 4 | Security/authorization messages | | syslog | 5 | Messages generated by syslogd | | lpr | 6 | Line printer subsystem | | news | 7 | Network news subsystem | | uucp | 8 | UUCP subsystem | | cron | 9 | Clock daemon | | authpriv | 10 | Security/authorization messages | | ftp | 11 | FTP daemon | | local0-7 | 16-23 | Local use facilities |

Essential Commands for Log Analysis

Basic Log Viewing Commands

#### cat Command The cat command displays entire file contents:

`bash cat /var/log/messages `

Notes: Use with caution on large files as it displays everything at once.

#### less and more Commands These commands provide paginated viewing:

`bash less /var/log/messages more /var/log/syslog `

less Command Navigation: - Space: Next page - b: Previous page - /pattern: Search forward - ?pattern: Search backward - q: Quit

#### tail Command Displays the last lines of a file:

`bash

Show last 10 lines (default)

Show last 20 lines

Follow log file in real-time

Follow multiple files

Notes: The -f option is invaluable for monitoring active logs in real-time.

#### head Command Displays the first lines of a file:

`bash

Show first 10 lines (default)

Show first 20 lines

Advanced Log Analysis Commands

#### grep Command Search for specific patterns in log files:

`bash

Search for specific term

Case-insensitive search

Show line numbers

Search multiple files

Exclude lines containing pattern

Show context around matches

#### awk Command Process and extract specific fields:

`bash

Extract timestamp and message

Print lines containing specific pattern

Count occurrences of specific patterns

#### sed Command Stream editor for filtering and transforming text:

`bash

Extract lines between specific dates

Remove timestamp for cleaner output

#### sort and uniq Commands Organize and count log entries:

`bash

Sort log entries by timestamp

Count unique IP addresses in auth log

Find most common error messages

Specialized Log Analysis Tools

#### journalctl Command For systemd-based systems, journalctl provides advanced log querying:

`bash

View all journal entries

View entries for specific service

View entries since specific time

View entries for specific priority

Follow journal in real-time

View kernel messages

View logs for specific time range

#### dmesg Command Display kernel ring buffer messages:

`bash

Show all kernel messages

Show messages with timestamps

Filter by facility

Show only error and warning messages

Follow kernel messages

Log File Analysis Techniques

Time-Based Analysis

#### Filtering by Date and Time

`bash

Extract logs for specific date

Extract logs for specific hour

Extract logs for time range using awk

#### Creating Time-Based Reports

`bash

Count messages per hour

Daily message count

Error Analysis and Troubleshooting

#### Common Error Patterns

`bash

Find all error messages

Authentication failures

Disk errors

Network issues

#### Security Analysis

`bash

Failed login attempts

Successful logins

Sudo usage

Root login attempts

Performance Analysis

#### System Resource Monitoring

`bash

Memory-related messages

CPU-related messages

Service restart patterns

Log Rotation and Management

Understanding Log Rotation

Log rotation prevents log files from consuming excessive disk space by archiving old logs and creating new ones.

#### logrotate Configuration

The /etc/logrotate.conf file contains global settings:

`bash

View logrotate configuration

View service-specific configurations

#### Common logrotate Directives

| Directive | Purpose | Example | |-----------|---------|---------| | daily | Rotate logs daily | daily | | weekly | Rotate logs weekly | weekly | | monthly | Rotate logs monthly | monthly | | size | Rotate when size reached | size 100M | | rotate | Number of old logs to keep | rotate 4 | | compress | Compress rotated logs | compress | | delaycompress | Delay compression until next rotation | delaycompress | | missingok | Don't error if log file missing | missingok | | notifempty | Don't rotate empty files | notifempty |

#### Manual Log Rotation

`bash

Force log rotation

Test log rotation without executing

Rotate specific configuration

Log Cleanup Commands

`bash

Find large log files

Clear log file contents without deleting

Remove old compressed logs

Archive old logs

Monitoring and Alerting

Real-Time Log Monitoring

#### Using tail for Real-Time Monitoring

`bash

Monitor authentication attempts

Monitor system messages

Monitor multiple logs simultaneously

#### Creating Custom Monitoring Scripts

`bash #!/bin/bash

Simple log monitor script

LOGFILE="/var/log/messages" PATTERN="error"

tail -f $LOGFILE | while read line; do if echo "$line" | grep -qi "$PATTERN"; then echo "ALERT: Error detected at $(date)" echo "$line" # Send notification or take action fi done `

Log Analysis Scripts

#### Error Summary Script

`bash #!/bin/bash

Generate error summary report

LOGFILE="/var/log/messages" DATE=$(date +%Y-%m-%d)

echo "Error Summary Report for $DATE" echo "================================"

echo -e "\nError Count by Type:" grep -i "error\|fail\|critical" $LOGFILE | \ awk '{print $5}' | sort | uniq -c | sort -nr

echo -e "\nTop 10 Error Messages:" grep -i "error" $LOGFILE | \ awk '{for(i=6;i<=NF;i++) printf "%s ", $i; print ""}' | \ sort | uniq -c | sort -nr | head -10 `

#### Security Analysis Script

`bash #!/bin/bash

Security log analysis

AUTH_LOG="/var/log/auth.log"

echo "Security Analysis Report" echo "======================="

echo -e "\nFailed Login Attempts by IP:" grep "Failed password" $AUTH_LOG | \ awk '{print $11}' | sort | uniq -c | sort -nr

echo -e "\nSuccessful Logins:" grep "Accepted password" $AUTH_LOG | \ awk '{print $1, $2, $3, $9, $11}' | tail -10

echo -e "\nSudo Usage:" grep "sudo" $AUTH_LOG | \ awk '{print $1, $2, $3, $5}' | tail -10 `

Best Practices for Log Management

Security Considerations

1. Protect Log Files: Ensure appropriate permissions on log files `bash

Set proper permissions

2. Regular Monitoring: Implement automated monitoring for critical events 3. Centralized Logging: Consider using centralized logging solutions for multiple systems 4. Log Integrity: Implement log signing or checksums for critical systems

Performance Optimization

1. Regular Cleanup: Implement proper log rotation and cleanup procedures 2. Storage Management: Monitor disk usage in /var/log 3. Selective Logging: Configure appropriate log levels to avoid excessive logging 4. Compression: Use compression for archived logs to save space

Documentation and Procedures

1. Document Log Locations: Maintain documentation of custom log locations 2. Standardize Analysis: Create standard procedures for common log analysis tasks 3. Emergency Procedures: Establish procedures for log analysis during incidents 4. Regular Reviews: Schedule regular log reviews for security and performance

Troubleshooting Common Issues

Log File Issues

#### Permission Problems `bash

Check log file permissions

Fix common permission issues

#### Disk Space Issues `bash

Check disk usage

Find largest log files

Emergency log cleanup

#### Missing Log Files `bash

Restart logging service

Check service status

Analysis Challenges

#### Large File Handling `bash

Use split to handle large files

Use zcat for compressed files

#### Complex Pattern Matching `bash

Use extended regular expressions

Multiple pattern search

This comprehensive guide provides the foundation for understanding and effectively managing system logs in the /var/log directory. Regular practice with these commands and techniques will develop proficiency in system log analysis, which is essential for maintaining system health, security, and performance.