What is Complete journalctl Guide: Linux System Log Management about?

Master journalctl commands for viewing, filtering, and managing systemd logs in Linux. Essential guide for system administrators and troubleshooting.

Who should read this article?

This article is perfect for technology professionals, developers, and anyone interested in system administration looking to enhance their skills and knowledge.

How long does it take to read?

This article takes approximately 11 minutes to read and contains 2135 words of expert insights and practical information.

What topics are covered?

This article covers key topics including: Linux, journalctl, logging, systemd, troubleshooting, providing comprehensive insights for technology professionals.

Complete journalctl Guide: Linux System...

Complete Guide to journalctl: Viewing and Managing System Logs in Linux

Introduction

journalctl is the primary command-line utility for querying and displaying logs from the systemd journal. It provides a centralized logging system that collects and manages log data from various sources including system services, kernel messages, and user applications. Unlike traditional syslog systems that store logs in plain text files, systemd's journal stores logs in a binary format that offers better performance, indexing, and metadata support.

The systemd journal captures structured logging data with rich metadata, making it easier to filter, search, and analyze system events. This comprehensive guide covers all aspects of using journalctl effectively for system administration and troubleshooting.

Basic Syntax and Structure

The basic syntax of journalctl follows this pattern:

`bash journalctl [OPTIONS...] [MATCHES...] `

Where: - OPTIONS are command-line flags that control output format, time ranges, and behavior - MATCHES are field-based filters to narrow down log entries

Core Commands and Options

Basic Viewing Commands

#### Display All Logs `bash journalctl ` Shows all available log entries from oldest to newest. This command displays the entire journal, which can be overwhelming on systems with extensive logging.

#### Display Recent Logs `bash journalctl -n 20 ` Shows the last 20 log entries. The -n option (or --lines) limits the number of displayed entries.

#### Follow Live Logs `bash journalctl -f ` Continuously displays new log entries as they are written to the journal, similar to tail -f. This is essential for real-time monitoring.

#### Reverse Order Display `bash journalctl -r ` Shows logs in reverse chronological order (newest first). Useful when you want to see the most recent events immediately.

Time-Based Filtering

#### Show Logs Since Specific Time `bash journalctl --since "2024-01-15 10:00:00" journalctl --since "1 hour ago" journalctl --since yesterday journalctl --since "2 days ago" `

#### Show Logs Until Specific Time `bash journalctl --until "2024-01-15 15:30:00" journalctl --until "30 minutes ago" `

#### Combine Since and Until `bash journalctl --since "2024-01-15 09:00:00" --until "2024-01-15 17:00:00" `

Time Format Examples

| Format | Example | Description | |--------|---------|-------------| | Absolute | 2024-01-15 14:30:00 | Full date and time | | Relative | 1 hour ago | Relative to current time | | Date only | 2024-01-15 | Entire day | | Keywords | yesterday, today | Common time references | | ISO format | 2024-01-15T14:30:00 | ISO 8601 format |

Service-Specific Logging

View Logs for Specific Service

`bash journalctl -u nginx.service journalctl -u ssh journalctl --unit=apache2.service `

#### Multiple Services `bash journalctl -u nginx.service -u apache2.service `

#### Service Logs with Time Filter `bash journalctl -u nginx.service --since "1 hour ago" `

System Boot Logs

#### Current Boot Logs `bash journalctl -b journalctl --boot `

#### Previous Boot Logs `bash journalctl -b -1 # Previous boot journalctl -b -2 # Two boots ago `

#### List Available Boots `bash journalctl --list-boots `

Example output: ` -2 caf0524a1d394ce0bdbcff75b94444fe Tue 2024-01-14 08:15:23 CET—Tue 2024-01-14 18:45:12 CET -1 4c8b1c4d2e3f4a5b6c7d8e9f0a1b2c3d Wed 2024-01-15 09:30:45 CET—Wed 2024-01-15 23:15:30 CET 0 1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d Thu 2024-01-16 08:00:00 CET—Thu 2024-01-16 23:59:59 CET `

Priority and Log Level Filtering

Priority Levels

| Priority | Numeric | Description | |----------|---------|-------------| | emerg | 0 | Emergency - system is unusable | | alert | 1 | Alert - action must be taken immediately | | crit | 2 | Critical - critical conditions | | err | 3 | Error - error conditions | | warning | 4 | Warning - warning conditions | | notice | 5 | Notice - normal but significant condition | | info | 6 | Informational - informational messages | | debug | 7 | Debug - debug-level messages |

Filter by Priority

`bash journalctl -p err # Show only errors and above journalctl -p warning # Show warnings and above journalctl --priority=crit # Show critical and above journalctl -p 0..3 # Show priorities 0 through 3 `

Advanced Filtering Options

Filter by Process ID

`bash journalctl _PID=1234 `

Filter by User ID

`bash journalctl _UID=1000 journalctl _UID=0 # Root user logs `

Filter by Executable

`bash journalctl _EXE=/usr/bin/ssh journalctl /usr/bin/nginx `

Filter by Command Name

`bash journalctl _COMM=sshd journalctl _COMM=systemd `

Kernel Messages

`bash journalctl -k journalctl --dmesg journalctl _TRANSPORT=kernel `

Output Formatting Options

JSON Output

`bash journalctl -o json # JSON format journalctl -o json-pretty # Pretty-printed JSON journalctl -o json-sse # JSON for server-sent events `

Other Output Formats

| Format | Command | Description | |--------|---------|-------------| | short | journalctl -o short | Default format (syslog-style) | | short-iso | journalctl -o short-iso | ISO 8601 timestamps | | short-precise | journalctl -o short-precise | Microsecond precision | | verbose | journalctl -o verbose | All available fields | | export | journalctl -o export | Binary export format | | cat | journalctl -o cat | Only message field |

Example Output Formats

#### Short Format (Default) ` Jan 16 10:30:15 server systemd[1]: Started Session 2 of user root. Jan 16 10:30:20 server sshd[1234]: Accepted publickey for user from 192.168.1.100 `

#### Verbose Format ` Tue 2024-01-16 10:30:15.123456 CET [s=1a2b3c4d...] _HOSTNAME=server _TRANSPORT=syslog PRIORITY=6 _UID=0 _GID=0 _COMM=systemd MESSAGE=Started Session 2 of user root. `

Field-Based Filtering

Available Fields

| Field | Description | Example | |-------|-------------|---------| | _SYSTEMD_UNIT | Systemd unit name | nginx.service | | _HOSTNAME | System hostname | webserver01 | | _TRANSPORT | Log transport method | syslog, kernel, journal | | _PID | Process ID | 1234 | | _UID | User ID | 0, 1000 | | _GID | Group ID | 0, 1000 | | _COMM | Command name | sshd, nginx | | _EXE | Executable path | /usr/sbin/sshd | | _CMDLINE | Complete command line | /usr/sbin/sshd -D | | SYSLOG_FACILITY | Syslog facility | 10 (security/authorization) | | SYSLOG_IDENTIFIER | Syslog identifier | sshd |

Complex Filtering Examples

#### Multiple Field Filters `bash journalctl _SYSTEMD_UNIT=sshd.service _UID=0 journalctl SYSLOG_IDENTIFIER=kernel PRIORITY=3 `

#### Combining with Time Filters `bash journalctl _SYSTEMD_UNIT=nginx.service --since "1 hour ago" -p warning `

Journal Management Commands

Disk Usage Information

`bash journalctl --disk-usage `

Example output: ` Archived and active journals take up 512.0M in the file system. `

Vacuum Operations

#### Clean by Size `bash sudo journalctl --vacuum-size=100M `

#### Clean by Time `bash sudo journalctl --vacuum-time=7d sudo journalctl --vacuum-time=1month `

#### Clean by Number of Files `bash sudo journalctl --vacuum-files=5 `

Verify Journal Integrity

`bash journalctl --verify `

Rotate Journals

`bash sudo systemctl kill --kill-who=main --signal=SIGUSR2 systemd-journald.service `

Configuration and Persistence

Journal Configuration File

The main configuration file is located at /etc/systemd/journald.conf.

#### Key Configuration Options

| Option | Values | Description | |--------|--------|-------------| | Storage | auto, persistent, volatile, none | Where to store journal data | | SystemMaxUse | Size (e.g., 100M, 1G) | Maximum disk space for system journal | | RuntimeMaxUse | Size | Maximum disk space for runtime journal | | MaxFileSec | Time (e.g., 1month, 1week) | Maximum time to store entries | | MaxRetentionSec | Time | Maximum time to retain entries |

#### Example Configuration `ini [Journal] Storage=persistent SystemMaxUse=500M RuntimeMaxUse=100M MaxFileSec=1month MaxRetentionSec=3month Compress=yes `

Making Journal Persistent

`bash sudo mkdir -p /var/log/journal sudo systemctl restart systemd-journald `

Practical Examples and Use Cases

System Troubleshooting

#### Failed Service Investigation `bash

Check service status and recent logs

systemctl status nginx.service journalctl -u nginx.service --since "10 minutes ago" -p err

Get detailed failure information

journalctl -u nginx.service -n 50 --no-pager `

#### Boot Issues Analysis `bash

Check last boot for errors

journalctl -b -p err

Compare with previous boot

journalctl -b -1 -p err

Check specific boot phase

journalctl -b _SYSTEMD_UNIT=systemd-logind.service `

#### Security Monitoring `bash

SSH login attempts

journalctl _SYSTEMD_UNIT=sshd.service --since "today"

Failed authentication attempts

journalctl _SYSTEMD_UNIT=sshd.service | grep "Failed password"

Sudo usage

journalctl _COMM=sudo --since "yesterday" `

Performance Monitoring

#### High CPU Usage Investigation `bash

Check system load messages

journalctl -p warning --since "1 hour ago" | grep -i "load"

Check for OOM killer activity

journalctl -k | grep -i "killed process" `

#### Disk Space Issues `bash

Check for disk space warnings

journalctl | grep -i "no space left"

Check filesystem errors

journalctl -k | grep -i "filesystem" `

Application Debugging

#### Web Server Logs `bash

Apache/Nginx error logs

journalctl -u apache2.service -p err --since "today"

Follow live web server logs

journalctl -u nginx.service -f

Filter by specific error types

journalctl -u nginx.service | grep "404\|500\|502" `

#### Database Logs `bash

MySQL/PostgreSQL logs

journalctl -u mysql.service --since "1 hour ago" journalctl -u postgresql.service -p warning `

Advanced Search Techniques

Regular Expression Filtering

`bash

Using grep with journalctl

journalctl -u sshd.service | grep -E "(Failed|Accepted)"

Case-insensitive search

journalctl | grep -i "error" `

Multiple Condition Searches

`bash

Combine multiple units and time ranges

journalctl -u nginx.service -u apache2.service --since "today" -p err

Complex field matching

journalctl _SYSTEMD_UNIT=sshd.service + _UID=0 `

Export and Analysis

#### Export to File `bash

Export specific time range

journalctl --since "2024-01-15" --until "2024-01-16" > system_logs.txt

Export in JSON format for analysis

journalctl -o json --since "today" > logs.json `

#### Export for External Tools `bash

Export for log analysis tools

journalctl -o export > journal_export.log `

Monitoring and Alerting Scripts

Basic Monitoring Script

`bash #!/bin/bash

monitor_errors.sh - Monitor for critical errors

LOGFILE="/var/log/error_monitor.log" THRESHOLD=5

Count critical errors in last hour

ERROR_COUNT=$(journalctl --since "1 hour ago" -p err --no-pager | wc -l)

if [ $ERROR_COUNT -gt $THRESHOLD ]; then echo "$(date): High error count detected: $ERROR_COUNT" >> $LOGFILE # Send alert (email, notification, etc.) fi `

Service Health Check

`bash #!/bin/bash

service_health.sh - Check service health via logs

SERVICE="nginx.service" ERRORS=$(journalctl -u $SERVICE --since "5 minutes ago" -p err --no-pager -q)

if [ -n "$ERRORS" ]; then echo "Service $SERVICE has errors:" echo "$ERRORS" exit 1 fi `

Performance Considerations

Optimizing Journal Performance

#### Journal Size Management `bash

Check current journal size

journalctl --disk-usage

Set size limits in journald.conf

SystemMaxUse=500M RuntimeMaxUse=100M `

#### Query Optimization Tips

| Technique | Command Example | Benefit | |-----------|----------------|---------| | Limit output | journalctl -n 100 | Faster display | | Use specific time ranges | journalctl --since "1 hour ago" | Reduced data processing | | Filter early | journalctl -u service.name -p err | Less data to process | | Use --no-pager | journalctl --no-pager | Direct output for scripts |

Troubleshooting Common Issues

Journal Not Persistent

`bash

Check if journal directory exists

ls -la /var/log/journal/

Create persistent journal directory

sudo mkdir -p /var/log/journal sudo chown root:systemd-journal /var/log/journal sudo chmod 2755 /var/log/journal sudo systemctl restart systemd-journald `

High Disk Usage

`bash

Check journal disk usage

journalctl --disk-usage

Clean old entries

sudo journalctl --vacuum-time=30d sudo journalctl --vacuum-size=500M `

Missing Logs

`bash

Check journald service status

systemctl status systemd-journald

Verify configuration

journalctl -u systemd-journald.service

Check for configuration errors

sudo journalctl --verify `

Integration with System Administration

Combining with Other Tools

#### Integration with systemctl `bash

Check service status and logs together

systemctl status nginx.service && journalctl -u nginx.service -n 20 `

#### Integration with monitoring tools `bash

Export metrics for monitoring

journalctl --since "1 hour ago" -p err --no-pager | wc -l `

Log Rotation and Archival

`bash

Manual rotation

sudo systemctl kill --kill-who=main --signal=SIGUSR2 systemd-journald.service

Automated cleanup script

#!/bin/bash journalctl --vacuum-time=30d journalctl --vacuum-size=500M `

Security Considerations

Access Control

Journal files are readable by users in the systemd-journal group: `bash

Add user to systemd-journal group

sudo usermod -a -G systemd-journal username

Check group membership

groups username `

Sensitive Information

`bash

Filter out sensitive information when sharing logs

journalctl -u service.name | sed 's/password=[^[:space:]]*/password=/g' `

Conclusion

journalctl is an essential tool for system administrators and developers working with systemd-based Linux distributions. Its powerful filtering capabilities, structured logging format, and integration with systemd services make it superior to traditional log file analysis methods. Understanding its various options and use cases enables effective system monitoring, troubleshooting, and maintenance.

The key to mastering journalctl lies in understanding its field-based filtering system, time-based queries, and output formatting options. Combined with proper journal configuration and maintenance practices, it provides a robust foundation for system logging and analysis.

Regular practice with different filtering combinations and integration into monitoring workflows will help administrators leverage the full potential of systemd's logging capabilities. Whether troubleshooting service failures, investigating security incidents, or monitoring system performance, journalctl provides the tools necessary for comprehensive log analysis and system administration.