vmstat Command: Comprehensive System Performance Monitoring
Table of Contents
1. [Introduction](#introduction) 2. [Syntax and Basic Usage](#syntax-and-basic-usage) 3. [Command Options](#command-options) 4. [Output Fields Explanation](#output-fields-explanation) 5. [Practical Examples](#practical-examples) 6. [Performance Analysis](#performance-analysis) 7. [Advanced Usage](#advanced-usage) 8. [Troubleshooting Common Issues](#troubleshooting-common-issues) 9. [Best Practices](#best-practices) 10. [Related Commands](#related-commands)Introduction
The vmstat (Virtual Memory Statistics) command is a powerful system monitoring tool available on Unix-like operating systems that provides detailed information about system performance metrics. It displays statistics about processes, memory, paging, block I/O, traps, disks, and CPU activity. This command is essential for system administrators and developers who need to monitor system performance, identify bottlenecks, and troubleshoot performance issues.
The vmstat command reports information about kernel threads, virtual memory, disks, traps, and CPU activity. It helps in understanding how system resources are being utilized and can reveal performance problems such as memory shortages, excessive disk I/O, or CPU bottlenecks.
Syntax and Basic Usage
Basic Syntax
Parameters Explanation
| Parameter | Description | |-----------|-------------| | delay | Time interval between updates in seconds | | count | Number of updates to display before exiting | | options | Various flags to modify output format |
Simple Usage Examples
Display current system statistics once
vmstatDisplay statistics every 2 seconds
vmstat 2Display statistics every 5 seconds for 10 iterations
vmstat 5 10Command Options
Primary Options Table
| Option | Long Form | Description | |--------|-----------|-------------| | -a | --active | Display active and inactive memory | | -f | --forks | Display number of forks since boot | | -m | --slabs | Display slabinfo | | -n | --one-header | Display header only once | | -s | --stats | Display various event counters and memory statistics | | -d | --disk | Display disk statistics | | -D | --disk-sum | Display disk statistics summary | | -p | --partition | Display partition specific statistics | | -S | --unit | Define display unit (k, K, m, M) | | -t | --timestamp | Add timestamp to output | | -w | --wide | Wide output format | | -V | --version | Display version information | | -h | --help | Display help information |
Unit Specification Options
| Unit | Description | Multiplier | |------|-------------|------------| | k | kilobytes | 1024 bytes | | K | kilobytes | 1000 bytes | | m | megabytes | 1024^2 bytes | | M | megabytes | 1000^2 bytes |
Output Fields Explanation
Standard vmstat Output Fields
When you run vmstat without options, it displays output in several columns grouped by category:
#### Process Fields (procs)
| Field | Description | |-------|-------------| | r | Number of runnable processes waiting for run time | | b | Number of processes in uninterruptible sleep |
#### Memory Fields (memory)
| Field | Description | Unit | |-------|-------------|------| | swpd | Virtual memory used | KB | | free | Idle memory | KB | | buff | Memory used as buffers | KB | | cache | Memory used as cache | KB | | inact | Inactive memory (-a option) | KB | | active | Active memory (-a option) | KB |
#### Swap Fields (swap)
| Field | Description | Unit | |-------|-------------|------| | si | Memory swapped in from disk | KB/s | | so | Memory swapped out to disk | KB/s |
#### I/O Fields (io)
| Field | Description | Unit | |-------|-------------|------| | bi | Blocks received from block device | blocks/s | | bo | Blocks sent to block device | blocks/s |
#### System Fields (system)
| Field | Description | Unit | |-------|-------------|------| | in | Number of interrupts per second | interrupts/s | | cs | Number of context switches per second | switches/s |
#### CPU Fields (cpu)
| Field | Description | Unit | |-------|-------------|------| | us | Time spent running non-kernel code | percentage | | sy | Time spent running kernel code | percentage | | id | Time spent idle | percentage | | wa | Time spent waiting for I/O | percentage | | st | Time stolen from virtual machine | percentage |
Practical Examples
Example 1: Basic System Monitoring
Display current system statistics
vmstatSample output:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 1234567 89012 345678 0 0 12 34 567 890 5 2 92 1 0Example 2: Continuous Monitoring
Monitor system every 3 seconds
vmstat 3Monitor system every 5 seconds for 12 iterations
vmstat 5 12Example 3: Memory Analysis with Active/Inactive
Display active and inactive memory
vmstat -aSample output with active/inactive memory:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free inact active si so bi bo in cs us sy id wa st 0 0 0 987654 123456 234567 0 0 8 16 234 456 3 1 95 1 0Example 4: Disk Statistics
Display disk statistics
vmstat -dSample output:
disk- ------------reads------------ ------------writes----------- -----IO------ total merged sectors ms total merged sectors ms cur sec sda 123456 7890 9876543 234567 345678 1234 5678901 456789 0 123 sdb 45678 2345 3456789 89012 123456 567 2345678 123456 0 45Example 5: System Statistics Summary
Display comprehensive system statistics
vmstat -sSample output:
8388608 K total memory 1234567 K used memory 987654 K active memory 456789 K inactive memory 2345678 K free memory 123456 K buffer memory 3456789 K swap cache 4194304 K total swap 0 K used swap 4194304 K free swap 12345 non-nice user cpu ticks 6789 nice user cpu ticks 2345 system cpu ticks 987654 idle cpu ticks 1234 IO-wait cpu ticks 56 IRQ cpu ticks 78 softirq cpu ticks 0 stolen cpu ticks 123456 pages paged in 67890 pages paged out 0 pages swapped in 0 pages swapped out 1234567 interrupts 2345678 CPU context switches 1234567890 boot time 12345 forksExample 6: Partition-Specific Monitoring
Monitor specific partition
vmstat -p /dev/sda1Display partition statistics
reads read sectors writes requested writes
12345 6789012 23456 4567890Example 7: Fork Statistics
Display fork statistics since boot
vmstat -fSample output:
123456 forksExample 8: Slab Information
Display kernel slab allocator information
vmstat -mSample output (truncated):
Cache Num Total Size Pages ext4_inode_cache 1234 2345 896 17 dentry 5678 7890 192 20 buffer_head 2345 3456 104 39Performance Analysis
Interpreting vmstat Output for Performance Issues
#### CPU Performance Analysis
| Scenario | Indicators | Interpretation | |----------|------------|----------------| | CPU Bound | High 'us' + 'sy', Low 'id' | System is CPU constrained | | I/O Wait | High 'wa', Processes in 'b' | Disk I/O bottleneck | | System Calls | High 'sy' | Kernel spending too much time | | Context Switching | High 'cs' | Too many process switches |
#### Memory Performance Analysis
| Scenario | Indicators | Action Required | |----------|------------|-----------------| | Memory Shortage | Low 'free', High 'swpd' | Add more RAM | | Swap Activity | Non-zero 'si' and 'so' | Memory pressure | | Buffer Usage | High 'buff' | Good disk caching | | Cache Usage | High 'cache' | Good file system caching |
#### I/O Performance Analysis
| Metric | Normal Range | Problem Indicators | |--------|--------------|-------------------| | bi (blocks in) | Varies | Consistently high values | | bo (blocks out) | Varies | Sudden spikes | | Processes in 'b' | 0-2 | Consistently > 5 |
Performance Thresholds
#### CPU Utilization Guidelines
| Metric | Good | Warning | Critical | |--------|------|---------|----------| | CPU Idle (id) | > 70% | 30-70% | < 30% | | I/O Wait (wa) | < 10% | 10-30% | > 30% | | System (sy) | < 20% | 20-40% | > 40% |
#### Memory Usage Guidelines
| Metric | Good | Warning | Critical | |--------|------|---------|----------| | Free Memory | > 20% | 5-20% | < 5% | | Swap Usage | 0% | 1-10% | > 10% | | Swap Activity | 0 | Occasional | Constant |
Advanced Usage
Combining vmstat with Other Tools
#### Using vmstat with awk for Custom Analysis
`bash
Monitor only high CPU usage
vmstat 1 | awk '$13 + $14 > 80 {print strftime("%Y-%m-%d %H:%M:%S"), $0}'Monitor high memory usage
vmstat 1 | awk '$4 < 100000 {print "Low memory alert:", $0}'Monitor I/O wait issues
vmstat 1 | awk '$16 > 10 {print "High I/O wait:", $0}'#### Creating Performance Logs
`bash
Log vmstat output with timestamps
vmstat -t 5 >> /var/log/vmstat.logCreate rotating performance logs
vmstat 60 1440 > vmstat_$(date +%Y%m%d).log &Scripting with vmstat
#### Performance Alert Script
`bash
#!/bin/bash
performance_monitor.sh
THRESHOLD_CPU=80 THRESHOLD_MEM=90 THRESHOLD_SWAP=10
while true; do
vmstat 1 2 | tail -1 | while read line; do
cpu_usage=$(echo $line | awk '{print 100-$15}')
swap_used=$(echo $line | awk '{print $3}')
if [ $cpu_usage -gt $THRESHOLD_CPU ]; then
echo "ALERT: High CPU usage: ${cpu_usage}%"
fi
if [ $swap_used -gt $THRESHOLD_SWAP ]; then
echo "ALERT: Swap usage: ${swap_used}KB"
fi
done
sleep 60
done
`
Integration with Monitoring Systems
#### Nagios Integration
`bash
vmstat check for Nagios
#!/bin/bash WARN_CPU=80 CRIT_CPU=90CPU_IDLE=$(vmstat 1 2 | tail -1 | awk '{print $15}') CPU_USED=$((100 - CPU_IDLE))
if [ $CPU_USED -gt $CRIT_CPU ]; then
echo "CRITICAL - CPU usage ${CPU_USED}%"
exit 2
elif [ $CPU_USED -gt $WARN_CPU ]; then
echo "WARNING - CPU usage ${CPU_USED}%"
exit 1
else
echo "OK - CPU usage ${CPU_USED}%"
exit 0
fi
`
Troubleshooting Common Issues
High Load Average Investigation
When system load is high, use vmstat to identify the cause:
`bash
Check if processes are waiting for CPU (high 'r' column)
vmstat 1If 'r' > number of CPUs: CPU bottleneck
If 'b' is high: I/O bottleneck
If 'wa' is high: Disk I/O issues
Memory Issues Diagnosis
`bash
Check memory pressure
vmstat -s | grep -E "(memory|swap)"Monitor swap activity
vmstat 1 | grep -v "swap" | awk '$7 > 0 || $8 > 0 {print "Swap activity detected"}'I/O Performance Problems
`bash
Monitor disk I/O patterns
vmstat -dCheck for I/O bottlenecks
vmstat 1 | awk '$16 > 20 {print "High I/O wait:", $16"%"}'Best Practices
Monitoring Guidelines
1. Baseline Establishment: Run vmstat during normal operations to establish baseline metrics 2. Regular Intervals: Use consistent monitoring intervals (1, 5, or 60 seconds) 3. Historical Data: Maintain logs for trend analysis 4. Alert Thresholds: Set appropriate thresholds based on your system's baseline
Performance Optimization Tips
#### CPU Optimization - Monitor 'us' and 'sy' columns for CPU-bound processes - High 'cs' indicates excessive context switching - Use 'r' column to identify CPU queue length
#### Memory Optimization - Keep swap usage minimal - Monitor buffer and cache usage for efficiency - Watch for memory leaks through increasing 'swpd'
#### I/O Optimization - Monitor 'bi' and 'bo' for disk throughput - High 'wa' indicates I/O bottlenecks - Use 'b' column to identify blocked processes
Automation and Alerting
#### Automated Monitoring Script
`bash
#!/bin/bash
system_health_check.sh
LOG_FILE="/var/log/system_health.log" DATE=$(date '+%Y-%m-%d %H:%M:%S')
Get vmstat data
VMSTAT_OUTPUT=$(vmstat 1 3 | tail -1)Extract key metrics
CPU_IDLE=$(echo $VMSTAT_OUTPUT | awk '{print $15}') CPU_IOWAIT=$(echo $VMSTAT_OUTPUT | awk '{print $16}') FREE_MEM=$(echo $VMSTAT_OUTPUT | awk '{print $4}') SWAP_USED=$(echo $VMSTAT_OUTPUT | awk '{print $3}')Log the data
echo "$DATE CPU_IDLE:$CPU_IDLE CPU_IOWAIT:$CPU_IOWAIT FREE_MEM:$FREE_MEM SWAP_USED:$SWAP_USED" >> $LOG_FILESend alerts if thresholds exceeded
if [ $CPU_IDLE -lt 10 ]; then echo "$DATE ALERT: Low CPU idle: $CPU_IDLE%" | mail -s "CPU Alert" admin@domain.com fiif [ $CPU_IOWAIT -gt 30 ]; then
echo "$DATE ALERT: High I/O wait: $CPU_IOWAIT%" | mail -s "I/O Alert" admin@domain.com
fi
`
Related Commands
Complementary System Monitoring Tools
| Command | Purpose | Relationship to vmstat | |---------|---------|----------------------| | top | Real-time process viewer | Shows process-level details | | htop | Enhanced process viewer | More user-friendly than top | | iostat | I/O statistics | Detailed disk I/O analysis | | sar | System activity reporter | Historical performance data | | free | Memory usage | Detailed memory information | | ps | Process status | Current process information | | netstat | Network statistics | Network connection details | | mpstat | CPU statistics | Per-CPU performance metrics |
Command Combinations for Comprehensive Monitoring
`bash
Complete system overview
vmstat 1 5 && iostat -x 1 5 && free -hMemory and swap analysis
vmstat -s && free -h && cat /proc/meminfo | head -20CPU and load analysis
vmstat 1 10 && uptime && cat /proc/loadavgThe vmstat command is an indispensable tool for system administrators and performance analysts. Its ability to provide real-time insights into system resource utilization makes it essential for maintaining optimal system performance, identifying bottlenecks, and troubleshooting performance issues. Regular monitoring with vmstat, combined with proper interpretation of its output, enables proactive system management and helps prevent performance problems before they impact users.