Monitoring I/O with iostat: Complete Guide

Introduction

The iostat command is a powerful system administration tool used to monitor input/output (I/O) device loading and CPU utilization on Unix-like systems. It provides detailed statistics about disk activity, CPU usage, and system performance, making it an essential utility for system administrators, performance analysts, and developers who need to diagnose and optimize system performance.

What is iostat

iostat is part of the sysstat package, a collection of performance monitoring tools for Linux systems. The tool generates reports showing CPU statistics and I/O statistics for devices and partitions. It helps identify bottlenecks in storage subsystems and provides insights into system performance characteristics.

The command works by reading kernel statistics from /proc/stat for CPU information and /proc/diskstats for disk I/O information. It can display both current statistics and historical data over specified time intervals.

Installation

Linux Distributions

#### Ubuntu/Debian `bash sudo apt update sudo apt install sysstat `

#### CentOS/RHEL/Fedora `bash

For CentOS/RHEL 7 and earlier

For CentOS/RHEL 8+ and Fedora

#### Arch Linux `bash sudo pacman -S sysstat `

Verification

Basic Syntax and Options

Command Structure

Core Parameters

| Parameter | Description | Example | |-----------|-------------|---------| | interval | Time between reports (seconds) | iostat 2 | | count | Number of reports to generate | iostat 2 5 | | options | Various flags to modify output | iostat -x |

Essential Options

| Option | Description | Use Case | |--------|-------------|----------| | -c | Display CPU utilization only | CPU performance analysis | | -d | Display device utilization only | Disk I/O analysis | | -x | Extended statistics | Detailed performance metrics | | -k | Display statistics in kilobytes | Memory-friendly units | | -m | Display statistics in megabytes | Large system analysis | | -h | Human-readable format | Easy interpretation | | -t | Include timestamp | Historical analysis | | -p | Display statistics for partitions | Partition-level monitoring | | -N | Display registered device mapper names | LVM/device mapper systems | | -z | Omit output for inactive devices | Clean output |

Understanding iostat Output

Default Output Format

When executed without options, iostat displays both CPU and disk statistics:

`bash iostat `

#### CPU Statistics Section ` avg-cpu: %user %nice %system %iowait %steal %idle 2.45 0.01 1.23 0.87 0.00 95.44 `

| Metric | Description | Normal Range | |--------|-------------|--------------| | %user | CPU time spent in user mode | 0-100% | | %nice | CPU time spent on low-priority processes | 0-5% typically | | %system | CPU time spent in kernel mode | 0-30% typically | | %iowait | CPU time waiting for I/O operations | 0-20% acceptable | | %steal | CPU time stolen by hypervisor | 0% on physical systems | | %idle | CPU idle time | Higher is better |

#### Device Statistics Section ` Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 5.23 12.45 45.67 123456 456789 sdb 2.10 8.90 23.45 89012 234567 `

| Metric | Description | Significance | |--------|-------------|--------------| | tps | Transfers per second | I/O request rate | | kB_read/s | Kilobytes read per second | Read throughput | | kB_wrtn/s | Kilobytes written per second | Write throughput | | kB_read | Total kilobytes read | Cumulative read data | | kB_wrtn | Total kilobytes written | Cumulative write data |

Extended Statistics (-x option)

The extended format provides comprehensive I/O metrics:

`bash iostat -x `

#### Extended Output Fields

| Field | Description | Optimal Value | Warning Threshold | |-------|-------------|---------------|-------------------| | r/s | Read requests per second | Workload dependent | N/A | | w/s | Write requests per second | Workload dependent | N/A | | rkB/s | Kilobytes read per second | Higher for read-heavy | N/A | | wkB/s | Kilobytes written per second | Higher for write-heavy | N/A | | rrqm/s | Read requests merged per second | Higher is better | N/A | | wrqm/s | Write requests merged per second | Higher is better | N/A | | %rrqm | Percentage of read requests merged | >50% good | <10% poor | | %wrqm | Percentage of write requests merged | >50% good | <10% poor | | r_await | Average read request wait time (ms) | <10ms ideal | >20ms concerning | | w_await | Average write request wait time (ms) | <10ms ideal | >20ms concerning | | aqu-sz | Average queue size | <2 good | >10 problematic | | rareq-sz | Average read request size (KB) | Larger is better | <4KB small | | wareq-sz | Average write request size (KB) | Larger is better | <4KB small | | svctm | Average service time (deprecated) | N/A | N/A | | %util | Device utilization percentage | <80% good | >95% saturated |

Practical Examples

Basic Monitoring

#### Single Report `bash iostat ` Displays current CPU and I/O statistics since system boot.

#### Continuous Monitoring `bash iostat 2 ` Updates statistics every 2 seconds continuously.

#### Limited Reports `bash iostat 2 10 ` Generates 10 reports with 2-second intervals.

CPU-Focused Analysis

#### CPU-Only Statistics `bash iostat -c 1 5 ` Shows only CPU utilization for 5 intervals of 1 second each.

#### CPU with Timestamps `bash iostat -c -t 5 ` Displays CPU statistics with timestamps every 5 seconds.

Disk I/O Analysis

#### Disk-Only Statistics `bash iostat -d 2 ` Shows only disk I/O statistics updated every 2 seconds.

#### Extended Disk Statistics `bash iostat -dx 3 ` Provides detailed disk metrics every 3 seconds.

#### Specific Device Monitoring `bash iostat -dx sda 2 5 ` Monitors only the sda device with extended statistics.

#### Human-Readable Format `bash iostat -dxh 1 ` Displays disk statistics in human-readable format.

Advanced Monitoring Scenarios

#### Partition-Level Analysis `bash iostat -p sda 2 ` Shows statistics for all partitions on sda device.

#### Network File System Monitoring `bash iostat -n 2 ` Includes network filesystem statistics.

#### Device Mapper Monitoring `bash iostat -N 2 ` Uses device mapper names instead of device names.

#### Clean Output (Active Devices Only) `bash iostat -dx -z 2 ` Omits devices with zero activity for cleaner output.

Performance Analysis and Interpretation

CPU Performance Indicators

#### High %iowait Analysis `bash iostat -c 1 `

When %iowait is consistently above 20%: - Storage subsystem may be bottlenecked - Applications are waiting for disk I/O - Consider faster storage or I/O optimization

#### CPU Utilization Patterns `bash iostat -c -t 2 30 `

Monitor for 1 minute to identify: - Peak usage periods - System vs user CPU consumption - I/O wait patterns

Disk Performance Analysis

#### Identifying Bottlenecks `bash iostat -dx 1 `

Key indicators of disk bottlenecks:

| Metric | Bottleneck Indicator | Action Required | |--------|---------------------|-----------------| | %util | >95% consistently | Add storage capacity | | await | >20ms average | Optimize I/O patterns | | aqu-sz | >10 requests | Reduce concurrent I/O | | svctm | Increasing trend | Hardware investigation |

#### Request Size Analysis `bash iostat -dx -k 2 `

Small request sizes indicate: - Random I/O patterns - Potential for optimization - Need for I/O coalescing

Large request sizes suggest: - Sequential I/O patterns - Efficient data access - Good application design

Workload Characterization

#### Read vs Write Analysis `bash iostat -dx 5 12 `

Monitor for 1 minute to determine: - Read/write ratio - Peak activity periods - Access pattern consistency

#### Sequential vs Random I/O `bash iostat -dx -t 1 60 `

Indicators of access patterns:

| Pattern | Characteristics | Optimization | |---------|----------------|--------------| | Sequential | Large request sizes, high throughput | Prefetching, larger buffers | | Random | Small request sizes, high IOPS | SSD storage, caching | | Mixed | Variable metrics | Workload separation |

Troubleshooting Common Issues

High I/O Wait Times

#### Diagnosis `bash iostat -dx 1 10 `

Look for: - High await values (>20ms) - High %util (>90%) - Large queue sizes (aqu-sz >5)

#### Solutions 1. Identify processes causing high I/O 2. Optimize application I/O patterns 3. Consider storage upgrades 4. Implement caching strategies

Low Throughput Issues

#### Analysis Command `bash iostat -dxk -t 2 30 `

Check for: - Small request sizes (<64KB) - Low merge rates (%rrqm, %wrqm <10%) - High request rates with low throughput

#### Optimization Strategies 1. Increase application buffer sizes 2. Enable I/O scheduling optimizations 3. Tune filesystem parameters 4. Consider RAID configurations

Storage Capacity Planning

#### Trend Analysis `bash iostat -dx -t 300 288 `

Run for 24 hours with 5-minute intervals to: - Identify peak usage periods - Calculate growth trends - Plan capacity requirements - Optimize maintenance windows

Integration with Other Tools

Combining with sar

Terminal 1

Terminal 2

Correlate I/O and CPU statistics for comprehensive analysis.

Using with iotop

Terminal 1

Terminal 2

Identify specific processes causing I/O activity.

Integration with vmstat

Combined system overview

Monitor memory, CPU, and I/O simultaneously.

Scripting and Automation

Basic Monitoring Script

Basic I/O monitoring script

LOGFILE="/var/log/iostat_monitor.log" INTERVAL=60 COUNT=1440 # 24 hours of minutes

echo "Starting I/O monitoring at $(date)" >> $LOGFILE iostat -dx -t $INTERVAL $COUNT >> $LOGFILE 2>&1 `

Alert Script

I/O performance alert script

THRESHOLD_UTIL=90 THRESHOLD_AWAIT=20

while true; do OUTPUT=$(iostat -dx 1 2 | tail -n +4 | tail -n +7) echo "$OUTPUT" | while read line; do if [[ $line =~ ^[a-z] ]]; then UTIL=$(echo $line | awk '{print $NF}' | cut -d. -f1) AWAIT=$(echo $line | awk '{print $(NF-3)}' | cut -d. -f1) DEVICE=$(echo $line | awk '{print $1}') if [ "$UTIL" -gt "$THRESHOLD_UTIL" ]; then echo "ALERT: High utilization on $DEVICE: ${UTIL}%" fi if [ "$AWAIT" -gt "$THRESHOLD_AWAIT" ]; then echo "ALERT: High await time on $DEVICE: ${AWAIT}ms" fi fi done sleep 60 done `

Performance Report Generator

Generate daily I/O performance report

DATE=$(date +%Y%m%d) REPORT_FILE="/tmp/io_report_$DATE.txt"

echo "Daily I/O Performance Report - $DATE" > $REPORT_FILE echo "=======================================" >> $REPORT_FILE echo "" >> $REPORT_FILE

echo "Current I/O Statistics:" >> $REPORT_FILE iostat -dx >> $REPORT_FILE

echo "" >> $REPORT_FILE echo "CPU Utilization:" >> $REPORT_FILE iostat -c >> $REPORT_FILE

echo "" >> $REPORT_FILE echo "Top 5 Busiest Devices:" >> $REPORT_FILE iostat -dx | tail -n +8 | sort -k10 -nr | head -5 >> $REPORT_FILE `

Best Practices

Monitoring Guidelines

1. Baseline Establishment - Run iostat during normal operations - Document typical performance ranges - Identify daily/weekly patterns

2. Alert Thresholds - %util > 80% for sustained periods - await > 20ms average - %iowait > 25% CPU time

3. Data Collection - Use appropriate intervals (1-5 seconds for troubleshooting) - Collect data during peak hours - Maintain historical records

Performance Optimization

1. I/O Scheduling `bash # Check current scheduler cat /sys/block/sda/queue/scheduler # Set deadline scheduler for databases echo deadline > /sys/block/sda/queue/scheduler `

2. Queue Depth Tuning `bash # Check current queue depth cat /sys/block/sda/queue/nr_requests # Increase for high-throughput workloads echo 256 > /sys/block/sda/queue/nr_requests `

3. Read-ahead Optimization `bash # Check current read-ahead blockdev --getra /dev/sda # Set optimal read-ahead for sequential workloads blockdev --setra 4096 /dev/sda `

Conclusion

The iostat command is an indispensable tool for system administrators and performance engineers. Its comprehensive metrics provide deep insights into system I/O behavior and CPU utilization patterns. Regular monitoring with iostat helps identify performance bottlenecks, plan capacity requirements, and optimize system configurations.

Understanding the various output fields and their implications enables effective troubleshooting and performance tuning. Combined with other system monitoring tools, iostat forms part of a complete performance analysis toolkit essential for maintaining optimal system performance.

Proper interpretation of iostat output requires understanding both the technical metrics and the specific workload characteristics of your systems. Regular monitoring, baseline establishment, and proactive analysis help prevent performance issues and ensure optimal system operation.