Linux Performance Troubleshooting with vmstat and mpstat

The load average on your 4-core server is sitting at 8, applications feel sluggish, and users are complaining. You SSH in and need to know fast: are you CPU-bound, memory-starved, or stuck waiting on I/O? Linux performance troubleshooting starts with vmstat and mpstat because they show the whole system in real time without installing agents or parsing gigabytes of logs. Both tools are already on your server, both update every second, and both give you the high-level view you need to decide where to dig next.

Most guides list the columns and move on. This one shows you what the output actually looks like when something is broken: the exact numbers that mean you're thrashing in swap, the patterns that indicate a context switch storm, the CPU percentages that tell you whether your problem is in userspace or the kernel. You'll see side-by-side comparisons of normal and problem output so you know what to look for when you're under pressure.

When to Use vmstat vs. mpstat

vmstat is your first stop. It shows system-wide CPU, memory, swap, I/O and context switches in a single line that updates every second. Run vmstat 1 and you get a snapshot of the entire machine: runnable processes, swap activity, CPU breakdown, all in one place. If the problem is obvious (swap thrashing, CPU pegged, I/O wait dominating) you'll see it immediately.

mpstat breaks down CPU time per core. When vmstat shows high CPU utilization but you need to know whether all cores are busy or one is pegged at 100% while the rest idle, run mpstat -P ALL 1. This matters for single-threaded applications that can't use more than one core, or for interrupt storms hitting a single CPU. Basic invocation: mpstat 1 for aggregate stats across all cores, mpstat -P ALL 1 to see each core separately.

vmstat ships with the procps or procps-ng package, installed by default on most distributions. mpstat is part of sysstat, which you may need to install (apt install sysstat or yum install sysstat). When you need per-process detail (which process is burning CPU or leaking memory) you move to top or pidstat, but that's a second step. Start with the system view. For more on interpreting system-wide metrics, see Linux Load Average Explained: CPU Pressure, Runnable Processes, and Bottlenecks.

Reading vmstat Output: Column by Column

Run vmstat 1 and you get output like this:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 1523840 245120 3421568  0    0    12    45  320  580 15  3 82  0  0
 1  0      0 1521920 245120 3421824  0    0     0    16  415  692 18  4 78  0  0
 3  0      0 1519232 245120 3422336  0    0     0     0  398  671 16  3 81  0  0
12  1   4096 245632 245120 3398144  0  512   128   520 1823 8934 78 18  0  4  0
11  2  18432 198400 245120 3385216 256 1024   64   1028 1654 9245 74 20  0  6  0

The first line after the header is a summary since boot; ignore it. Watch the lines that follow. Each updates every second (the 1 in vmstat 1).

Memory Columns: swpd, free, buff, cache

swpd is how much memory has been swapped to disk, in kilobytes. Zero is good. Any non-zero value is a warning sign if it's growing under load. In the example above, swpd jumps from 0 to 4096 to 18432 in two seconds: the system is running out of RAM and swapping actively.

free is unused RAM. Low free memory is normal on Linux because the kernel uses spare RAM for buff (buffer cache, for block device metadata) and cache (page cache, for file contents). High buff/cache is healthy; the kernel will reclaim it instantly if an application needs it. But if free drops to tens of megabytes and swpd is growing, you're out of memory.

Swap Activity: si and so

si (swap in) and so (swap out) are in kilobytes per second. Any non-zero so under load means the kernel is evicting pages to disk because RAM is full. In the example, so is 512 then 1024 KB/s: pages are being written to swap. This kills performance. Swapping in (si) happens when the kernel tries to bring those pages back; 256 KB/s swap-in means the application is thrashing. If you see sustained swap activity, you need more RAM or you need to kill memory-hungry processes.

CPU Columns: us, sy, id, wa, st

us is user time (applications), sy is system time (kernel), id is idle, wa is I/O wait, st is steal time (VMs only). These are percentages. In the problem lines above, us is 78% and 74%, sy is 18% and 20%, id is 0: no idle time. This is a CPU bottleneck combined with memory pressure (the swap activity). If wa were high (say 40%) and us + sy low, the problem would be I/O, not CPU. st above 5% means your VM host is overcommitted and starving your guest of CPU cycles; complain to your provider or move to a less crowded host.

Processes and Context Switches

r is the number of runnable processes waiting for CPU time. On a 4-core server, r of 12 means 8 processes are queued. Compare r to your core count; consistently higher means CPU saturation. b is processes blocked waiting for I/O (disk, NFS). cs is context switches per second. Normal is a few hundred to a few thousand. In the example, cs jumps to 8934 and 9245: this can indicate too many threads or processes competing for resources, or rapid process creation and teardown. in is interrupts per second; spikes here can mean network packet floods or misbehaving hardware.

For detailed documentation on every field, see the vmstat man page.

Diagnosing CPU Bottlenecks with vmstat

A CPU bottleneck looks like this: us at 80-90%, id near 0, r greater than the number of cores. Here's normal output on a 4-core server with light load:

procs ------cpu-----
 r  b us sy id wa st
 1  0 22  5 73  0  0
 2  0 19  4 77  0  0

And here's the same server CPU-bound:

procs ------cpu-----
 r  b us sy id wa st
12  0 85 10  5  0  0
11  0 87  9  4  0  0

r at 12 on a 4-core machine means 8 processes are queued waiting for CPU. us at 85-87% means applications are burning cycles. id at 4-5% means the CPUs are almost never idle. This is a CPU bottleneck. Your next step is top or pidstat 1 to identify which process is responsible: database query, web server threads, a runaway build job. For a deeper dive into process-level CPU monitoring, see Linux Process Resource Monitoring: CPU, Memory, and Load Analysis.

High User Time: Application or Database Load

When us dominates (70%+) and sy is low, an application is doing heavy computation. Common causes: a database running a full table scan, a web application executing expensive business logic, image or video processing, compilation. If r is much higher than your core count, you have more work than CPU capacity. Solutions: optimize the slow code, add more cores, or offload work to a queue.

High System Time: Kernel Overhead or Network Stack

sy above 30% means the kernel is busy. Common causes: high network throughput (the kernel processes every packet), heavy filesystem metadata operations (creating or deleting thousands of small files), or excessive context switching. If sy is high and cs is 50,000+, you likely have too many threads contending for locks or rapid process churn. Check sar -n DEV 1 for packet rates or ss -s for socket counts. If sy is high during a backup or find traversal, that's expected. If it's high with no obvious I/O, profile with perf top to see which kernel functions are hot.

Diagnosing Memory Pressure with vmstat

Low free memory is normal if buff and cache are high. Linux uses every spare byte of RAM for disk cache; the kernel reclaims it instantly when applications need it. The problem is swap activity. Here's a server under memory pressure:

procs -----------memory---------- ---swap--
 r  b   swpd   free   buff  cache   si   so
 3  0      0 2145280 312448 4523008  0    0
 4  1   8192 1823104 312448 4198400  0  512
 5  1  24576 1456128 312448 3845632 128 1024
 6  2  49152 1124736 312448 3421184 256 1536
 7  2  81920  845312 312448 2998272 384 2048

Watch si and so. Both are non-zero and growing over ten seconds: the kernel is swapping pages out to disk (so) and swapping them back in (si). This is active swapping, a performance killer. Even if free is positive, if so stays above zero under load, you don't have enough RAM for the workload. Applications stall waiting for disk I/O every time they touch a swapped page.

Swap Activity: The Red Flag

If swpd is static and so is zero, old pages were swapped out but the system is stable now. No new swapping means the kernel found equilibrium. But if free and buff/cache are both near zero and so is climbing, you're out of memory and the kernel is desperately paging. Next step: check the OOM killer in dmesg | grep -i oom for recent kills, or run ps aux --sort=-%mem | head -20 to find the memory hog. Look for processes with high RSS (resident set size). Kill or restart the offender, or add RAM.

On servers with sufficient RAM, you can lower vm.swappiness (default 60) to discourage swapping. Set sysctl vm.swappiness=10 to make the kernel prefer reclaiming page cache over swapping application memory. But that's tuning, not diagnosis. If you're swapping under normal load, you need more memory or fewer processes.

Free Memory and the Page Cache

Here's healthy memory usage on the same server:

procs -----------memory---------- ---swap--
 r  b   swpd   free   buff  cache   si   so
 2  0      0  512384 318720 5234176  0    0
 1  0      0  498112 318720 5248448  0    0

free is only 512 MB but cache is over 5 GB. The kernel is using RAM for disk cache. swpd and so are both zero: no swapping. This is normal. If an application allocates a gigabyte, the kernel will drop a gigabyte of cache and hand it over. Don't panic when free is low; panic when so is non-zero. For a comprehensive guide to memory tuning, see Linux Performance Tuning.

Using mpstat to Identify Per-Core Bottlenecks

Run mpstat -P ALL 1 5 to see per-core CPU usage updated every second for five iterations. Here's output on a 4-core server running a single-threaded application:

Linux 5.15.0 (server01)     01/15/2025  _x86_64_

10:23:45 AM  CPU    %usr   %sys  %iowait  %idle
10:23:46 AM  all   25.12   3.24     0.00  71.64
10:23:46 AM    0   98.00   2.00     0.00   0.00
10:23:46 AM    1    1.00   1.00     0.00  98.00
10:23:46 AM    2    2.00   2.00     0.00  96.00
10:23:46 AM    3    1.00   1.00     0.00  98.00

CPU0 is pegged at 100% (%usr 98%, %sys 2%), while the other three cores are idle. The aggregate line shows 25% CPU usage, and vmstat would show 75% idle, but the application is still slow because it can't use more than one core. This is a single-threaded bottleneck. The solution is code change (parallelize the workload) or scale horizontally by running multiple instances. You can't fix this by adding cores to the same process.

Spotting Single-Threaded Workloads

If all cores show even load (80% %usr across all four), you have a true multi-threaded CPU bottleneck. Here's what that looks like:

10:25:12 AM  CPU    %usr   %sys  %iowait  %idle
10:25:13 AM  all   82.45   8.12     0.00   9.43
10:25:13 AM    0   81.00   9.00     0.00  10.00
10:25:13 AM    1   84.00   7.00     0.00   9.00
10:25:13 AM    2   82.00   8.00     0.00  10.00
10:25:13 AM    3   83.00   8.00     0.00   9.00

All cores are near 80-84% busy. The application is using all available CPU. Your options: optimize the code, reduce the workload, or add more cores. Compare this to the single-threaded case where adding cores doesn't help.

Interpreting Steal Time on VMs

The %steal column in mpstat output shows time your VM wanted to run but the hypervisor scheduled another VM instead. On bare metal, steal time is always zero. On VMs, non-zero steal means the host is oversold: too many VMs competing for the same physical cores. Here's a VM with noticeable steal:

10:27:08 AM  CPU    %usr   %sys  %steal  %idle
10:27:09 AM  all   45.23  12.34   18.76  23.67
10:27:09 AM    0   46.00  13.00   19.00  22.00
10:27:09 AM    1   44.00  12.00   18.00  26.00

Steal at 18-19% is serious. Your VM is losing nearly a fifth of its CPU time to the hypervisor. 5% steal is noticeable in latency-sensitive applications; 20%+ means your workload is fighting for CPU with other tenants. Check with your cloud provider or resize the instance to a less crowded host. Some providers offer dedicated or burstable instance types; if you're on burstable (like AWS t3 instances), you may have exhausted your CPU credits.

Correlating vmstat with Other Tools

When vmstat points to a bottleneck, you need a second tool to confirm and identify the culprit. Here's how to interpret vmstat output and choose the next step.

When vmstat Shows I/O Wait

If wa (I/O wait) is above 20% and b (blocked processes) is greater than zero, the bottleneck is disk or network I/O, not CPU or memory. Here's what that looks like:

procs ------cpu-----
 r  b us sy id wa st
 1  3 12  8 45 35  0
 2  4 10  7 38 45  0

wa at 35-45% and b at 3-4 means processes are blocked waiting for I/O. Next step: run iostat -x 1 to see per-disk %util. If a disk shows 100% utilization, that's your bottleneck. Use iotop to find which process is doing the I/O. If wa is high but iostat shows low disk utilization (say 20%), suspect slow NFS or network-attached storage. The processes are waiting on network I/O, not local disk. Check NFS server health, network latency, or mount options like rsize and wsize.

When Context Switches Spike

If cs (context switches) exceeds 50,000 and in (interrupts) is also high, you have a network interrupt storm or too many threads competing for resources. Here's an example:

-system--
  in    cs
 8234 12450
62389 78234
61923 76891

Interrupts jump from 8,000 to over 60,000 per second, and context switches follow. Check /proc/interrupts to see which device is generating interrupts. If a network interface dominates, you may have a packet flood or misconfigured interrupt affinity. Run sar -n DEV 1 to see per-interface packet rates. If rxpck/s is in the hundreds of thousands, you're handling a high packet rate. Consider irqbalance to distribute interrupts across cores, or enable receive-side scaling (RSS) on the NIC. If cs is high but in is normal, you have too many threads or processes. Check thread counts with ps -eLf | wc -l; thousands of threads on a small server can cause context switch overhead.

Use this decision tree: high wa → run iostat -x 1; high cs and normal CPU → check thread counts and /proc/interrupts; low free and swap out → run ps aux --sort=-%mem to find memory hogs. If r (runnable processes) stays above twice your core count and CPU is not at 100%, you might have lock contention or scheduler issues; perf or strace would be the next step, but that's advanced territory. For a structured approach to combining these tools, see Linux Troubleshooting Techniques.

Common vmstat Patterns and What They Mean

When you're staring at vmstat output under pressure, you need to recognize patterns instantly. Here are six common scenarios with example column values and what they tell you:

Use this as a quick reference when you need to classify a problem fast. The first three columns you check should be r (compare to core count), so (swap out, should be zero), and id (idle time). If id is near zero, look at us, sy and wa to determine whether it's CPU, kernel overhead or I/O. If so is non-zero, you have a memory problem regardless of what else is happening. On VMs, always check st in mpstat output; steal time above 10% means your performance problem is outside your control.

Building a Quick Performance Check Script

When a user reports slowness at 3 AM or you suspect a recurring problem during peak hours, you want a snapshot of system state without manually running commands. This script captures vmstat, mpstat, memory and load average, then logs to a file with timestamps:

#!/bin/bash
set -euo pipefail

LOG="/var/log/perf-check.log"
INTERVAL=300  # 5 minutes
DURATION=3600 # 1 hour

echo "=== Performance check started at $(date) ===" >> "$LOG"

for i in $(seq 1 $((DURATION / INTERVAL))); do
    {
        echo "--- Snapshot at $(date) ---"
        uptime
        free -m
        vmstat 1 5
        mpstat -P ALL 1 5
        echo ""
    } >> "$LOG"
    sleep "$INTERVAL"
done

echo "=== Performance check ended at $(date) ===" >> "$LOG"

This runs vmstat 1 5 and mpstat -P ALL 1 5 every five minutes for an hour, appending to /var/log/perf-check.log. The vmstat 1 5 gives you five one-second samples, enough to see variability without flooding the log. free -m shows memory in megabytes, easier to read than kilobytes. uptime captures load averages for context. Use set -euo pipefail to catch errors early; the script stops if any command fails.

If you run this continuously via cron, configure log rotation. Add to /etc/logrotate.d/perf-check:

/var/log/perf-check.log {
    daily
    rotate 7
    compress
    missingok
    notifexist
}

This keeps seven days of logs and compresses old ones. Run the script manually during a known problem window (a batch job, peak traffic hour) or schedule it with cron for recurring issues: 0 14 * * * /usr/local/bin/perf-check.sh runs it daily at 2 PM. This is not a replacement for proper monitoring (use Prometheus with node_exporter for that) but it gives you a forensic snapshot when you need to diagnose a specific incident. Automate when the problem occurs at predictable times; run manually for one-off investigations.

Your Next Steps After Identifying the Bottleneck

Once you've identified the bottleneck with vmstat and mpstat, your next steps depend on what you found:

• CPU bottleneck: Run top or pidstat 1 to identify the process burning CPU. If it's a database, check slow query logs. If it's a web server, profile the application code. If mpstat shows one core pegged, the application is single-threaded; parallelize or run multiple instances. If all cores are saturated, optimize the hot path or add capacity. See Linux Process Resource Monitoring: CPU, Memory, and Load Analysis for process-level diagnostics.
• Memory pressure: Use ps aux --sort=-%mem to find the largest processes. Check dmesg | grep -i oom for recent out-of-memory kills. If a process is leaking memory, restart it and file a bug. If legitimate workload exceeds RAM, add memory or tune vm.swappiness and cache limits. The kernel's memory management tunables are documented at kernel.org/doc/Documentation/sysctl/vm.txt.
• I/O bottleneck: Run iostat -x 1 to see per-disk utilization. If a disk is at 100% %util, use iotop to find the process. For slow NFS, check mount options, server health, and network latency. If local disk is the limit, consider faster storage or spreading load across multiple volumes.
• Context switch storm: Check ps -eLf | wc -l for thread count. If you have thousands of threads, tune application thread pools. If interrupts are high (/proc/interrupts), enable irqbalance or tune NIC interrupt coalescing.
• VM steal time: Contact your cloud provider or resize to a dedicated instance type. On AWS, switch from t3 to c5/m5 instances. On overcommitted hosts, migrate to a different availability zone or physical host.

For a comprehensive guide to tuning after diagnosis, see Linux Performance Tuning. Keep a performance baseline: run vmstat and mpstat during normal operation and save the output. When you're troubleshooting, compare current numbers to your baseline to see what changed. Download our Linux Commands: Performance & Debugging (Part 2) cheat sheet for a quick reference to all these tools.

Frequently Asked Questions