sed and awk Tutorial: Text Processing Power Tools for Linux Administrators

sed and awk are two of the most powerful text processing tools in the Linux toolkit. Together, they can handle virtually any text transformation task, from simple search-and-replace to complex data processing. This tutorial covers both tools with practical examples.

sed: Stream Editor

Basic Substitution

# Replace first occurrence on each line
sed 's/old/new/' file.txt

# Replace all occurrences on each line
sed 's/old/new/g' file.txt

# Case-insensitive replace
sed 's/old/new/gi' file.txt

# In-place editing
sed -i 's/old/new/g' file.txt

# In-place with backup
sed -i.bak 's/old/new/g' file.txt

Line Operations

# Print specific lines
sed -n '5p' file.txt           # Line 5
sed -n '5,10p' file.txt        # Lines 5-10
sed -n '/pattern/p' file.txt   # Lines matching pattern

# Delete lines
sed '5d' file.txt              # Delete line 5
sed '/^#/d' file.txt           # Delete comment lines
sed '/^$/d' file.txt           # Delete empty lines

# Insert and append
sed '3i\New line before line 3' file.txt
sed '3a\New line after line 3' file.txt

Practical sed Examples

# Change config values
sed -i 's/^port=.*/port=8080/' config.ini

# Remove HTML tags
sed 's/<[^>]*>//g' page.html

# Add line numbers
sed = file.txt | sed 'N; s/\n/\t/'

# Extract between patterns
sed -n '/START/,/END/p' logfile.txt

awk: Pattern-Action Language

Field Processing

# Print specific columns
awk '{print $1, $3}' file.txt

# Custom field separator
awk -F':' '{print $1, $3}' /etc/passwd
awk -F',' '{print $2}' data.csv

# Print with formatting
awk '{printf "%-20s %10s\n", $1, $2}' file.txt

Pattern Matching

# Print lines matching pattern
awk '/error/' logfile.txt

# Print lines where field matches condition
awk '$3 > 100' data.txt
awk '$1 == "admin"' users.txt

# Conditional processing
awk '{ if ($3 > 80) print $1, "PASS"; else print $1, "FAIL" }' grades.txt

Calculations and Aggregation

# Sum a column
awk '{sum += $2} END {print "Total:", sum}' sales.txt

# Average
awk '{sum += $2; count++} END {print "Average:", sum/count}' data.txt

# Count occurrences
awk '{count[$1]++} END {for (key in count) print key, count[key]}' access.log

# Find max value
awk 'BEGIN {max=0} {if ($2 > max) {max=$2; line=$0}} END {print line}' data.txt

Practical awk Examples

# Parse Apache access logs
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10

# Extract and sum disk usage
df -h | awk 'NR>1 {print $5, $6}'

# Process CSV files
awk -F',' 'NR>1 {sum[$2] += $3} END {for (dept in sum) print dept, sum[dept]}' employees.csv

# Generate reports
awk -F':' '{
    if ($3 >= 1000) users++
    else system++
} END {
    print "Regular users:", users
    print "System accounts:", system
}' /etc/passwd

Combining sed and awk

# Clean and process log data
cat access.log | sed '/^$/d' | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn

# Transform CSV to formatted output
cat data.csv | sed '1d' | awk -F',' '{printf "| %-20s | %10s | %8s |\n", $1, $2, $3}'

Quick Reference

• Use sed for: search and replace, line editing, stream transformation
• Use awk for: column extraction, calculations, pattern-based processing
• Combine both with pipes for complex transformations

sed and awk are timeless tools that every Linux administrator should master. They are available on every Unix system and can solve text processing challenges that would require significantly more code in other languages.