uniq Command
Beginner Text Processing man(1)Report or filter out repeated lines
👁 10 views
📅 Updated: Mar 15, 2026
SYNTAX
uniq [OPTION]... [INPUT [OUTPUT]]
What Does uniq Do?
The uniq command filters or reports adjacent repeated lines in sorted input. It can remove duplicates, count occurrences, show only duplicated lines, or show only unique lines.
Important: uniq only detects adjacent duplicates, so input must be sorted first (using sort) unless you specifically want to find consecutive repeated lines. The sort | uniq pipeline is one of the most common text processing patterns in Linux.
uniq is essential for log analysis, data deduplication, and frequency analysis. Combined with sort and head, it enables powerful one-liner analytics like finding the most common IP addresses, frequent errors, or top requested URLs.
Important: uniq only detects adjacent duplicates, so input must be sorted first (using sort) unless you specifically want to find consecutive repeated lines. The sort | uniq pipeline is one of the most common text processing patterns in Linux.
uniq is essential for log analysis, data deduplication, and frequency analysis. Combined with sort and head, it enables powerful one-liner analytics like finding the most common IP addresses, frequent errors, or top requested URLs.
Options & Flags
| Option | Description | Example |
|---|---|---|
| -c | Prefix lines with occurrence count | sort data.txt | uniq -c |
| -d | Only print duplicate lines | sort names.txt | uniq -d |
| -u | Only print unique (non-duplicated) lines | sort names.txt | uniq -u |
| -i | Case-insensitive comparison | sort -f data.txt | uniq -i |
| -f N | Skip first N fields when comparing | uniq -f 1 log.txt |
| -s N | Skip first N characters when comparing | uniq -s 10 timestamps.txt |
| -w N | Compare only first N characters | uniq -w 20 data.txt |
Practical Examples
#1 Remove duplicate lines
Sorts and removes adjacent duplicates. This is the standard deduplication pattern.
$ sort file.txt | uniq#2 Count occurrences
Finds the 10 most common lines in a log file with counts.
$ sort access.log | uniq -c | sort -rn | head -10
Output:
247 GET /api/status\n 189 GET /index.html
#3 Find duplicate entries
Shows only lines that appear more than once.
$ sort emails.txt | uniq -d#4 Find unique-only entries
Shows only lines that appear exactly once (no duplicates).
$ sort data.txt | uniq -u#5 Top IP addresses from access log
Extracts IPs, counts unique occurrences, and shows the top 20.
$ awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20
Output:
1523 192.168.1.100\n 847 10.0.0.50
#6 Case-insensitive dedup
Counts occurrences ignoring case differences.
$ sort -f names.txt | uniq -icTips & Best Practices
Always sort first: uniq only removes ADJACENT duplicates. Always pipe through sort first: sort file | uniq. Without sort, non-adjacent duplicates will remain.
Frequency analysis pattern: The classic pattern: sort | uniq -c | sort -rn | head -N gives you the top N most frequent items in any list.
sort -u as alternative: For simple deduplication, sort -u is more efficient than sort | uniq since it does both in a single pass.
Frequently Asked Questions
Why does uniq not remove all duplicates?
uniq only removes adjacent duplicate lines. You must sort the input first: sort file.txt | uniq. Alternatively, use sort -u for combined sort and dedup.
How do I count duplicate occurrences?
Use sort file.txt | uniq -c. This prefixes each line with its count. Add | sort -rn to see the most frequent first.
How do I find lines that appear more than once?
Use sort file.txt | uniq -d to show only lines that have duplicates. Add -c for counts: sort file | uniq -cd.
Related Commands
More Text Processing Commands
Master Linux with Professional eBooks
Curated IT eBooks covering Linux, DevOps, Cloud, and more
Browse Books →