The uniq command in Linux
uniq collapses adjacent duplicate lines, and with -c prefixes each with its count. The one rule that explains everything: it only compares neighbors, so input must be sorted first. sort | uniq -c | sort -nr is the frequency-table pipeline — one of the most-typed sequences in all of Unix.
How uniq works
uniq holds exactly one line in memory: the previous one. Each new line is compared against it; equal means suppressed (or counted), different means the held line is emitted and replaced. One pass, O(1) memory, arbitrarily large input — and the adjacent-only rule is not a limitation but the design that makes that possible.
The sort-first requirement follows directly: sorting is what makes duplicates adjacent. This division of labor is deliberately Unix — sort knows ordering, uniq knows adjacency, and the pipeline composes them. When you need order-preserving deduplication instead, you are asking for a hash table, which is awk's '!seen[$0]++' in disguise.
Syntax
uniq [OPTIONS] [FILE]Common options
| Option | What it does |
|---|---|
-c | Prefix each line with its occurrence count. |
-d | Only lines that were duplicated. |
-u | Only lines that were unique. |
-i | Case-insensitive comparison. |
How to use uniq: examples
$ sort names.txt | uniqThe deduplicated list (equivalent to sort -u).
$ awk '{print $1}' access.log | sort | uniq -c | sort -nr | headTop clients by request count — the canonical frequency pipeline.
$ sort emails.txt | uniq -dShow only the duplicates — find the double signups.
Real-world use cases for uniq
Top-N anything
The frequency pipeline — extract | sort | uniq -c | sort -nr | head — answers "most common X" for IPs in logs, error types, user agents, words in a corpus. Learn it as one unit; you will type it weekly forever.
Set operations with files
Duplicated signups across two exports: cat a.txt b.txt | sort | uniq -d prints only lines present in both. -u gives the symmetric difference. Poor-man's SQL over text files, zero setup.
Pro tips and common mistakes
- uniq without sort in front is almost always a bug — adjacency is the contract.
- uniq -c then sort -nr, in that order: count first, rank second.
- Just deduplicating? sort -u is one process and one intention.
Frequently asked questions about uniq
Why did uniq leave duplicates?
They were not adjacent. Sort first: sort file | uniq. This trips everyone exactly once.
uniq or sort -u?
For plain dedup, sort -u is one process. uniq earns its place when you need counts (-c) or duplicate/unique filtering (-d/-u).
Order-preserving dedup?
awk '!seen[$0]++' file — no sorting, first occurrence wins.