Pipes and Redirection: the Idea That Makes the Shell Powerful

Individually, Unix commands are small and almost simplistic. The reason the terminal beats graphical tools for text work is one idea: every command reads from an input stream and writes to output streams, and you can plug those streams into files and into each other. Master this and you can compose tools that were never designed to work together.

The three streams

Every process starts life with three open channels:

Stream	Number	Default	Purpose
stdin	0	your keyboard	Input the program reads
stdout	1	your screen	Normal results
stderr	2	your screen	Errors and diagnostics

The crucial detail: results and errors travel on separate streams, even though both appear on your screen by default. That separation is what lets you save clean output to a file while still seeing errors, or silence the noise while keeping the data.

Redirecting output: > and >>

ls -l > listing.txt     # stdout goes to the file, REPLACING its contents
echo "new line" >> log.txt   # stdout is APPENDED to the file

The single most important distinction in this guide: > truncates the file first — its previous contents are gone the instant the shell parses the command. >> preserves and appends. When accumulating anything (logs, notes, results of repeated runs), >> is what you want.

Classic self-inflicted wound: sort data.txt > data.txt empties the file before sort reads it — you lose the data. Redirect to a new file, or use sort -o data.txt data.txt.

Redirecting errors: 2> and 2>&1

find / -name "*.conf" 2> /dev/null      # errors vanish, results stay visible
make > build.log 2>&1                    # BOTH streams into one file
cmd > out.log 2> err.log                 # results and errors to separate files

2> means "redirect stream 2". /dev/null is the system's black hole — anything written there is discarded, which makes 2> /dev/null the standard way to mute permission-denied noise from find. 2>&1 means "send stream 2 wherever stream 1 currently points" — note it must come after the stdout redirection to capture both in one file.

Pipes: plugging commands together

A pipe | connects one command's stdout to the next command's stdin — no intermediate file, both processes running simultaneously:

ps aux | grep nginx
history | grep ssh
du -h --max-depth=1 | sort -hr | head -n 10

Read pipelines left to right as an assembly line. The third example: measure directory sizes → sort them largest-first → keep the top ten. Each stage is a simple tool; the pipeline is the program. This compositional style is why grep, awk, cut and head are designed to read stdin when given no file.

A pipeline: the shell wires stream 1 of the producer to stream 0 of the consumer. Errors travel apart on stream 2.

tee: watch and save at once

./deploy.sh | tee deploy.log         # see it live AND keep a copy
./deploy.sh 2>&1 | tee deploy.log    # including errors

tee duplicates its input: one copy continues to the screen (or the next pipe stage), one copy goes to a file. Named after a T-shaped pipe fitting — the plumbing metaphor is official.

Pipelines worth stealing

# Most common words in a file
tr ' ' '\n' < essay.txt | sort | uniq -c | sort -nr | head

# Which IPs hit your server most
awk '{print $1}' access.log | sort | uniq -c | sort -nr | head

# Total size of all .log files
find . -name "*.log" -exec du -b {} + | awk '{s+=$1} END {print s/1024/1024 " MB"}'

# Watch a log for errors, live
tail -f app.log | grep --line-buffered ERROR

Try it

This sandbox supports pipes and both redirection operators. A good sequence: ls | grep notes, then echo "first" > test.txt, echo "second" >> test.txt, cat test.txt — and see the overwrite-vs-append difference with your own eyes:

Keep going

Lessons 9–19 of the interactive course drill redirection and pipes step by step, and the command reference covers each pipeline tool in depth.