Input, Output, Pipes, and Redirection

Linux FoundationsLesson 7 of 960 minutesLessonBuilds on: Processes, Jobs, and SignalsNo lab next

The question: where will these bytes go?

Command-line programs usually have no idea whether a person, a file, or another program is sitting on the other end of their input and output. Linux hands each process numbered file descriptors, and a descriptor is just a process-local handle on an open stream (a number, and nothing more mysterious than that). The shell wires up the first three of them before the program’s even started running.

That indirection is the whole trick, and it’s a really good one. A program can write its ordinary results to descriptor 1 without needing separate output code for terminals, files, and pipes. The shell changes what the descriptor is connected to, and the program keeps writing to the same interface it always did.

Create and enter a disposable directory:

mkdir -p ~/underdunn-course/exercises/streams
cd ~/underdunn-course/exercises/streams
pwd

Every redirection target in this lesson lives inside that directory, and that isn’t an accident. Check pwd before you run any command containing a > (it’s a habit worth building now rather than after the first accident).

Three standard streams

Most command-line processes begin with three numbered stream interfaces:

0, standard input: bytes supplied to the process.
1, standard output: ordinary result bytes.
2, standard error: diagnostic bytes.

Run a command that produces one ordinary line and one error:

ls /etc/os-release missing-entry

Both lines show up in the terminal, and they really did travel there through different streams.

Standard output and standard error are separate because ordinary results and diagnostics do genuinely different jobs. You’ll often want to save the data for another tool and still watch the failures go past on screen. If both kinds of bytes shared one undifferentiated channel, every downstream tool would have to pick error messages out of the data by guesswork, which works about as well as it sounds.

The observed destinations give this relationship a concrete reference:

How the three standard streams connect to a command

stdin0 · input

command

stdout1 · results

stderr2 · errors

Predict before overwriting

Redirection is shell syntax, not program syntax. The shell spots the > before ls ever starts, opens the destination file, and connects it to descriptor 1. Only then does ls run, writing to standard output exactly as it always does. So redirection works perfectly well with programs that have never heard of the > character, which is most of them.

Predict whether result.txt will contain one line or two after:

ls /etc/os-release missing-entry > result.txt

The terminal should still show you the missing-entry error. Now check the file:

cat result.txt

Expected file content:

/etc/os-release

So the observation is that > redirected standard output and nothing else. The diagnostic stayed on standard error, where it started.

Now redirect only the error:

ls /etc/os-release missing-entry 2> errors.txt
cat errors.txt

The valid path stays on the terminal, and errors.txt now holds the diagnostic.

The 2 selects descriptor 2 explicitly, and plain > is just shorthand for descriptor 1. I’d read redirections left to right, as changes to the process’s connections rather than decorations hanging off a filename.

Send both streams to one place

Sometimes you want the results and the diagnostics together in one record. 2>&1 says to point descriptor 2 at wherever descriptor 1 currently points. The &1 means “the current destination of descriptor 1”, and not a file literally named 1.

ls /etc/os-release missing-entry > combined.txt 2>&1
cat combined.txt

Order matters here, and this is probably the part that catches everybody out at least once. The shell applies redirections strictly left to right. > combined.txt points descriptor 1 at the file, and then 2>&1 points descriptor 2 at that same file. Write the two the other way around and descriptor 2 ends up pointed at descriptor 1’s original terminal destination, from before you moved it.

The descriptor-duplication examples in Bash redirections explain the ordering rule. Draw the destinations after each redirection before running a variant.

Overwrite and append differ

With >, the shell opens the destination in a mode that truncates the existing contents before the command writes anything at all. With >>, it opens the file for appending instead, and nothing gets truncated. Notice the truncation happens before the program has produced a single useful byte (the file is empty the instant the shell opens it), so check the directory and destination name before running the command.

Create a known baseline:

printf 'first\n' > notebook.txt

Predict each final file before running:

printf 'second\n' > notebook.txt
cat notebook.txt
printf 'third\n' >> notebook.txt
cat notebook.txt

After the overwrite, only second is left. After the append, third follows it on a new line.

A pipe connects two process interfaces

A pipe is a kernel-managed byte channel. The shell connects one process’s standard output to the next one’s standard input and then starts the whole pipeline, and those processes will usually run at the same time rather than one after the other. A pipe preserves byte order and understands absolutely nothing else. Rows, columns, and meaning are interpretations that belong entirely to the programs on either end.

Predict the count:

printf 'map\nheap\nmap\n' | grep '^map$' | wc -l

Expected output:

The first process writes three lines to standard output, and the pipe delivers those bytes to grep as its standard input. The second pipe then hands the matching rows to wc, which counts them.

Every stage should have one simple contract. Here printf produces lines, grep keeps the matching ones, and wc -l counts whatever survives that. When the final number comes out wrong, walk those contracts from left to right to find the first output that differs from your prediction.

Now change exactly one variable, the grep pattern, from map to heap. Predict the new count first, then check it.

Display and save with tee

printf 'evidence\n' | tee captured.txt
cat captured.txt

tee writes its input to standard output and to the named file at the same time, which is exactly what it’s for. Reach for it whenever you want to see the bytes and keep them at the same time.

The name suggests a T-shaped plumbing fitting: one input branching into two outputs. Here the branches are the terminal and the file, both receiving a copy of the same bytes.

Common failures and recovery

You overwrote something valuable: stop there. Nothing in this lesson can recover an unbacked-up file, and that’s exactly why every target lives in the disposable directory.
An error didn’t get captured: use 2> for standard error rather than plain >.
The pipeline count isn’t what you expected: run each stage on its own and read its complete output before you reconnect the pipe.
Quotes or spaces changed the data: use the exact printf format and quoted argument shown above.
The directory is full of stale results: check pwd, then remove only result.txt, errors.txt, notebook.txt, and captured.txt.

Check the result and clean up

Check the files you redirected into: > replaces their contents, while >> appends. Standard error reaches a different destination unless you redirect it too.

Then remove the disposable directory:

cd ~/underdunn-course/exercises
rm -rf streams

If a line appeared in the wrong place, follow its stream and the redirections from left to right.

Next we’ll combine these stream interfaces with small text tools to answer a bounded evidence question.

Check your understanding

Knowledge check

Question 1 of 2

A command uses >out 2>&1. Where do its stdout and stderr go?

Only stdout goes to outBoth streams go to outOnly stderr goes to out