Pipelines and Text Search

Linux FoundationsLesson 8 of 975 minutesLessonBuilds on: Input, Output, Pipes, and RedirectionNo lab next

Learning objectives

By the end of this lesson

• Use grep, cut, sort, uniq, and wc for one narrow evidence question.
• Explain what each pipeline stage receives and produces.
• Use find with an observed property rather than a guessed name.
• Check a pipeline’s intermediate output before trusting its result.

The question: which subject appears most often?

Small Unix tools can answer a larger question when we connect them in a pipeline. Each stage transforms the previous stage’s output. If an early filter selects the wrong records, the final count may still look reasonable, so we’ll check the intermediate results as we build it.

We’ll treat the input as a tiny table encoded in text, where each line is one record. The vertical bars split each record into fields: outcome, subject, and action. The job is to narrow the records down, pick out one field from what’s left, put the equal values next to each other, and then count them. At every stage we should be able to say what went in and what came out.

Work only inside a disposable directory:

mkdir -p ~/underdunn-course/exercises/text-evidence
cd ~/underdunn-course/exercises/text-evidence

Create events.txt with this exact content in your text editor:

allow|map-reader|opened maps
deny|guest|opened journal
allow|map-reader|read header
deny|guest|opened journal
deny|builder|wrote output
deny|guest|opened journal

Check the input with wc -l events.txt before you go building anything on top of it, because a pipeline is only as trustworthy as the file underneath. Expected evidence:

6 events.txt

Build the answer one stage at a time

Question: among denied events, which subject occurs most often?

Predict how many denied rows there are, and then run:

grep '^deny|' events.txt

Expected rows:

deny|guest|opened journal
deny|guest|opened journal
deny|builder|wrote output
deny|guest|opened journal

That pattern ^deny| has two pieces worth naming. In a regular expression, ^ anchors the match to the beginning of a line. The trailing | is a literal character here, because basic grep usually won’t treat it as alternation without an extended-expression option. So the pattern selects records whose first field carries the outcome prefix we’re after.

Select the second pipe-delimited field:

grep '^deny|' events.txt | cut -d'|' -f2

You should get guest, guest, builder, and then guest, each on its own line.

cut uses the structure we specify: -d'|' sets the field delimiter and -f2 selects the second field. It doesn’t know that we intend that field to be a subject. If the input format changes, the command may still succeed while selecting different data. Check its output against the file format before counting it.

Group equal values and count them:

grep '^deny|' events.txt | cut -d'|' -f2 | sort | uniq -c

Expected output, ignoring leading spacing:

1 builder
3 guest

So the observation is simple. guest appears in three of the denied rows and builder in exactly one. The conclusion is narrower, and it holds only for this six-row dataset: guest is its most frequent denied subject.

The GNU grep manual is the place to check the matching rule. A filter can run successfully while selecting the wrong records.

How far the conclusion reaches

That conclusion is deliberately bounded, and the boundary is doing real work. This file establishes nothing whatsoever about what happens elsewhere on the system.

This is just technical honesty. The pipeline measured the supplied file, and not all logs, all time, or all Ubuntu systems. To repeat the measurement, use the same input and command. A different log may give you a different count.

Explain the interfaces

Stage	Receives	Produces
`grep ’^deny	’`	Six complete event rows.
`cut -d’	’ -f2`	Denied rows.
sort	Subject names in event order.	Equal names adjacent.
uniq -c	Grouped names.	One count per name.

uniq only ever combines adjacent equal lines, so pulling sort out changes a precondition that really matters.

That’s a hidden dependency between stages, and it’s worth spotting. sort isn’t there to make the output pretty. It puts every occurrence of a name into one group so uniq -c can give us one total for that name. When you come to explain the pipeline, “sorts names” is incomplete, while “places equal names next to each other so that uniq combines all occurrences into one count” states the actual causal relationship.

Practice: change one filter

Predict what happens when the first stage selects allowed events instead, and then run:

grep '^allow|' events.txt | cut -d'|' -f2 | sort | uniq -c

Check the difference. Only the filter changed here, and the field selection and counting stages stayed exactly as they were.

Then answer one more question. How many rows mention journal? Build the answer with grep and wc -l, check it, and check both the command and the number it gave you.

Find entries by observed properties

find answers a genuinely different kind of question. Rather than transforming some text that was handed to it, find walks its way through a filesystem hierarchy and tests every entry it meets against a list of predicates you supply. -maxdepth 1 limits how far the walk goes, -type f requires a regular file, and -empty tests the entry’s size or contents as find defines it.

Create two small files and one directory:

touch empty-record
mkdir archive

Predict which names this command will report:

find . -maxdepth 1 -type f -empty -print

The result should include ./empty-record and leave out the directory. events.txt isn’t empty, so it won’t appear either.

The name empty-record is convenient for us, and the command doesn’t lean on it at all. That’s the whole point. A property-based search will usually turn up an object even when its name is unfamiliar or actively misleading.

Common failures and recovery

• Repeated names come back with separate counts: confirm sort really does come before uniq -c.
• cut prints whole rows: check the delimiter is the quoted single character |, and that the field is -f2.
• Nothing matches at all: run wc -l events.txt, then replace the file with the exact six-line dataset above.
• The shell interprets the | inside a pattern: quote the pattern or the delimiter exactly as shown.
• Practice files got mixed in with unrelated work: check pwd, which has to end in /underdunn-course/exercises/text-evidence.

To get back to a known state, run cd ~/underdunn-course/exercises and remove only the named text-evidence directory.

Check your result

Look back at the filtered rows and the final count. Each counted row must have passed the filter, and repeated subjects can contribute several rows. If a count looks wrong, inspect the output immediately before the counting stage.