Linux Investigation Lab

Linux FoundationsLab 2 of 2 · Investigative75–90 minutesLabBuilds on: Documentation and Software

This is the last Linux Foundations lab, and the commands in it are all familiar by now. What’s changed is the layout: no copy-and-paste transcript this time. Your job is to work out which observation answers each question, then enter the four results below.

Before you begin

This lab pulls together skills from right across the module, so you should already be comfortable with all of the following:

reading a symbolic and numeric permission mode, from Users, Groups, and Permissions
starting a background process, recording its PID with $!, and verifying it with ps, from Processes, Jobs, and Signals
reading a live process through /proc/<pid>/, from the same lesson
sending SIGTERM to one specific PID and collecting it with wait, again from that lesson
redirecting with >, combining streams with 2>&1, and building a grep, cut, sort, and uniq pipeline, from Input, Output, Pipes, and Redirection and Pipelines and Text Search

If any of those is still unfamiliar, go back to that lesson first. This lab asks you to combine those skills, so practice any uncertain steps before starting it.

Scope and safety

The extracted linux-investigation-v1.0.0 directory is the whole target, and nothing outside it is in scope. Run all of this inside the disposable course VM, and once the archive is downloaded the lab needs no network access at all.

You’ll start one copy of watcher.sh, and send a signal only to the PID you recorded from that exact command. Don’t reach for pkill, killall, sudo, or a wildcard. Name-based or bulk selection could affect another process, and this lab needs no administrative privileges. The skill being practiced is acting on exactly one process, positively identified by its PID.

Download and verify the artifact

Download linux-investigation-v1.0.0.tar inside the course VM and save it in ~/underdunn-course/downloads.

Expected archive SHA-256:

c0e89ec2a45205d9ea43f0c614ec79a8a51a95863598b196b81a72e810d61b45

Verify and extract it:

cd ~/underdunn-course/downloads
sha256sum linux-investigation-v1.0.0.tar
mkdir -p ~/underdunn-course/labs/linux-investigation
tar -xf linux-investigation-v1.0.0.tar -C ~/underdunn-course/labs/linux-investigation
cd ~/underdunn-course/labs/linux-investigation/linux-investigation-v1.0.0

Stop if that archive hash doesn’t match. A fresh extraction gives you README.txt, events.log, and watcher.sh, and nothing else.

Milestone 1: inspect the executable mode

Work out the numeric permission mode of watcher.sh. Check the symbolic form, the numeric form, its owner and group, and the command that produced all of it.

Predict the numeric mode from the symbolic listing before you ask any tool for a formatted numeric value.

You used stat formatting back in the permissions lesson, and stat --help will probably remind you of the format placeholders faster than the manual will. Don’t change the permissions on anything here.

Use the three-digit numeric mode in the answer field below.

Milestone 2: start the watcher with an environment value

Read README.txt first. Then find out what watcher.sh needs by running it, and not by reading its shell code (that restriction is deliberate). Run it once with nothing set, and watch what it does:

./watcher.sh

It prints an error and exits straight away. That error message is your evidence for what the script wants before it’ll start its loop. If it doesn’t exit on its own, press Ctrl+C.

Now supply that value for a single run. Writing NAME=value immediately before a command sets the variable for that one command only, and your own shell’s environment stays completely untouched. Predict whether a value supplied this way shows up through /proc, and then run the command below, which does three things at once:

starts one bounded background copy with RESEARCH_TAG set to pond
sends both ordinary and error output into observer.log
records the PID straight away

RESEARCH_TAG='pond' ./watcher.sh > observer.log 2>&1 &
watcher_pid=$!
printf 'watcher_pid=%s\n' "$watcher_pid"

Confirm that this PID belongs to watcher.sh:

ps -p "$watcher_pid" -o pid,ppid,stat,comm,args

Now confirm the value really did reach the process. The file /proc/$watcher_pid/environ holds the process’s environment. Its entries are separated by NUL bytes rather than newlines, so reading it raw just runs everything together into one mess. tr '\0' '\n' turns each NUL byte into a newline, < feeds the file in as standard input, and grep keeps only the line you want:

tr '\0' '\n' < "/proc/$watcher_pid/environ" | grep '^RESEARCH_TAG='

Check the value assigned to RESEARCH_TAG in the answer field below.

Milestone 3: stop only the recorded process

Predict the last watcher message, and predict whether the PID will still be there after wait. Then send SIGTERM to the PID stored in watcher_pid:

kill -TERM "$watcher_pid"
wait "$watcher_pid"

wait asks the shell to hang on until that background job finishes. Now read observer.log, and use ps to confirm the PID has really gone (an empty ps result is the evidence you want here).

The script records the signal name after signal=, and that value goes in the signal answer field below.

If the process doesn’t exit, check the PID again and give it a second before looking once more. Usually it’s a slow shutdown rather than a stuck one. Don’t escalate to SIGKILL until you’ve proven the correct process is genuinely still there.

Milestone 4: identify the repeated denied subject

events.log holds three fields separated by |: time, subject, and result. Work out which subject has the most denied records.

Look at the unfiltered file first, predict the most frequent denied subject, and only then build the pipeline. Revise your hypothesis only after you’ve recorded each intermediate output.

Build the pipeline in stages and inspect each stage’s output:

keep only the denied records
extract just the subject field
group equal subjects together
count each group
sort those counts from largest to smallest

The subject on the highest-count row goes in the subject answer field below. If two subjects come out tied, either the pipeline or the interpretation is wrong. The supplied evidence has exactly one clear result.

Layered hints

Hint 1: The tools already appeared together

Go back to Pipelines and Text Search. You want grep, cut, sort, and uniq -c, followed by a numeric reverse sort.

Hint 2: Sort the counts numerically

After uniq -c, sort -nr places the largest numeric count first.

Hint 3: Build the complete pipeline

Start with grep 'denied' events.log, pipe that through cut -d'|' -f2, and then add sort, uniq -c, and sort -nr one stage at a time. Check each intermediate output rather than pasting the finished pipeline in with no evidence behind it.

Check the four values

Enter the mode, signal name, subject and environment value you found.

Common failures

Permission denied when starting watcher.sh: inspect its mode, and confirm you’re in the extracted version directory. Don’t change the supplied permissions, and don’t add sudo.
wait says the PID isn’t a child of this shell: go back to the terminal that launched the background job. Shell job state doesn’t travel between terminals.
/proc/PID/environ is missing: confirm the recorded PID still exists and still belongs to the watcher before you repeat the observation.
Two subjects come out tied: read the output after every pipeline stage. Confirm you kept only the denied records, and that you sorted the subjects before uniq -c.

Check your result

The checker should accept the mode, signal name, environment value and event-analysis result. If it rejects a value, recheck it at its source. Confirm that the watcher has stopped with ps before leaving the lab.

Reset and cleanup

Before resetting, check the PID you recorded:

ps -p "$watcher_pid"

If the watcher is still running, send it SIGTERM and wait for it. Then reset by deleting only ~/underdunn-course/labs/linux-investigation/linux-investigation-v1.0.0 with Ubuntu Files, and extract the verified archive again. Keep any notes or scripts you want outside the directory before resetting it.