Shell, Identity, and Expansion

Linux FoundationsLesson 2 of 960 minutesLessonBuilds on: Linux and the Operating SystemNo lab next

Learning objectives

By the end of this lesson

• Explain the difference between a terminal, shell, command, and argument.
• Predict how quoting and variable expansion change an argument list.
• Distinguish a shell variable from an exported environment variable.
• Use type and PATH to explain how the shell locates a command.

The question: what does a program receive?

The shell reads the line you type, and it gets to it before any other program does. It handles the quoting, expands the variables, and chops the result into arguments.

This is one of the most important bits of plumbing in the whole system, and it trips people up for a long time. What you type is never handed to a program as one untouched sentence. The shell interprets some of those characters first, so spaces separate arguments, quotes protect spaces, $name gets replaced with a value, and a bare command name (one with no slash in it) gets resolved through a search path. If you don’t know which layer made a change, Python or grep ends up taking the blame for an argument it never actually received.

Keep these four roles separate:

• The terminal displays text and carries keyboard input.
• The shell reads and interprets a command line.
• The command names something the shell can run or perform.
• An argument is one value passed to that command.

The command-line input arrives as an ordered list of arguments. The shell removes the quotes used to group arguments; those quote characters aren’t part of the values passed to the program. A process can also receive input through its environment and streams, which we’ll examine separately. So we’ll use a tiny Python program as an inspection window, which makes the boundary something you can look at instead of something you have to imagine.

Build the inspection window

We’ll keep this lesson’s files in one throwaway directory. Two small commands set that up, and you’ll be using them constantly from here on. mkdir -p PATH creates a directory, quietly making any missing parents along the way, and cd PATH moves your shell into it so everything you run afterward happens there.

The leading ~ is shorthand the shell expands to your home directory, so ~/underdunn-course means the underdunn-course folder in your home. The next lesson digs into how paths get resolved. For now these two commands are all you need:

mkdir -p ~/underdunn-course/exercises/shell-basics
cd ~/underdunn-course/exercises/shell-basics

Open a file named show-args.py in the text editor you used during setup. Enter this complete program:

import sys

print(f"argument count: {len(sys.argv) - 1}")
for position, value in enumerate(sys.argv[1:], start=1):
    print(f"argument {position}: <{value}>")

Save it, and then confirm the file is really there. ls lists directory entries, and the -l option prints one entry per line with details like size and permissions:

ls -l show-args.py

Predict the argument list

Before each run, work out how many arguments the shell will pass. Compare that with the numbered values the program prints.

python3 show-args.py river stone
python3 show-args.py "river stone"

Expected evidence from the first run:

argument count: 2
argument 1: <river>
argument 2: <stone>

Expected evidence from the second:

argument count: 1
argument 1: <river stone>

The Python program did not strip those quotes. The quotes told the shell to keep the space inside a single argument, and the quote characters themselves never reached Python at all.

In the first command the shell saw an unquoted space and used it as a separator, and in the second the quotes told it the space belonged inside the current argument. This is why “put quotes around paths with spaces” works, but the underlying rule is a lot more useful than the folklore version. Quote a value whenever it has to cross the shell boundary in one piece.

Observe variable expansion

Assign a shell variable:

subject="process maps"

Predict both argument lists, then run:

python3 show-args.py $subject
python3 show-args.py "$subject"

Expected evidence from the unquoted expansion:

argument count: 2
argument 1: <process>
argument 2: <maps>

Expected evidence from the quoted one:

argument count: 1
argument 1: <process maps>

The stored value was identical both times, and only the quoting changed. So put quotes around an expansion any time the value needs to arrive in one piece:

printf 'current user: <%s>\n' "$USER"

A shell variable is just a name attached to some text in the current shell. Expansion is what the shell does when it replaces $subject with that text while assembling the arguments. Notice the stored value didn’t split itself. Word splitting happened only because the expansion was left unquoted. That distinction starts to matter once your values contain spaces, wildcard characters, or nothing at all.

Bash shell expansions describes the expansion stages behind these argument lists. It is worth consulting when removing a pair of quotes changes the result.

Shell variables and child environments

When the shell starts another program, Linux creates a new process for it. We call that a child process, because the shell process is the one that started it. The child gets its own argument list, plus a collection of name and value strings that’s called the environment.

An ordinary shell variable is private to the shell. export marks one for inclusion in the environments of child processes started afterward. Exporting does not make a value global across the machine, and it never travels backward into a parent process (a lot of confusing shell scripts come from expecting that it does).

Run:

COURSE_STAGE="linux"
python3 -c "import os; print(os.getenv('COURSE_STAGE'))"

Expected output:

None

The variable exists in your current shell, and it was never handed to the child process. Now change exactly one thing:

export COURSE_STAGE
python3 -c "import os; print(os.getenv('COURSE_STAGE'))"

Expected output:

linux

The observation is that the child got the value only after the export. The conclusion we can draw is narrower than it looks: exported variables become part of the environment given to child processes started afterward, and nothing more than that.

Clean up that variable:

unset COURSE_STAGE subject

How the shell finds a command

The first word of a command still needs an identity before anything can happen. Some names refer to behavior built into the shell itself, and others refer to executable files sitting on your disk. The shell has to decide which one it is dealing with first.

Ask the shell what two command names mean:

type cd
type python3

Typical evidence:

cd is a shell builtin
python3 is /usr/bin/python3

cd has to change the current shell’s own working directory, so it’s implemented inside the shell. python3 normally resolves to an executable file instead.

Think about why that has to be true. An external cd program would be completely useless for navigation, because it could change its own directory, exit, and leave your shell exactly where it started. A shell builtin can change the state of the shell you’re still sitting in.

Print the search list used for command names:

printf '%s\n' "$PATH"

PATH is a colon-separated list of directories, and I’d recommend leaving it completely alone for this lesson (nothing here needs it changed). Use type command-name to observe what the shell actually decided rather than guessing from the variable.

The shell searches those directories left to right whenever a command name has no slash in it, and the first matching executable normally wins. That ordering is useful, and it is also why editing PATH carelessly can quietly select the wrong program or make familiar commands vanish. type reports the decision the shell actually made, so use it to check your prediction from PATH.

Practice: change one shell decision

Pick a three-word value and run it through show-args.py twice, once unquoted and once quoted (something with obvious word boundaries works best). Compare the argument counts and values. Quoting should keep the three words together as one argument.

Then create a variable named EVIDENCE_MODE, check whether a child Python process can see it, export it, and check again. Clean it up with unset when you’re done.

Common failures and recovery

• python3: can't open file: run cd ~/underdunn-course/exercises/shell-basics, then check ls -l show-args.py. You’re almost certainly in the wrong directory.
• Python reports a syntax error: replace the whole file with the complete source block above, rather than trying to repair several lines at once.
• A variable reaches a child when you didn’t expect it to: it was probably already exported. Run unset VARIABLE_NAME, assign it again, and repeat the test.
• A command can’t be found after you experimented with PATH: close the terminal and open a fresh one. Never paste a guessed PATH value in from another system.
• The arguments differ from your prediction: write the difference down before you rerun anything. That unexpected argument list is exactly the evidence this lesson exists to produce.

Check your result

In the quoted-path example, the path with spaces should arrive as one argument. In the export example, the child sees the variable only after it is exported. If either result differs, compare the quoting and export step with the example above.