Function Calls and the Stack

How Programs WorkLesson 5 of 790 minutesLessonBuilds on: Machine Instructions and RegistersNo lab next

Learning objectives

By the end of this lesson

• Explain why call state cannot live in one fixed place per function.
• Explain why a calling convention is required.
• Identify common x86-64 argument and return-value registers.
• Describe what call, ret, rsp, and rbp contribute.
• Connect local C variables to stack storage without claiming every local always lives there.

Where does a function put things?

Each active function call needs storage for the values it still uses and a way to return to its caller. Some values can stay in registers, but registers are limited and shared by the functions executing on that thread. A caller may need to preserve a value while another function uses the same register.

So try the obvious answer. Give each function a fixed area of memory picked once at compile time, and let it keep its locals and its return location there. That would give us somewhere to store the values, but what happens if another call to the same function begins before the first one returns?

The trouble shows up the moment main calls classify and something inside classify calls classify again. The second call writes its locals over the first call’s, the first call’s return location goes with them, and the outer call eventually returns into whatever the inner one left lying around. An unfinished call still needs its state, so the storage has to belong to the call and not to the function.

The fix follows from the shape of the problem. Hand out storage in the order calls happen, then take it back in the reverse order, and everything lines up. For ordinary nested calls, the innermost call returns before the caller resumes. That’s a stack, and it’s the [stack] mapping you already found in /proc/PID/maps two lessons ago. Each call gets fresh space on the way in and gives it back on the way out.

In this stack model, nested calls keep separate local state until each call returns

Step 1main runs

mainlocals + return address

One call, one block.

Step 2main calls classify

main

classify · call 1its own locals + return address

The new block goes below, toward lower addresses.

Step 3classify calls classify

main

classify · call 1

classify · call 2separate storage, same function

Two calls to one function, two blocks.

Step 4the inner call returns

main

classify · call 1still intact, still needed

call 2 spacehanded back

Call 1 still has its saved state.

That’s what turns recursion from a special case into an ordinary one. Each block is drawn as one opaque unit here, and the rest of the lesson pries one open. The picture also leaves out a limit worth knowing about. Stack space is limited. Recursion that keeps adding frames can exhaust it and lead to the SIGSEGV described in the virtual-memory lesson.

Why the two ends of a call need an agreement

Storage is half the problem. The other half is coordination. A caller and its callee, meaning the function it calls, are usually compiled separately, sometimes years apart and by different compilers, and the two never get a chance to negotiate. main has to leave the argument somewhere classify will look for it, and classify has to leave its result somewhere main will read. If they disagree about the register, the callee can read the wrong value without any warning at the call itself.

So the answer gets fixed in advance and written down, and everything built for the platform follows the same one. That agreement is an application binary interface, or ABI. On 64-bit Linux, the System V AMD64 ABI passes the first integer or pointer arguments in rdi, rsi, rdx, rcx, r8, and r9, and integer return values come back in rax.

An ABI settles more than calls, so the part we’ve just described has its own name. The calling convention is the rules for one function calling another: registers for the arguments, a landing place for the return value, and responsibility for preserving things. You’ll meet the phrase in compiler documentation and in every disassembly discussion from here on, and it’s the reason two functions compiled a decade apart still manage to agree.

Arguments beyond those available registers go on the stack, and floating-point values follow a further set of rules again. We’ll introduce the exceptions as we actually need them, so you can connect each rule to a call you’re actually examining.

The maintained System V AMD64 ABI specifies the argument, return, and preservation conventions for this platform. It is the reference to consult when a function needs more registers than this example.

rsp, call, and ret

The processor tracks the current top of that region in rsp, and on x86-64 the stack grows toward lower addresses, so pushing data decreases rsp while removing data increases it.

call places a return address on the stack and transfers execution to its target. ret removes that address and resumes execution there. That saved address is really the whole explanation for how nested functions find their way back to the right caller.

Compilers will often use rbp as a stable frame pointer while rsp moves around underneath it. Optimized code often drops that frame pointer entirely and keeps the locals in registers instead (which is why optimized listings look so different). A stack-frame diagram is a useful model, and it isn’t a law requiring every function to look identical.

Inspect a call

Reuse the unoptimized decision program from the previous lesson:

objdump -d -Mintel decision | less

Find main first, then the call to classify inside it. Immediately before that call, find how the value 12 reaches the first argument register. Inside classify, look at whether the compiler bothers storing that argument in stack memory at all. And near the end, you’ll find the return value sitting in eax, followed by the ret instruction.

Before you read the annotated walk-through, predict the first integer argument register, the effect call has on rsp, and which register will carry the integer return value. Compare your prediction with the listing.

Annotated call walk-through

mov edi,0xc          ; first integer argument = 12
call classify        ; save next instruction address, jump
mov esi,eax          ; use classify's return value
lea rdi,[format]     ; first argument for printf
mov eax,0            ; variadic-call convention detail
call printf

Put that together with the callee and the evidence chain reads constant 12 → edi → [rbp-4] → cmp/branch → eax=14 → caller uses eax.

Address-by-address call state

Say rsp is 0x7fffffffdff0 before the call, and the next instruction sits at 0x401185:

Moment	rsp	State change
before call	0x7fffffffdff0	edi=12
return saved	0x7fffffffdfe8	memory there holds 0x401185
after push rbp	0x7fffffffdfe0	old rbp saved
after mov rbp,rsp	0x7fffffffdfe0	stable frame base
local write	unchanged	four bytes at 0x7fffffffdfdc (rbp-4) hold 12
after pop rbp	0x7fffffffdfe8	caller rbp restored
after ret	0x7fffffffdff0	execution resumes at 0x401185

Stack addresses decrease as state gets pushed onto it, which surprises a lot of people the first time. Ordinary memory access doesn’t move rsp at all, and some instruction has to go and update it explicitly.

call and the function prologue move rsp toward lower addresses while preserving return and frame state

0x...dff0

caller statersp before call · edi=12

lower addresses

0x...dfe8

saved return addressplaced by call · removed by ret

0x...dfe0

saved rbpplaced by push rbp

0x...dfdc

local value = 12four bytes at rbp-4

Checkpoint

Reconstruct one real call

Use a call-site address and its encoded length to calculate the return address. Check it against the next instruction’s address. Follow the relative rsp changes; a scratch sketch can help keep the pushes and pops straight.

Modification exercise

Add combine(int first, int second, int third) returning first + second * third. Find the three argument registers and follow one return through the function. Then build with -O2 and compare which instructions remain. An annotated listing is optional.

Confirm the syntax before you assign operand roles, don’t label stack slots without an access supporting them, and rebuild with -O0 -g whenever optimization removes the thing you were trying to look at.

Checkpoint

Draw the evidence, not a generic stack

Draw only the state your own disassembly supports: the caller, the saved return address, the frame pointer if there’s one, and any stack slots actually referenced. Keep instruction addresses labeled separately from stack addresses.

Change optimization, then compare

Build an optimized version:

gcc -O2 -g -o decision-o2 decision.c
objdump -d -Mintel decision-o2 | less

The compiler may well inline classify, simplify the arithmetic, or work the result out before the program ever runs. None of that is cheating. The compiler’s job is preserving the required observable behavior, and it was never preserving a nice teaching diagram for us.

Check which functions and stack operations survived that. The comparison shows why we need to inspect the actual build before describing its calls or stack layout.

Common failures and recovery

• No named classify function turns up: rebuild the learning target with -O0 -g, because optimization has probably inlined it.
• Your return-address calculation is wrong: use the address immediately after the complete call instruction, including its encoded length.
• A generic stack diagram disagrees with you: keep only the slots and register changes your actual instructions support.
• The source and the listing disagree: remove the binary, rebuild from the recorded source, and check its timestamp.

What this lets us investigate

You can now follow arguments into a function, find its return path, and explain how stack state supports nested calls. Next we’ll compare stack lifetime against heap allocation and static storage.

Check your understanding

Knowledge check

Question 1 of 2

An x86-64 near call begins at 0x401120 and occupies five bytes. Which return address is saved?

0x4011250x401120The value in rsp

Check answer