Function Calls and the Stack

How Programs WorkLesson 5 of 775 minutesLab included

Learning objectives

By the end of this lesson

• Explain why a calling convention is required.
• Identify common x86-64 argument and return-value registers.
• Describe what call, ret, rsp, and rbp contribute.
• Connect local C variables to stack storage without claiming every local always lives there.

Functions need an agreement

Separate compiled functions must agree about where arguments arrive, where results leave, which registers must survive, and how control returns. That agreement is an application binary interface, or ABI. On 64-bit Linux, the System V AMD64 ABI normally passes the first integer or pointer arguments in rdi, rsi, rdx, rcx, r8, and r9; integer return values use rax.

Arguments beyond the available registers use the stack. Floating-point values follow additional rules. We will introduce exceptions when we need them rather than attempting to memorize an ABI document as punishment.

The stack is process memory with a convention

The stack is a writable memory region used for call and temporary state. Register rsp points near its current top. On x86-64, pushing data decreases rsp; removing it increases rsp.

call places a return address on the stack and transfers execution to its target. ret removes that address and resumes there. Corrupting saved control data can therefore change where execution continues, which is why stack memory matters to exploit development.

Compilers may use rbp as a stable frame pointer while rsp moves. Optimized code often omits that frame pointer and keeps locals in registers. A stack-frame diagram is a useful model, not a law requiring every function to look identical.

Inspect a call

Reuse the unoptimized decision program from the previous lesson:

objdump -d -Mintel decision | less

Find main, then the call to classify. Immediately before the call, locate how the value 12 reaches the first argument register. Inside classify, observe whether the compiler stores that argument in stack memory. Near the end, find the return value in eax and the ret instruction.

Checkpoint

Draw the evidence, not a generic stack

Draw only the state supported by your disassembly: the caller, saved return address, frame pointer if present, and any stack slots actually referenced. Label instruction addresses separately from stack addresses.

Knowledge check

Question 1 of 3

Where does the first integer argument normally arrive on x86-64 Linux?

rdiraxrsp

Check answer

Change optimization, then compare

Build an optimized version:

gcc -O2 -g -o decision-o2 decision.c
objdump -d -Mintel decision-o2 | less

The compiler may inline classify, simplify arithmetic, or compute the result before runtime. This is not cheating. The compiler’s job is to preserve required observable behavior, not preserve a pleasant teaching diagram.

Record which functions and stack operations survived. The difference demonstrates why researchers trust the binary in front of them more than assumptions based on one source build.

What this gives us as researchers

You can now follow arguments into a function, locate its return path, and explain how stack state supports nested calls. Next we will compare stack lifetime with heap allocation and static storage.