LAB-19Medium

Stack Smash Analysis

Given a vulnerable function's disassembly, determine exactly how many bytes of overflow it takes to reach the saved return address.

20 min5 prerequisites6 tools

Disassembly

00:  push rbp
01:  mov rbp, rsp
02:  sub rsp, 0x30
03:  lea rax, [rbp-0x30]
04:  mov rdi, rax
05:  mov rsi, [rbp+0x10]
06:  call strcpy
07:  leave 
08:  ret

Hints

Hint 1

lea rax, [rbp-0x30] tells you exactly where the buffer starts relative to rbp — not rbp-0x20, even though the source only declared a smaller array. Compilers round stack allocations up.

Hint 2

The stack grows downward, but writing INTO a buffer via strcpy fills upward — toward the saved RBP and the return address above it.

Hint 3

Offset to the return address = distance from the buffer start to rbp (0x30) + 8 more bytes for the saved RBP slot itself.

Submit Findings

Prerequisites

  • e1-stack-overflows
  • a4-stack-functions
  • Stack frame layout (prologue/epilogue)
  • Buffer overflow mechanics
  • Stack canaries /GS

Methodology

Investigation Methodology

1. Reconstruct Stack Frame

Higher addresses
+------------------+ <-- rbp + 0x10 (arg, e.g., [rbp+0x10])
| ...              |
+------------------+ <-- rbp + 0x08 | Return address |
+------------------+ <-- rbp + 0x00 | Saved RBP      |
+------------------+ <-- rbp - 0x30 | Buffer start   | (0x30 = 48 bytes)
+------------------+ <-- rbp - 0x40 | (alignment padding if any)
Lower addresses

2. Calculate Overflow Offset

  • Buffer starts at rbp - 0x30 (48 bytes from frame base)
  • strcpy writes upward (toward higher addresses)
  • Must cross:
    1. 48 bytes of buffer
    2. 8 bytes of saved RBP
  • Total = 56 bytes to reach return address

3. leave Instruction

leave
; Equivalent to:
mov rsp, rbp    ; Deallocate locals, RSP = RBP
pop rbp         ; Restore caller's RBP

Then ret pops return address from [RSP] (which now points to it).

4. Mitigations

Mitigation How It Stops This
Stack Canary (/GS) Random cookie between buffer and saved RBP. Checked before ret. Overflow overwrites canary → detected.
DEP/NX Stack not executable. Shellcode on stack won't run.
ASLR Stack/base addresses randomized. Hard to guess return address target.
CFG Indirect calls validated. ret not directly checked but call targets are.
SafeSEH Exception handlers validated.
Control Flow Guard Valid indirect call targets only.

5. Exploitation Pattern

  1. Overflow buffer → overwrite saved RBP (garbage) → overwrite return address
  2. Return address = address of shellcode / ROP gadget
  3. If canary: leak or brute-force canary first
  4. If DEP: ROP chain (reuse existing code)
  5. If ASLR: info leak to defeat randomization

Common Pitfalls

  • Confusing `sub rsp, 0x30` = 48 bytes (not 30)
  • Forgetting the 8-byte saved RBP slot between buffer and return address
  • Assuming buffer is exactly at `rbp-0x20` — compiler rounds up (alignment)
  • Not accounting for shadow space (32 bytes) in offset calculations
  • Missing that `strcpy` stops at NULL byte — exploit must avoid 0x00

Tools

  • Ghidra (Stack view, Function graph)
  • Binary Ninja (Stack layout)
  • x64dbg / GDB (step through, watch stack)
  • pwntools / ROPgadget (exploit dev)
  • Checksec (mitigation check)
  • ROP chain generators

References

  • Lesson E1: Stack Overflows
  • Lesson A4: Stack & Functions
  • Smashing the Stack for Fun and Profit (Aleph One)
  • Windows /GS Implementation
  • ROP Emporium (practice)

Extensions & Challenges

  • Calculate offset for a buffer at `rbp-0x28` with `strncpy` (size limit)
  • Build a ROP chain to bypass DEP for this binary
  • Research: Stack pivoting — moving RSP to controlled buffer
  • Practice: Exploit a real binary with canary + ASLR + DEP (ROP Emporium)
  • Write a stack overflow detector using binary analysis (pattern search)