Shellcode & Calling Conventions

Code With No Fixed Address

Once you redirect RIP (see Stack Overflows), you need something for it to land on. Shellcode is a small, self-contained machine-code payload written to run correctly no matter where in memory it ends up — because you rarely know the exact address in advance, and ASLR guarantees you won’t.

Position-Independent by Necessity

Ordinary compiled code can reference globals and strings by absolute address because the loader fixes those addresses at load time. Shellcode has no loader doing it favors — it’s dropped into whatever memory the overflow landed it in. So it:

• Never hardcodes absolute addresses of its own data (a string it needs, for example). Instead it computes its own location at runtime, often with a call/pop trick: call pushes the address of the next instruction, and a following pop retrieves it into a register.
• Resolves any function it needs (like a libc function or a syscall) either by calling the kernel directly via syscall, or by walking loader structures at runtime to find a function’s address — it can’t just link against it.

The Linux x86-64 Syscall Convention

Shellcode talks to the kernel directly, bypassing libc entirely, using the syscall instruction. The calling convention is fixed:

Register	Role
rax	Syscall number (which syscall to invoke)
rdi	Argument 1
rsi	Argument 2
rdx	Argument 3
r10	Argument 4 (note: not rcx — syscall clobbers rcx and r11)
r8	Argument 5
r9	Argument 6

A classic execve("/bin/sh", NULL, NULL) payload sets rax = 59 (the execve syscall number), rdi to a pointer to the string "/bin/sh", and rsi/rdx to zero, then executes syscall. Every instruction in that sequence has to be built without touching anything the loader would normally hand you — including the string itself, which the shellcode typically constructs on the stack.

Why No Null Bytes

Shellcode frequently travels as a C string — through strcpy, sprintf, or similar functions that stop copying at the first 0x00 byte. A single null byte anywhere in the payload truncates it before it reaches its target. This forces awkward instruction choices: instead of mov eax, 59 (which encodes with high zero bytes for a small immediate), shellcode authors use xor eax, eax followed by mov al, 59, trading a null-free instruction for one extra byte. Every constant, every register clear, gets rewritten to avoid 0x00 appearing anywhere in the encoded bytes.

The NOP Sled

Even with a computed offset (see e1’s 56-byte calculation), hitting the exact start address of your shellcode can be unreliable — stack layout can shift by a few bytes depending on environment variables, argv, or compiler version. A NOP sled is a run of 0x90 (nop) instructions placed before the real payload:

[ nop nop nop nop ... nop ][ shellcode ]
 ^ return address can land anywhere in here and still slide into the payload

If the overwritten return address lands anywhere inside the sled, execution just slides forward, one no-op at a time, until it reaches the real shellcode. It converts a precise-address problem into a land-anywhere-in-a-wide-region problem — at the cost of a payload that’s easy to fingerprint (a long run of 0x90 is a strong signal for any scanner or IDS).

Takeaway

Shellcode can’t assume anything about where it lives — it computes its own position, talks to the kernel directly, and can’t contain the one byte (0x00) most string functions treat as “stop.”