Binary exploitation

Before the session

• Nothing to install. Students need a browser and https://ctfpal.com. Confirm the room can reach it once; after that it works offline.
• Read the lesson yourself first - about 35 minutes.
• Have one worked example ready to paste. The classroom link builder on the instructor page turns it into a URL that opens preloaded.

Objectives

Written as things a student can do afterwards, so they can be assessed rather than asserted.

1. Read NX, PIE, canary, and RELRO from a binary and say what each rules out
2. Find an overflow offset with a de Bruijn pattern in a single crash
3. Build a ret2libc chain, including stack alignment
4. Turn a leaked pointer into a libc base and resolve arbitrary symbols
5. Turn a format string into an arbitrary read and then an arbitrary write
6. Name the four heap bugs that matter and the allocator behaviour each one abuses
7. Write shellcode that survives the challenge's constraints - length, character set, and seccomp

Running order (150 min)

Tools used

• ELF, PE and Mach-O binary analyzer - Parse headers, sections, imports, and symbols from Linux, Windows, and macOS binaries, with C++ symbol demangling and gadget discovery.https://ctfpal.com/?tool=elf-pe-analyzer
• Buffer overflow offset finder - Find the exact number of bytes before the saved return address using a de Bruijn pattern and the value from a single crash.https://ctfpal.com/?tool=buffer-overflow-offset
• Cyclic pattern generator and offset finder - Generate a de Bruijn sequence and find the exact overflow offset from a crashed register value - the pwntools cyclic workflow, in the browser.https://ctfpal.com/?tool=cyclic-pattern-generator
• ROP gadget finder - Search a binary for return-oriented programming gadgets, filter by the registers they touch, and exclude ones containing bad bytes.https://ctfpal.com/?tool=rop-gadget-finder
• ROP chain and payload builder - Assemble an exploit payload from padding, addresses, and raw bytes, with a live hexdump and offset ruler - the pwntools flat() workflow.https://ctfpal.com/?tool=rop-payload-builder
• Libc base address calculator - Turn a leaked libc pointer into the library’s base address, then resolve any other symbol - the arithmetic every ret2libc exploit runs on.https://ctfpal.com/?tool=libc-base-calculator
• Format string exploit builder - Find your input’s position on the stack and build %n write primitives - turning an uncontrolled printf into an arbitrary read and write.https://ctfpal.com/?tool=format-string-exploit
• Glibc heap exploitation helper - Model glibc chunk layout, bin behaviour, and tcache - the size classes and metadata that heap challenges turn on.https://ctfpal.com/?tool=heap-exploitation-helper
• Shellcode assembler and library - Assemble x86 and x86-64 shellcode, or pick a ready execve(/bin/sh) payload, with null-byte-free variants and length reporting.https://ctfpal.com/?tool=shellcode-assembler
• Struct pack and unpack (p32, p64, u32, u64) - Convert integers to little-endian byte strings and back at 8, 16, 32, and 64 bits - the pwntools p32/p64 helpers without the install.https://ctfpal.com/?tool=struct-pack-unpack
• Pwntools exploit script generator - Generate a working pwntools template with the right context, process or remote connection, and the boilerplate every exploit repeats.https://ctfpal.com/?tool=pwntools-script-generator

Reading

• From crash to shell: stack overflows, offsets, ret2win, and ret2libc - 8 min. A segfault is not an exploit. The path from an unexpected crash to a controlled instruction pointer to a shell, with the mitigation checks that decide which technique you need and the stack-alignment detail that breaks working exploits.
• Integer bugs: overflow, signedness, truncation, and the off-by-one - 6 min. The length check that passes because the length wrapped, the negative index that survives a bounds check, and the 16-bit truncation that turns 65,540 into 4. Where arithmetic bugs come from and how to spot them in a decompiler.
• Format string bugs: when %n writes where you point - 5 min. A single printf(user_input) is a read and a write primitive in disguise. How %x leaks the stack, %s dereferences arbitrary pointers, and %n turns a logging bug into a controlled memory write - the whole ladder from leak to code execution.
• The glibc heap: chunks, bins, and the four bugs that matter - 9 min. What malloc actually stores, why tcache made modern heap exploitation easy, and how use-after-free, double free, tcache poisoning and a one-byte overflow each turn into an arbitrary write. The menu-driven heap challenge, decoded.
• Writing shellcode that fits: constraints, encoders, and seccomp - 7 min. The execve stub in twenty-three bytes, and what to do when the buffer is short, the bytes must be printable, nulls are forbidden, or seccomp has taken execve away. Shellcode as a constraint-satisfaction problem.

Practice set

Real picoCTF challenges tagged with this module’s techniques, easiest first. Assign the first three in class and the rest as homework.

1. format string 0 - picoCTF 2024, easy
2. heap 0 - picoCTF 2024, easy
3. PIE TIME - picoCTF 2025, easy
4. babygame01 - picoCTF 2023, medium
5. Binary Gauntlet 0 - picoCTF 2021, medium
6. Binary Gauntlet 1 - picoCTF 2021, medium
7. Binary Gauntlet 2 - picoCTF 2021, medium
8. Binary Gauntlet 3 - picoCTF 2021, medium
9. buffer overflow 0 - picoCTF 2022, medium
10. babygame02 - picoCTF 2023, hard
11. babygame03 - picoCTF 2024, hard
12. Bizz Fuzz - picoCTF 2021, hard

Checkpoint (gradeable)

Given a 64-bit binary with NX and no PIE, produce a working ret2libc exploit and explain each entry in the chain.

Deliberately a produced artefact rather than a quiz question: it is either there or it is not, which makes it fast to mark and hard to bluff. Every tool in ctfpal is deterministic, so two students who did the work correctly hand in the same value.

Where the room gets stuck

The alignment `ret` gadget is the single most common blocker: the exploit is correct and crashes inside `system` anyway. Warn about it in advance or students will conclude their whole chain is wrong.

• Forgetting the stack-alignment ret gadget on 64-bit. The chain is correct and crashes inside system anyway, and students conclude the whole thing is wrong.
• Hardcoding a libc address from your own machine. The remote libc differs; the leak exists so that the base is computed, not assumed.
• Counting the offset by hand from the source. A cyclic pattern gives it in one crash and does not care about padding, alignment or the compiler's opinions.
• Testing only under a debugger. GDB disables ASLR and rewrites the environment, so the stack layout differs from the one the target actually runs with and a working exploit can still be untested.
• Building a chain without checking which bytes survive the read. scanf stops at whitespace and strcpy at a null, so the primitive that gets your payload in defines the alphabet the whole chain has to live inside.

If a student wants the subject, not the answer

Chapter-level references, so a student can be pointed at twenty pages rather than at a book. Nothing here is required to complete the module.

• Hunting Security Bugs, Tom Gallagher, Bryan Jeffries, and Lawrence Landauer. Chapter 8, Buffer Overflows and Stack and Heap Manipulation. The bug from the finder's side - where overflows come from - rather than only how to ride one.
• Hunting Security Bugs, Tom Gallagher, Bryan Jeffries, and Lawrence Landauer. Chapter 9, Format String Attacks. Why a format string is a write primitive at all, stated more carefully than most references bother to.
• Practical Binary Analysis, Dennis Andriesse. Chapter 1, Anatomy of a Binary. The layout and loading model every one of these exploits is manipulating.
• A Bug Hunter's Diary, Tobias Klein. Chapter 4, NULL Pointer FTW. One real bug followed from discovery to working exploit, which is the arc a CTF challenge compresses.

If you finish early

• Hand out a challenge from the cross-CTF index in this category - each one has published solutions to compare afterwards.
• Run the same input through Identify and let the class argue with the ranking. Disagreeing with a confidence score is where the technique actually lands.
• Ask a student to break their own example - construct an input that defeats the tool, and explain why.