Reverse engineering
Get from an unknown executable to the one function that decides whether your input is right, and recognise which of four shapes the check takes.
By the end you can
- Establish format, architecture and whether a binary is stripped, and say what each implies
- Use strings and cross-references to find the check function without reading from main
- Classify a flag check as direct comparison, transform-then-compare, hash-then-compare, or a constraint system
- Invert a transform-then-compare check using the constants already in the binary
- Choose between static reading and running it under a debugger, and say what each answers faster
- Read managed bytecode - JVM, .NET, Python, WebAssembly - back to something close to source
- Say when a decompiler helps and when its output is a reconstruction to be distrusted
1. Read
Reading a binary you have never seen before
A reversing challenge hands you an executable and no question. The order that gets you to the check function fastest - protections, strings, symbols, cross-references - and the four comparison patterns that account for most flag checks.
Watching a binary run: dynamic analysis for reversing
Static disassembly tells you what a binary can do; running it tells you what it actually does. Tracing syscalls and library calls, breaking at the right moment in a debugger, and when to escalate to instrumentation or symbolic execution.
Reversing managed code: .NET, Java, and Python bytecode
When the binary is not machine code but bytecode with names attached, the job stops being disassembly and becomes reading. Decompiling .NET and Java back to source, disassembling .pyc, and what obfuscators actually take away.
Reversing WebAssembly: a stack machine in the browser tab
Reading .wasm as text, finding the exported function that checks your flag, following the linear memory that holds the strings, and why the JavaScript glue is usually where the answer is.
2. Use the tools
In the order they come up while solving. Read what each one does, or go straight to the workspace tab that runs it.
- ELF, PE and Mach-O binary analyzer
Parse headers, sections, imports, and symbols from Linux, Windows, and macOS binaries, with C++ symbol demangling and gadget discovery.
Open it in the workspace - Strings extractor for binaries and blobs
Pull printable ASCII and UTF-16 strings out of any file, filtered by length, with flag-pattern highlighting.
Open it in the workspace - Java .class disassembler
Disassemble JVM bytecode, read the constant pool, and recover strings and logic from .class and .jar files.
Open it in the workspace - Python .pyc bytecode disassembler
Disassemble compiled Python, read code objects and constants, and recover logic from a .pyc without running it.
Open it in the workspace - WebAssembly disassembler
Convert a .wasm module to readable WAT, list exports and imports, and follow the control flow.
Open it in the workspace - Endianness converter and byte swapper
Swap byte order for 16-, 32-, and 64-bit values, with a big-integer preview - the conversion every memory address needs before it goes in a payload.
Open it in the workspace - Struct pack and unpack (p32, p64, u32, u64)
Convert integers to little-endian byte strings and back at 8, 16, 32, and 64 bits - the pwntools p32/p64 helpers without the install.
Open it in the workspace - Hex viewer and hexdump
Inspect any file byte by byte with a side-by-side hex and ASCII view, offsets, and structure highlighting.
Open it in the workspace
3. Try one now
Generated in your browser and checked in your browser. No account, nothing to download, and a fresh one whenever you want another.
4. Practise on the real thing
Real picoCTF challenges that use these techniques, easiest first. 101 match in total - see the full index.
Common mistakes
The wrong turns this topic reliably produces. Written as the mistake rather than the rule, because the rule is easy to agree with and easy to walk straight past.
- Starting at main and reading forward. Strings and cross-references reach the check function in a minute; linear reading takes an hour and usually stops short.
- Believing decompiler output. It is a reconstruction: types are guessed, and a wrong signature silently changes what the code appears to do.
- Patching the check instead of solving it. It gets you past a prompt, but the flag is normally derived from the input rather than printed after it.
- Reversing library code as if it were the author's. A statically linked binary carries whole libc functions with no symbols left on them, and an hour can go into a custom transform that turns out to be memcmp.
- Reading a strings sweep that found nothing as proof the binary is hard. A keygen or a constraint system derives the flag from your input rather than storing it, so an empty sweep is evidence about the shape of the check.
Checkpoint
Given a stripped 64-bit binary that transforms input before comparing it, name the transform, invert it, and produce the flag without running the binary.
Teaching note
Students start at main and read forward, which is the slowest possible route. Make the first exercise strings-plus-cross-references only, with the disassembler closed - once they have found a check function that way they rarely go back to reading linearly.
Go deeper
The lessons above are written to get you through a challenge. These go after the subject instead. Each one opens our notes on that chapter - what it argues, what to take from it and where it stops - so this is somewhere to read now rather than a book to buy first. Nothing here is affiliate-linked or sold by us.
Chapter 2, The ELF Format
Practical Binary Analysis - Dennis Andriesse
The format in the detail that makes a stripped binary readable rather than intimidating.
Chapter 6, Disassembly and Binary Analysis Fundamentals
Practical Binary Analysis - Dennis Andriesse
Why disassemblers disagree, and what linear sweep and recursive descent each get wrong.
Chapter 6, Application Reverse Engineering
Attacking Network Protocols - James Forshaw
Reversing pointed at a specific question rather than at a whole binary, which is the CTF posture.
Chapter 6, Hybrid Analysis in Reverse Engineering
From Day Zero to Zero Day - Eugene Lim
Combining static and dynamic evidence deliberately, instead of switching tools when stuck.
Every book the curriculum cites has a page in the library.