Reverse engineering
Get from an unknown executable to the one function that decides whether your input is right, and recognise which of four shapes the check takes.
By the end you can
- Establish format, architecture and whether a binary is stripped, and say what each implies
- Use strings and cross-references to find the check function without reading from main
- Classify a flag check as direct comparison, transform-then-compare, hash-then-compare, or a constraint system
- Invert a transform-then-compare check using the constants already in the binary
- Say when a decompiler helps and when its output is a reconstruction to be distrusted
1. Read
Reading a binary you have never seen before
A reversing challenge hands you an executable and no question. The order that gets you to the check function fastest - protections, strings, symbols, cross-references - and the four comparison patterns that account for most flag checks.
Reversing managed code: .NET, Java, and Python bytecode
When the binary is not machine code but bytecode with names attached, the job stops being disassembly and becomes reading. Decompiling .NET and Java back to source, disassembling .pyc, and what obfuscators actually take away.
Reversing WebAssembly: a stack machine in the browser tab
Reading .wasm as text, finding the exported function that checks your flag, following the linear memory that holds the strings, and why the JavaScript glue is usually where the answer is.
2. Use the tools
In the order they come up while solving. Each opens the workspace tab that runs it.
ELF, PE and Mach-O binary analyzer
Parse headers, sections, imports, and symbols from Linux, Windows, and macOS binaries, with C++ symbol demangling and gadget discovery.
Strings extractor for binaries and blobs
Pull printable ASCII and UTF-16 strings out of any file, filtered by length, with flag-pattern highlighting.
ROP gadget finder
Search a binary for return-oriented programming gadgets, filter by the registers they touch, and exclude ones containing bad bytes.
Java .class disassembler
Disassemble JVM bytecode, read the constant pool, and recover strings and logic from .class and .jar files.
Python .pyc bytecode disassembler
Disassemble compiled Python, read code objects and constants, and recover logic from a .pyc without running it.
3. Try one now
Generated in your browser and checked in your browser. No account, nothing to download, and a fresh one whenever you want another.
4. Practise on the real thing
Real picoCTF challenges that use these techniques, easiest first. 49 match in total - see the full index.
Checkpoint
Given a stripped 64-bit binary that transforms input before comparing it, name the transform, invert it, and produce the flag without running the binary.
Teaching note
Students start at main and read forward, which is the slowest possible route. Make the first exercise strings-plus-cross-references only, with the disassembler closed - once they have found a check function that way they rarely go back to reading linearly.