Watching a binary run: dynamic analysis for reversing

There are two ways to understand a binary. Static analysis reads the instructions without running them - safe, complete, and slow, because you have to reason about every path yourself. Dynamic analysis runs the binary and watches what it does - fast and concrete, because the program resolves its own indirection, decrypts its own strings, and computes its own comparisons right in front of you. The reading-a-binary post covers the static side. This is the dynamic half, and for a lot of rev challenges it is the faster road to the flag.

The reason to reach for dynamic analysis first: a check that looks like a wall in the disassembler - an obfuscated comparison, a runtime-decrypted key, a value computed from the environment - is often trivial to just observe. You do not have to understand how the program builds the correct password if you can watch it build it and read the result.

Level 1: trace the calls, touch nothing

Before a debugger, run the two tracers. They require no setup and frequently answer the whole challenge. strace logs every system call - files opened, data read, network connections. ltrace logs every library call - and this is the one that wins CTF challenges, because the interesting comparison is usually a strcmp in libc.

ltrace ./crackme
# ...
# strcmp("hunter2", "s3cr3t_p4ss")  = -1     <-- the flag is right there

strace ./challenge 2>&1 | grep -E 'open|read|connect'
# reveals which file it reads the key from, or where it phones home

Level 2: the debugger, at the right moment

The skill in dynamic reversing is not knowing gdb commands - it is choosing where to stop. You do not want to single-step from main; you want to break at the one instruction where the secret is in a register or the comparison happens, and read state there. Use a debugger enhancement like GEF or pwndbg so every break shows registers, stack, and disassembly at once.

gdb ./challenge
(gdb) break strcmp          # or the specific address of the check
(gdb) run
(gdb) x/s $rsi              # the value being compared against yours
# or break just after a decrypt routine and dump its output buffer:
(gdb) break *0x401337
(gdb) run
(gdb) x/32xb $rax           # the freshly decrypted bytes

This is also how you defeat runtime string decryption. Many challenges store the flag or the check XORed or encrypted and decode it just before use, so strings finds nothing. You do not need to reverse the decryption - break immediately after the decode routine returns and dump the buffer it wrote. The program did the work; you just read the answer.

Level 3: instrumentation, for the whole run at once

When you need to observe every instruction rather than one breakpoint - to count how many times a loop runs, to log every byte compared, to trace which code executes for a given input - a debugger is too slow and too manual. Dynamic binary instrumentation injects your own code around each instruction as the program runs. Intel Pin and DynamoRIO are the tools; you write a small 'pintool' that, say, logs every cmp operand or records the full execution trace.

The CTF-flavoured use is automatic unpacking and coverage tracing. A packed binary decompresses itself into memory and jumps to the real code; an instrumentation tool can detect the moment execution enters freshly written memory and dump the unpacked payload right then. And a coverage trace - which basic blocks ran for input A versus input B - turns a check-the-flag-character-by-character binary into a side channel you can solve one byte at a time by watching where execution diverges.

Level 4: let a solver do the algebra

Some challenges compute a complex condition on your input - a nest of arithmetic and constraints that any single input either satisfies or does not. Rather than reverse the math, hand it to a symbolic execution engine like angr. You mark the input as symbolic, tell the engine which address means success and which means failure, and it explores paths and solves for an input that reaches success.

import angr, claripy
p = angr.Project("./challenge", auto_load_libs=False)
flag = claripy.BVS("flag", 8 * 32)                  # 32 symbolic bytes
st = p.factory.entry_state(stdin=flag)
sm = p.factory.simulation_manager(st)
sm.explore(find=0x40133c, avoid=0x401350)           # find "correct", avoid "wrong"
print(sm.found[0].posix.dumps(0))                    # the input that works

Choosing the level

1. strings, then ltrace/strace - free, and they solve naive crackmes outright.
2. gdb with GEF/pwndbg - break on the comparison or after a decrypt, read the state.
3. Pin/DynamoRIO - when you need every instruction: unpacking, coverage, loop counts.
4. angr - only when the check is a bounded constraint problem worth solving symbolically.