Reading a binary you have never seen before

The instinct on being handed a binary is to open a disassembler and start reading from main. It is almost always the slowest route. A reversing challenge has exactly one interesting function - the one that decides whether your input is right - and there are three or four ways to find it that take under a minute each.

What follows is the order, and the reason for each step being where it is.

Step 1: what kind of thing is this

Before anything else, establish the format, the architecture, and whether it is stripped. If it turns out to be bytecode for a managed runtime - .NET, JVM, Python - stop here and decompile instead; none of the rest of this post is the fastest route. Those three facts change every decision after them - a stripped static Go binary and a dynamically linked C binary with symbols are not the same problem at all.

file ./chal
# ELF 64-bit LSB pie executable, x86-64, dynamically linked,
# BuildID[sha1]=..., for GNU/Linux 3.2.0, not stripped

checksec --file=./chal          # NX, PIE, canary, RELRO
nm -C ./chal | head             # symbols, if any survived
strings -n 8 ./chal | head -50

• Not stripped means the function names are still there, and the one called check_flag or validate is your entry point. Read nm before you read code.
• Statically linked and enormous (several megabytes) usually means Go or Rust. Both have their own conventions - Go keeps function names in a runtime table even when stripped, and Rust's symbols are manglings you can demangle.
• Packed shows up as a tiny section table, very high entropy, and almost no strings. UPX identifies itself; anything else needs unpacking at runtime.

Step 2: strings, and then what references them

Strings is the highest-yield step in reversing and the most commonly under-used, because people run it, skim it, and move on. The value is not in the strings themselves - it is in the cross-references to them.

A binary that prints "Correct!" and "Nope, try again" contains both strings, and the function that references them *is* the check function. You have found the one interesting function in the program without reading a single instruction.

Two traps. First, strings defaults to a minimum length of four, which buries the output in noise: use -n 8 or higher and read the whole thing. Second, it only finds ASCII by default - a Windows binary keeps most of its strings as UTF-16, and strings -el is what sees them.

Step 3: the four shapes a flag check takes

Once you are looking at the check function, it will almost certainly be one of four things. Recognising which one you have is most of the work.

The second shape is worth dwelling on because it is where people lose time. A loop that XORs each byte with a key and compares to a table is not an encryption you have to break - it is a bijection you have to run backwards, and the answer is in the binary already. The moment you can name the transform, you are done.

Step 4: the decompiler, last rather than first

A decompiler is enormously useful and a bad place to start. Its output is a *reconstruction*, and reading a reconstruction of the wrong function costs the same as reading the right one. Once steps one to three have told you which function matters, the decompiler turns twenty minutes of assembly into two minutes of C.

Read its output with the reconstruction in mind. Types are inferred and often wrong; a char* that should be a struct pointer produces nonsense indexing. Variables named iVar3 carry no information. And a decompiler will happily present dead code from a branch that never executes as though it were part of the logic.

When static reading stalls

Some binaries resist static analysis by design - packed, obfuscated, or generating their check at runtime. That is where watching it run takes over, and where the anti-analysis catalogue tells you what you are fighting. The signal is that the static picture does not account for the observed behaviour: the program clearly compares something, and there is no comparison in the code you can see.

• Run it under a debugger and break on the comparison. ltrace alone solves a surprising number of challenges by showing the strcmp argument.
• Dump memory after unpacking. A packed binary is unpacked in memory by the time main runs; a dump at that point is the real program.
• Look for anti-debugging before assuming your setup is broken. ptrace(PTRACE_TRACEME) returning -1, timing checks, and /proc/self/status reads are the usual three, and all of them are easy to patch out once you know they are there.

The order, condensed

1. file, checksec, nm - what am I holding, and did the symbols survive?
2. strings -n 8, then cross-references to anything interesting.
3. Identify which of the four check shapes you have.
4. Decompile only the function you have already identified.
5. If the static picture does not explain the behaviour, run it.