Anti-analysis tricks, and how reversing challenges use them

A whole genre of reversing challenge is built to fight back. The binary refuses to run under a debugger, behaves differently inside a VM, or arrives packed so that strings and the disassembler show you nothing but a decompression stub. These are the same techniques real malware uses to frustrate analysts, and challenge authors borrow them because they turn a five-minute crackme into a satisfying fight. The good news: each trick has a well-known shape and a well-known counter.

The meta-strategy for all of them is the same. Anti-analysis is a *check* - and a check you can patch, exactly as in managed-code challenges. The program asks a question about its environment and branches on the answer. Find the check, and you can either satisfy it, lie to it, or patch the branch so the answer no longer matters. You almost never need to make the environment genuinely clean; you need to make the check believe it is.

Anti-debugging

The program detects that it is being debugged and bails, or takes a fake path that never reaches the flag. On Windows the canonical check reads one byte the OS sets when a debugger attaches.

• IsDebuggerPresent / the PEB. The Windows API IsDebuggerPresent just reads BeingDebugged at offset 2 of the Process Environment Block. Malware reads the PEB directly to skip the API call. The counter: break on the check and flip the returned value, or patch the byte to zero.
• ptrace self-attach (Linux). A process can be traced by only one tracer at a time, so a program calls ptrace(PTRACE_TRACEME) on itself - if a debugger is already attached, the call fails, revealing it. Counter: LD_PRELOAD a stub ptrace that always returns success, or NOP the call.
• Breakpoint detection. The program scans its own code for 0xCC (the software-breakpoint byte) or checksums itself to notice edits. Counter: use hardware breakpoints, which leave no 0xCC, or break before the scan and skip it.

# Defeat a Linux ptrace check by pre-loading a fake:
cat > fake.c <<'EOF'
long ptrace(int r, int p, void *a, void *d) { return 0; }
EOF
gcc -shared -fPIC fake.c -o fake.so
LD_PRELOAD=./fake.so ./challenge     # the self-trace now "succeeds"

Anti-VM and anti-sandbox

Because analysts work in virtual machines, malware checks whether it is inside one and stays dormant if so. Challenges use the same trick to gate the interesting behaviour behind an environment check.

• Artefact scanning. Look for VM-specific device names, registry keys, MAC address prefixes (VMware, VirtualBox), or the vmware/vbox guest tools. Counter: patch the check, or run on bare metal.
• CPUID hypervisor bit. The CPUID instruction sets a bit that reveals a hypervisor, and can return the hypervisor's vendor string. Counter: break at the CPUID and rewrite the result register.
• Timing and sleep-skipping. Sandboxes give a program limited time, so malware sleeps past the analysis window, or times a benign operation and notices a debugger's slowness. Counter: patch out the sleep, or speed up virtual time.

Packing: when the binary is hiding itself

A packed binary contains a compressed or encrypted payload plus a small stub. At runtime the stub unpacks the real code into memory and jumps to it. Statically, you see the stub and a high-entropy blob - the disassembly is meaningless and strings is empty. This is the single most common reason a rev challenge 'has no code'.

# Spot it:
strings challenge.exe | grep -i upx      # UPX leaves its name in section headers
# Entropy near 8.0 across the file also screams packed.

# The easy case - a standard packer with a public unpacker:
upx -d challenge.exe                       # UPX unpacks itself

# The general case - dump after the stub runs:
#   run under a debugger, break at the "original entry point" the stub
#   jumps to (often the first instruction in freshly-written memory),
#   then dump the process memory and rebuild the imports.

The universal unpacking method does not care which packer was used: the real code has to exist in cleartext in memory at the moment the CPU executes it, so you let the stub run, catch execution the instant it enters the unpacked region, and dump memory there. A debugger with a hardware breakpoint on the target region, or a Pin tool that watches for execution of freshly-written pages, both get you the payload.

The general answer to all of it

When the challenge is a folder of samples rather than one binary, the question is which of them is unlike the others, and that is triage at scale instead. For a single binary, every technique here reduces to the same three options, in increasing order of effort: satisfy the check (give it the clean environment it wants), lie to the check (intercept the environment query and fake the answer), or patch the check (edit the branch so its outcome is ignored). Patching is the blunt instrument that always works when you can find the branch - and finding the branch is just dynamic analysis: run to the point where behaviour diverges, and look at the comparison that decided it.

1. Identify the defence: is the binary refusing to run, behaving oddly, or showing no code at all?
2. No code → it is packed. Unpack it (UPX) or dump at OEP, then start over on the real payload.
3. Refuses under a debugger → find the anti-debug check; LD_PRELOAD or patch it.
4. Behaves differently in a VM → find the environment check; rewrite its result.
5. When in doubt, patch the branch. The check only matters if the program is allowed to act on it.