Triage at scale: hashing, similarity, and finding the odd sample

A particular kind of rev or forensics challenge does not hand you one binary - it hands you many. A folder of a hundred samples where one is the real payload and the rest are decoys. A set of files that are all 'the same malware' except one variant that carries the flag. A capture full of dropped executables. Reversing each by hand is exactly what the challenge is betting you will do. The faster path is to treat the pile as data: hash it, cluster it, and let the outlier fall out.

The techniques below come from real malware analysis, where analysts triage thousands of samples a day and cannot open each in a disassembler. The same tools collapse a CTF's decoy pile into a sorted set in seconds.

Cryptographic hashes: the exact-match layer

Start with the obvious. A SHA-256 of every file tells you which ones are byte-identical - though a Merkle-Damgard hash extends, so identical prefixes and identical files are not the same claim - deduplicating the pile and revealing that ninety of the hundred are literally the same file. It also lets you check each hash against VirusTotal or a known-sample database, which sometimes identifies the family (or the challenge's source) outright.

sha256sum * | sort | uniq -c -w64 | sort -rn
# Groups identical files. The counts tell you the decoy structure at a glance:
# 90 copies of one hash, 9 of another, and 1 unique file -> look at the unique one.

Fuzzy hashing: the similarity layer

Cryptographic hashes change completely when one byte changes, so they cannot tell you that two files are 99% the same. Fuzzy hashes are built to do exactly that: they produce a hash you can compare for similarity, giving a percentage rather than a yes/no. This is how you find the sample that is almost like the others but not quite - the modified one.

# ssdeep: context-triggered piecewise hashing, the classic.
ssdeep -r . > hashes.txt
ssdeep -m hashes.txt *              # match each file against the set, with %

# TLSH is the modern alternative - more robust, better at scale.
# A file that scores 95% against the herd but has a small unique region
# is where the planted difference (and the flag) lives.

Structural hashes: same code, different bytes

For executables specifically, you can hash structure rather than raw bytes, which catches samples that share code but were compiled or packed differently. The import hash (imphash) is a hash of a PE's import table in order - two binaries built from the same source have the same imports in the same order, so the same imphash, even if other bytes differ. Rich header hashes and section-layout hashes do the same job from other angles.

• imphash - identical import tables. Groups samples from the same toolchain/source even across recompiles.
• Rich header hash - the undocumented PE metadata Microsoft compilers embed; a strong fingerprint of the exact build environment.
• Section hashes - hash each section separately; a shared .text with a different .data isolates where the samples diverge.

YARA: describe the thing you are hunting for

Once you know what distinguishes the interesting sample - a string, a byte pattern, a specific import - a YARA rule turns that description into a scanner that finds every file matching it. YARA is the connective tissue of malware triage: you notice a feature in one sample and immediately ask 'which others have it?'.

rule has_flag_marker {
    strings:
        $a = "flag{" nocase
        $b = { 6a 00 68 ?? ?? ?? ?? e8 }      // a wildcarded byte pattern
    condition:
        any of them
}
// yara -r has_flag_marker.yar ./samples/

The triage order