Malware triage and defensive telemetry
Sort a pile of samples by similarity, defeat the tricks that stop them being analysed, and read the logs and telemetry that catch them.
By the end you can
- Triage a sample statically - imports, sections, entropy, strings - and predict what dynamic analysis will show
- Cluster related samples using import hashes and fuzzy hashing, and say what each measure actually compares
- Recognise packing from section entropy and an import table with almost nothing in it
- Name three anti-debug and three anti-VM checks, and the analysis change each one forces
- Read Windows event logs and process telemetry into a timeline that supports or refutes a hypothesis
- Explain what ETW and AMSI see, and where a defender's visibility ends
1. Read
Triage at scale: hashing, similarity, and finding the odd sample
When a challenge hands you a folder of a hundred binaries and one is different, reversing each by hand is the wrong move. Import hashing, fuzzy hashing, and YARA turn a pile of samples into a sorted, searchable set.
Anti-analysis tricks, and how reversing challenges use them
Malware-flavoured rev challenges borrow the real thing's defences: anti-debugging, anti-VM, timing checks, and packing. What each trick looks like in a disassembler and the one-line answer to each.
Log forensics and threat hunting for blue-team CTF
Defensive challenges hand you logs and ask what the attacker did. Building a timeline, following the kill chain, and using MITRE ATT&CK as a checklist to turn a pile of events into the story that holds the flag.
What the endpoint sees: ETW, AMSI, and userland hooks
Blue-team and detection challenges ask you to reason about how code gets watched on Windows. The three telemetry sources - ETW, AMSI, and inline API hooks - what each one records, and why evasion challenges target them.
2. Use the tools
In the order they come up while solving. Read what each one does, or go straight to the workspace tab that runs it.
- ELF, PE and Mach-O binary analyzer
Parse headers, sections, imports, and symbols from Linux, Windows, and macOS binaries, with C++ symbol demangling and gadget discovery.
Open it in the workspace - Strings extractor for binaries and blobs
Pull printable ASCII and UTF-16 strings out of any file, filtered by length, with flag-pattern highlighting.
Open it in the workspace - Office macro and OLE extractor
Extract VBA macros from DOCM, XLSM, and legacy OLE documents, and deobfuscate the string-concatenation tricks they hide behind.
Open it in the workspace - PowerShell deobfuscator
Peel -EncodedCommand, FromBase64String, [char] codes, format operators and backtick escapes off an obfuscated PowerShell command, and see what it actually does.
Open it in the workspace - Perceptual image hash: are these the same picture?
Compare two images by aHash, dHash and pHash in the browser - a re-encoded, resized or recompressed copy has a different checksum and nearly the same perceptual hash.
Open it in the workspace - MD5, SHA-1 and SHA-256 hash generator
Compute MD5, SHA-1, SHA-256, SHA-384, and SHA-512 of any text in the browser, using Web Crypto. Nothing is sent anywhere.
Open it in the workspace
3. Try one now
Generated in your browser and checked in your browser. No account, nothing to download, and a fresh one whenever you want another.
4. Practise on the real thing
Real picoCTF challenges that use these techniques, easiest first. 10 match in total - see the full index.
Common mistakes
The wrong turns this topic reliably produces. Written as the mistake rather than the rule, because the rule is easy to agree with and easy to walk straight past.
- Treating a hash match as identity and a hash mismatch as difference. A cryptographic hash answers 'the same file?'; imphash and ssdeep answer 'related?', which is the question triage is asking.
- Running an unpacked-looking sample and believing what it does. Sandbox-aware malware behaves for the sandbox, and the interesting branch is the one it did not take.
- Hunting in logs without a hypothesis. 'Look for something odd' returns everything; 'a service was created outside change windows' returns findings.
- Detonating a sample before recording what it is. Hashes, imports, sections and entropy take a minute and the sample cannot undo them; a live run on an unprepared host is a decision that cannot be taken back.
- Writing a YARA rule against a packed sample's bytes. Those change with every repack - a rule worth keeping matches the unpacked code, a configuration structure, or a habit of the builder.
Checkpoint
Given ten samples, group them by shared code, justify each grouping with the measure that produced it, and name the outlier that belongs to no group.
Teaching note
The instinct is to open one sample in a disassembler. Force the class to sort all ten first: triage is a comparative skill, and a student who has clustered a corpus reads any single sample faster afterwards.
Go deeper
The lessons above are written to get you through a challenge. These are the chapters to read when you want the subject instead - each one named so you can check it rather than take our word for it. Nothing here is affiliate-linked or sold by us.
Evasive Malware - Kyle Cucci
Chapter 10, Anti-Debugging
The checks named individually, with what each one is really measuring.
Evasive Malware - Kyle Cucci
Chapter 17, Packers and Unpacking Malware
How to get to the real code, which is a prerequisite for everything else in this module.
Malware Data Science - Joshua Saxe with Hillary Sanders
Chapter 5, Shared Code Analysis
The similarity measures behind clustering, explained well enough to know when each one lies.
Evading EDR - Matt Hand
Chapter 8, Event Tracing for Windows
Precisely what the defender's sensor sees, from the side that is trying to avoid it.
Evading EDR - Matt Hand
Chapter 10, Antimalware Scan Interface
Why script-based attacks are visible at all, and the assumptions that visibility rests on.