Malware triage and defensive telemetry

Before the session

• Nothing to install. Students need a browser and https://ctfpal.com. Confirm the room can reach it once; after that it works offline.
• Read the lesson yourself first - about 18 minutes.
• Have one worked example ready to paste. The classroom link builder on the instructor page turns it into a URL that opens preloaded.

Objectives

Written as things a student can do afterwards, so they can be assessed rather than asserted.

1. Triage a sample statically - imports, sections, entropy, strings - and predict what dynamic analysis will show
2. Cluster related samples using import hashes and fuzzy hashing, and say what each measure actually compares
3. Recognise packing from section entropy and an import table with almost nothing in it
4. Name three anti-debug and three anti-VM checks, and the analysis change each one forces
5. Read Windows event logs and process telemetry into a timeline that supports or refutes a hypothesis
6. Explain what ETW and AMSI see, and where a defender's visibility ends

Running order (120 min)

Tools used

• ELF, PE and Mach-O binary analyzer - Parse headers, sections, imports, and symbols from Linux, Windows, and macOS binaries, with C++ symbol demangling and gadget discovery.https://ctfpal.com/?tool=elf-pe-analyzer
• Strings extractor for binaries and blobs - Pull printable ASCII and UTF-16 strings out of any file, filtered by length, with flag-pattern highlighting.https://ctfpal.com/?tool=strings-extractor
• Office macro and OLE extractor - Extract VBA macros from DOCM, XLSM, and legacy OLE documents, and deobfuscate the string-concatenation tricks they hide behind.https://ctfpal.com/?tool=office-macro-extractor
• PowerShell deobfuscator - Peel -EncodedCommand, FromBase64String, [char] codes, format operators and backtick escapes off an obfuscated PowerShell command, and see what it actually does.https://ctfpal.com/?tool=powershell-deobfuscator
• Perceptual image hash: are these the same picture? - Compare two images by aHash, dHash and pHash in the browser - a re-encoded, resized or recompressed copy has a different checksum and nearly the same perceptual hash.https://ctfpal.com/?tool=perceptual-image-hash
• MD5, SHA-1 and SHA-256 hash generator - Compute MD5, SHA-1, SHA-256, SHA-384, and SHA-512 of any text in the browser, using Web Crypto. Nothing is sent anywhere.https://ctfpal.com/?tool=hash-generator

Reading

• Triage at scale: hashing, similarity, and finding the odd sample - 4 min. When a challenge hands you a folder of a hundred binaries and one is different, reversing each by hand is the wrong move. Import hashing, fuzzy hashing, and YARA turn a pile of samples into a sorted, searchable set.
• Anti-analysis tricks, and how reversing challenges use them - 5 min. Malware-flavoured rev challenges borrow the real thing's defences: anti-debugging, anti-VM, timing checks, and packing. What each trick looks like in a disassembler and the one-line answer to each.
• Log forensics and threat hunting for blue-team CTF - 5 min. Defensive challenges hand you logs and ask what the attacker did. Building a timeline, following the kill chain, and using MITRE ATT&CK as a checklist to turn a pile of events into the story that holds the flag.
• What the endpoint sees: ETW, AMSI, and userland hooks - 4 min. Blue-team and detection challenges ask you to reason about how code gets watched on Windows. The three telemetry sources - ETW, AMSI, and inline API hooks - what each one records, and why evasion challenges target them.

Practice set

Real picoCTF challenges tagged with this module’s techniques, easiest first. Assign the first three in class and the rest as homework.

1. Log Hunt - picoMini by CMU-Africa, easy
2. Event-Viewing - picoCTF 2025, medium
3. Gatekeeper - picoCTF 2026, medium
4. packer - picoCTF 2024, medium
5. unpackme - picoCTF 2022, medium
6. unpackme.py - picoCTF 2022, medium
7. WinAntiDbg0x100 - picoCTF 2024, medium
8. WinAntiDbg0x200 - picoCTF 2024, medium
9. WinAntiDbg0x300 - picoCTF 2024, medium
10. YaraRules0x100 - picoCTF 2025, medium

Checkpoint (gradeable)

Given ten samples, group them by shared code, justify each grouping with the measure that produced it, and name the outlier that belongs to no group.

Deliberately a produced artefact rather than a quiz question: it is either there or it is not, which makes it fast to mark and hard to bluff. Every tool in ctfpal is deterministic, so two students who did the work correctly hand in the same value.

Where the room gets stuck

The instinct is to open one sample in a disassembler. Force the class to sort all ten first: triage is a comparative skill, and a student who has clustered a corpus reads any single sample faster afterwards.

• Treating a hash match as identity and a hash mismatch as difference. A cryptographic hash answers 'the same file?'; imphash and ssdeep answer 'related?', which is the question triage is asking.
• Running an unpacked-looking sample and believing what it does. Sandbox-aware malware behaves for the sandbox, and the interesting branch is the one it did not take.
• Hunting in logs without a hypothesis. 'Look for something odd' returns everything; 'a service was created outside change windows' returns findings.
• Detonating a sample before recording what it is. Hashes, imports, sections and entropy take a minute and the sample cannot undo them; a live run on an unprepared host is a decision that cannot be taken back.
• Writing a YARA rule against a packed sample's bytes. Those change with every repack - a rule worth keeping matches the unpacked code, a configuration structure, or a habit of the builder.

If a student wants the subject, not the answer

Chapter-level references, so a student can be pointed at twenty pages rather than at a book. Nothing here is required to complete the module.

• Evasive Malware, Kyle Cucci. Chapter 10, Anti-Debugging. The checks named individually, with what each one is really measuring.
• Evasive Malware, Kyle Cucci. Chapter 17, Packers and Unpacking Malware. How to get to the real code, which is a prerequisite for everything else in this module.
• Malware Data Science, Joshua Saxe with Hillary Sanders. Chapter 5, Shared Code Analysis. The similarity measures behind clustering, explained well enough to know when each one lies.
• Evading EDR, Matt Hand. Chapter 8, Event Tracing for Windows. Precisely what the defender's sensor sees, from the side that is trying to avoid it.
• Evading EDR, Matt Hand. Chapter 10, Antimalware Scan Interface. Why script-based attacks are visible at all, and the assumptions that visibility rests on.

If you finish early

• Hand out a challenge from the cross-CTF index in this category - each one has published solutions to compare afterwards.
• Run the same input through Identify and let the class argue with the ranking. Disagreeing with a confidence score is where the technique actually lands.
• Ask a student to break their own example - construct an input that defeats the tool, and explain why.