Working a leaked dataset: OSINT on a pile of data

A distinct flavour of OSINT challenge does not send you searching the live internet - it hands you the data directly. A leaked CSV, a database dump, a scraped archive, a folder of documents, and a question that can only be answered by finding the right record inside it. The dataset is often large enough that opening it in a spreadsheet is hopeless, and the skill being tested is not searching the web but interrogating a pile of data efficiently.

This is a real discipline - investigative journalists do exactly this with leaked datasets - and it is the bulk-data half of the OSINT method. It comes down to a handful of command-line habits. The tools are unglamorous (grep, a bit of Python, jq) and that is the point: they scale to gigabytes where GUIs fall over.

First: understand the shape before you search

Before grepping for the answer, spend two minutes learning what you have. What formats are in here, how big, how structured? A blind search of data you do not understand wastes time and misses things sitting in a column you did not know existed.

find . -type f | sed 's/.*\.//' | sort | uniq -c | sort -rn   # what file types
find . -type f -printf '%s %p\n' | sort -rn | head             # the biggest files
head -5 data.csv                                                 # the columns
file *                                                           # true types, not extensions

grep is the whole toolkit, if you know it

For text data - CSVs, logs, JSON lines, exported documents - grep with regular expressions answers most challenges directly. The moves worth internalising:

grep -ri "target name" .                 # recursive, case-insensitive
grep -rn "flag{" .                        # with line numbers, to locate the hit
grep -rl "keyword" .                      # just which files match (narrow first)
grep -rE '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}' .   # all emails
grep -roE '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' .        # all IPv4 addresses
grep -A3 -B3 "the record"                 # context around a match

CSVs and structured dumps

A leaked dataset is very often a CSV, and CSVs have a trap: fields contain commas, quotes, and newlines, so naive splitting on commas corrupts your results. Use a real parser. For quick column work csvkit (csvcut, csvgrep, csvlook) handles quoting correctly; for anything with logic, a few lines of Python with the csv module is more reliable than any one-liner.

import csv
with open("dump.csv", newline="") as f:
    for row in csv.DictReader(f):          # respects quotes and embedded commas
        if row["city"] == "Austin" and int(row["amount"]) < 0:
            print(row["account"], row["amount"])   # the anomalous records

Documents carry more than their text

When the dump is documents - PDFs, Office files, images - the visible content is only half of it. Metadata routinely outlives the text: author names, GPS coordinates in photos, creation software, revision history, timestamps. A challenge that asks 'who made this' or 'where was this taken' is usually asking you to read the metadata, not the words.

exiftool -r -a -G1 documents/          # every metadata field, recursively
# GPS, author, software, and timestamps all print here.
strings -n 8 leaked.pdf | grep -iE 'author|creator|producer'

A note on handling

The workflow

1. Inventory the dump: file types, sizes, and the columns/structure of each.
2. For text, grep to the right file with -l, then extract with a targeted regex.
3. For CSVs, use a real parser (csvkit or Python csv) - never split on commas by hand.
4. When the answer is a correlation, join two files on a shared key.
5. For documents, read the metadata - author, GPS, timestamps - not just the content.
6. Work on a copy and keep the data local.