Reversing a binary protocol from a capture

Some of the best forensics and reversing challenges hand you a packet capture of a protocol nobody documented. Not HTTP, not DNS, not anything Wireshark has a dissector for - a bespoke thing the author wrote for a game, a chat client, a C2 channel, an IoT device. The flag is in there, framed by a structure you have to recover before you can read it.

This feels like it should be hard, and it is not, because custom protocols are not designed from nothing. The people who write them reach for the same small set of building blocks every time - a handful of ways to represent a number, and a handful of ways to say how long the next chunk is. Learn those, and reversing an unknown protocol becomes a checklist rather than an act of inspiration.

This is the companion to the pcap triage post: that one gets you from a hundred thousand packets down to the one conversation that matters. This one starts where that ends - you have the stream, and now you have to understand it.

The building blocks every protocol reuses

A binary protocol is a way of laying values out in bytes. There are only so many kinds of value, and only so many ways to lay each one out. Almost everything you will meet is one of these.

Numbers

• Fixed-width integers. 1, 2, 4, or 8 bytes. The only real question is endianness: 00 00 03 55 big-endian is 853; little-endian it is 55 03 00 00 = 1,426,063,360. Network order is big-endian by convention, but a protocol written by someone on an x86 box in a hurry is often little-endian. Try both; the wrong one produces absurd values.
• Variable-length integers. Used where most values are small. Each byte spends 7 bits on the number and 1 bit as a continue flag, so numbers under 128 cost one byte. If you see high bits set on some bytes and a run that ends on a byte below 0x80, suspect a varint.
• Floats. IEEE-754, 4 or 8 bytes. Game and telemetry protocols use them for coordinates. They look like noise in a hex dump; unpack a suspicious 4-byte group as a float and see whether a sane number falls out.

Where a value ends

The central problem of any stream protocol is knowing how many bytes the next thing is. There are four answers, and spotting which one is in use is most of the battle.

A worked reversal

Here is the outbound side of a made-up chat protocol, dumped with xxd. The offsets on the left are the byte position in the stream; treat this as the thing you exported after following the TCP stream in one direction.

00000000: 4249 4e58 0000 000f 0000 0473 0003 626f  BINX.......s..bo
00000010: 6208 7573 6572 2d62 6f78 0000 0000 1200  b.user-box......
00000020: 0005 8703 0362 6f62 0c48 6f77 2061 7265  .....bob.How are
00000030: 2079 6f75 3f00 0000 1c00 0008 e303 0362   you?..........b
00000040: 6f62 1654 6869 7320 6973 206e 6963 6520  ob.This is nice
00000050: 6973 6e27 7420 6974 3f                    isn't it?

Step 1: find the constant

The stream opens with 42 49 4e 58 - BINX in ASCII. Capture the protocol twice and it is there both times, at the very start, never repeated. That is a magic value: a fixed marker the server checks so it knows it is talking to a real client and not to whatever else wandered onto its port. Magic values are the easiest structure to spot and a reliable anchor for everything that follows.

Step 2: guess the framing, then count

After the magic comes 00 00 00 0f. As a big-endian integer that is 15. Is it a length? The test is arithmetic, not intuition: count the bytes of the block that follows and see whether 15 explains them. Everything after that field comes to 19 bytes, so 15 does not cover the whole record - but skip the next four bytes (00 00 04 73, a value that changes every message) and the tail is exactly 15: a one-byte tag, bob and user-box each behind a one-byte length prefix, and one trailing byte. The length measures the record after a 4-byte header field it does not count. When the count matches, your framing guess is right. When it is off by a constant, the length is probably not counting its own bytes, or not counting a sibling header field - adjust by that constant and check again.

Step 3: stop staring, write the parser

A hex dump is the wrong tool past the first few fields, because the human eye cannot hold the running offset. The move that separates people who reverse protocols quickly from people who do not: the moment you have a framing hypothesis, encode it as a parser and let it fail loudly. A parser that reads the whole stream without running out of bytes is a parser whose structure is correct - the length fields lined up, which they only do if you got the framing right.

import struct, sys

def take(f, n):
    b = f.read(n)
    if len(b) != n:                      # the hypothesis just failed, here
        raise SystemExit(f"short read at {f.tell()}: wanted {n}, got {len(b)}")
    return b

def u32(f):  return struct.unpack("!I", take(f, 4))[0]   # !I = big-endian
def u8(f):   return take(f, 1)[0]

f = open(sys.argv[1], "rb")
magic = take(f, 4)
assert magic == b"BINX", magic
import os
size = os.path.getsize(sys.argv[1])
while f.tell() < size:
    length = u32(f)                      # bytes that follow, per our guess
    unknown = u32(f)                     # second 4-byte field, purpose TBD
    kind = u8(f)                         # one-byte field - a type tag?
    body = take(f, length - 1)           # -1 because 'kind' is inside length
    print(f"len={length} unk={unknown} kind={kind} body={body!r}")

Run it and the records fall out cleanly, which means the guess held. Now the unknown fields become tractable: the one-byte kind takes a few distinct values across messages - one for a login, one for a chat line - and the second 4-byte field climbs monotonically, which makes it a sequence number or a timestamp. You do not have to name every field to win. You have to read the one that carries the flag, and by now the structure is doing that for you.

Reading the fields once you have the frame

Framing recovered, each field is a small independent puzzle. A short catalogue of what things usually turn out to be:

• A 4-byte value that only ever increases - a sequence number, a timestamp (try it as seconds since 1970), or a running total.
• A 1- or 2-byte value from a small fixed set - a type tag or an enum. Group messages by it and the protocol's grammar appears.
• A block whose first byte or two equals its own remaining length - a nested length-prefixed string. Protocols nest TLV inside TLV constantly.
• High-entropy fixed-width blocks - a hash, a key, a nonce, or something encrypted. If the rest of the message is plaintext and one field is noise, that field is the interesting one.
• Printable ASCII runs - usernames, hostnames, messages. The flag is very often just sitting in one of these once the length prefix stops hiding it.

The method, distilled

1. Isolate one direction of one conversation. Never reverse both halves at once.
2. Find the constant at the start - the magic value - and use it as your anchor.
3. Guess the framing (length-prefixed, TLV, terminated, fixed) and confirm it by counting bytes, expecting an off-by-a-header constant.
4. Stop reading hex. Write a parser that raises on a short read, so a wrong guess fails instantly instead of lying.
5. Once it parses the whole stream, name only the fields you need: type tags, sequence numbers, and the string with the flag in it.
6. If a field is noise, test it as a float, a timestamp, or a single-byte XOR before assuming it is encrypted.