Integer bugs: overflow, signedness, truncation, and the off-by-one

Most memory-corruption challenges do not hand you a missing bounds check. They hand you a bounds check that is present, looks correct, and is defeated by arithmetic. The check runs on a number; you control the number; and C's integer rules let that number behave in ways the author did not consider.

There are four such rules, and every integer bug in a CTF is one of them. Learning to see them in a decompiler is worth more than any single exploitation technique, because the arithmetic bug is what *creates* the stack overflow or the heap overflow you then go on to exploit.

Rule 1: unsigned arithmetic wraps

Unsigned integers are modular. 0 - 1 is not negative; it is 0xFFFFFFFF. That single fact produces the most common allocation bug there is:

size_t total = count * sizeof(struct item);   // count = 0x20000000, sizeof = 8
void *buf = malloc(total);                    // total wrapped to 0 on 32-bit
for (i = 0; i < count; i++) buf[i] = ...;     // writes 512M items into a 0-byte buffer

The check if (count > MAX) reject; may well be present. It just does not help, because the overflow happens in the multiply afterwards. Similarly if (offset + len > bufsize) reject; is defeated by choosing offset and len whose sum wraps: the addition produces a small number, the check passes, and the subsequent write uses the un-wrapped values.

Rule 2: signed and unsigned compare differently

The classic. A length is read into a signed int, checked against a maximum, then passed to a function taking a size_t:

int len;
scanf("%d", &len);
if (len > 64) { puts("too long"); return; }   // -1 > 64 is false: check passes
read(0, buf, len);                            // len converts to 0xFFFFFFFFFFFFFFFF

Supply -1. The signed comparison sees a small number and lets it through; the conversion to size_t at the call boundary reinterprets the same bits as enormous. One negative number defeats the entire check.

The mirror image is a negative *index*. if (idx >= COUNT) reject; written with a signed idx permits -3, and array[-3] reads or writes before the array - frequently landing on a saved pointer, a length field, or a function pointer that lives just above it in the struct.

Correct:   if (idx < 0 || idx >= COUNT) reject;
Vulnerable: if (idx >= COUNT) reject;            // signed idx
Vulnerable: if (idx > COUNT) reject;             // off by one, even if unsigned

Rule 3: narrowing truncates silently

Assigning a 32-bit value to a 16-bit variable keeps the low 16 bits and discards the rest, with no warning. So a length of 0x10004 becomes 4 - and if the check ran on the wide value and the copy ran on the narrow one, or vice versa, the two disagree.

unsigned int  n = read_length();       // 0x10004, passes "n > 4096"? no...
unsigned short m = n;                  // m == 4
char *buf = malloc(m);                 // 4 bytes
memcpy(buf, src, n);                   // 65,540 bytes. Heap overflow.

The pattern to look for is one value used at two different widths. In a decompiler this appears as a variable being accessed as eax in one place and ax or al in another, or as an explicit & 0xFFFF you did not expect. Struct fields declared as short or char for compactness are a rich source of this.

Rule 4: the off-by-one is its own class

Not overflow, just an incorrect boundary. It matters more than it sounds because a single byte at the right place is a complete exploit primitive.

• for (i = 0; i <= n; i++) where the buffer holds n elements. One element past the end.
• buf[strlen(buf)] = 0 after a read that already filled the buffer. One null byte past the end - which on the heap is the poison null byte, and on the stack frequently clobbers the low byte of the saved frame pointer.
• strncpy(dst, src, sizeof(dst)) followed by dst[sizeof(dst)] = 0. The same thing, written defensively and still wrong.
• A read of size + 1 bytes into a buffer of size, where the + 1 was meant to leave room for a terminator that the read does not write.

On the stack, a one-byte overflow into the saved rbp gives you partial control of the frame pointer, which after two returns can give you control of the stack pointer - the "off-by-one to stack pivot" that turns one byte into a full ROP chain.

Where to look, in a binary you have never seen

1. Every `malloc` argument. Is it a product? Is either factor attacker-controlled? Is there a check before it, and does that check run on the same expression?
2. Every `memcpy`, `read`, `recv` and `strncpy` length. Trace it back to its source. How many conversions does it pass through on the way?
3. Every array index. Signed or unsigned comparison, and is the lower bound checked at all?
4. Every loop bound. < or <=, and what is the array's real element count?
5. Every struct field used as a size. Its declared width versus the width of the value assigned to it.
6. Anything that parses a length from input - a TLV protocol, a file header, a length-prefixed field. This is where protocol reversing and integer bugs meet, and length fields in file formats are the classic hunting ground.

A worked example of the reasoning

__int64 add_note() {
  unsigned int size;
  int idx = get_free_slot();
  printf("size: ");
  __isoc99_scanf("%u", &size);
  if ( size > 0x400 ) { puts("too big"); return 0; }
  notes[idx] = malloc(size + 1);
  note_size[idx] = size;
  read(0, notes[idx], size + 1);
  return 1;
}

Walk it. size is unsigned and checked against 0x400, so no wrap there. But size + 1 appears twice, and note_size[idx] records size, not size + 1. So the allocation and the read agree with each other - and disagree with the recorded length by one. Whether that is exploitable depends on what edit_note does with note_size, and that is the next function to read.

That is the method in miniature. You are not looking for a missing check. You are looking for two pieces of code that compute the same quantity differently, and then asking which one the memory operation trusts. Every bug in this post is an instance of that question.