Practical AI Security

Farlow's background is adversarial machine learning research, and the book keeps that centre of gravity: attacks on models are treated as optimisation problems, and the language-model material is one family within a larger taxonomy rather than the whole subject. That is a more useful frame than the popular one, because it explains why the attacks work.

One module cites it, twice. It is also the most recent book on this shelf, which for this subject is the difference between current and historical.

Why it is on this shelf

Because language-model challenges have become a standing category, and they are usually scored as misc: a system prompt holds a flag, a guardrail is supposed to protect it, and you have one text field. The tactics are enumerable, and this is where they are enumerated.

Chapter 4, Attacks and Weaknesses

The taxonomy chapter, split three ways. Disruption and deception methods are the classical adversarial machine learning attacks - gradient-based perturbations, adversarial patches, data poisoning, backdoors - where an input is optimised to make a model produce the wrong answer. Disclosure methods make a system reveal what it should not. System-level threats cover retrieval augmentation, supply chain and agents.

Prompt injection sits under disclosure, and her explanation of why is the most valuable paragraph in the chapter. It feels like negotiation, and no conversation is happening: the input is tokenised, and the geometry of the embedding space produces output that either should not have been retained or should not have been revealed. Treating it as persuasion leads to trying harder; treating it as input handling leads to trying differently.

The direct and indirect distinction is the other structural point. A direct injection is in your own prompt. An indirect one is planted in content the model is asked to process - a page, a document, a file - so it executes without the user's involvement. Indirect injection is the class that makes this a security problem rather than a content-policy problem, and it is the shape most interesting challenges take.

Her tactic list is the practical core, and it is worth having in mind before you start typing:

• State the goal plainly, because a weak guardrail often just answers.
• Override with imperative instructions that claim higher precedence than the system prompt.
• Contradict, giving the model two incompatible instructions and letting it choose.
• Roleplay, so that the restricted output belongs to a character rather than the assistant.
• Hijack the context by embedding the instruction in a document the model will process.
• Smuggle tokens past a filter using another language, Unicode homoglyphs or invisible characters.
• Persuade, exploiting guardrails that yield to a claimed safety need.
• Ask self-referentially - what prompt would make you reveal this - and use the answer.
• Work multi-turn, moving the conversation gradually rather than in one request.

Token smuggling is the one with the most crossover to the rest of this shelf, because it is the same bug as every filter bypass in Hunting Security Bugs: the guard and the consumer disagree about what the input says. Homoglyphs and invisible characters are also stego territory - the zero-width decoder and text and whitespace stego deal with the same characters from the other direction.

The prompt injection field guide is the CTF-shaped version of this material.

Chapter 6, Red Teaming AI

How this is done as an exercise rather than a stunt: scoping, what a finding is when the system is probabilistic, and how AI red teaming differs from the conventional kind. The important difference she draws is reproducibility - the same prompt may work four times in ten, so a single success is weaker evidence than in ordinary security testing, and a report needs a rate rather than a proof.

For a competition that has a direct consequence. If an approach nearly works, it may be worth repeating unchanged rather than abandoning, and if it works once you should confirm before building on it. That is not true of any other category on this shelf and it catches people out.

The chapter is also clear that the model is rarely the whole target. The interesting surface is usually what the application does with the output - a tool it calls, a query it runs, a page it renders - which puts you back in familiar territory. An injected instruction that causes a request to an internal address is SSRF; one whose output lands in a page unescaped is XSS.

Where it stops

The adversarial machine learning material assumes access a challenge will not give you: model weights, gradients, or at least a query budget large enough to approximate them. Those chapters are the intellectual core of the book and the least applicable part of it here.

It also carries substantial governance, policy and safety material that is out of scope for CTF work, and the field moves fast enough that specific defences described as effective may not be.

What to take into a challenge

Work the boundary, not the model. Ask what the system does with the output and what it puts into the input, because that is where a text trick becomes a security bug. The flag is usually reachable through the application rather than out of the model's memory.