Prismata: A New Framework for Understanding and Mitigating Prompt Injection in LLMs

Prompt injection remains one of the most persistent and poorly understood security vulnerabilities in large language models. A new paper on arXiv, titled "Prismata: A Formal Framework for Prompt Injection Attacks and Defenses," takes a significant step toward clarity by proposing a rigorous taxonomy and a set of mitigation strategies grounded in information-flow control.

The authors define prompt injection as a violation of the intended separation between system prompts (instructions from the application developer) and user prompts (input from end users). They categorize attacks into three classes: direct injection, where malicious content is placed in the user input; indirect injection, where the model processes external content (e.g., from a retrieved document) containing adversarial instructions; and multi-turn injection, where an attacker gradually manipulates the model across several exchanges.

To formalize these attacks, the paper introduces Prismata, a framework that models the interaction between the model, the application, and the attacker as a game. This allows researchers to prove theoretical bounds on the effectiveness of various defenses. One key insight is that many current defenses—such as colored instruction prompts or output filtering—can be bypassed by sufficiently sophisticated attackers. The authors propose a new defense based on information-flow tracking, where the model maintains a provenance tag for each token indicating whether it originated from a system instruction, a user input, or an external source. By restricting which tags can influence certain outputs, the model can enforce separation even against adaptive adversaries.

The paper also presents empirical results on several open-source and proprietary LLMs, showing that the proposed defense reduces success rates of prompt injection attacks by over 90% with minimal impact on task accuracy. The experiments use a benchmark of 500 handcrafted attack prompts across multiple domains, including customer support, code generation, and content moderation.

While Prismata is primarily a research paper, its implications for developers are direct. The framework suggests that simply sanitizing inputs or appending preambles is insufficient—developers should consider adopting architectures that explicitly track the provenance of data flowing into model contexts. This is especially relevant for Retrieval-Augmented Generation (RAG) systems, where indirect injection via retrieved documents is a growing concern. The paper's formal approach also provides a foundation for building more rigorous security audits of LLM-based applications.

For a deeper dive, read the full paper on arXiv: Prismata: A Formal Framework for Prompt Injection Attacks and Defenses.