When the Performer Breaks Character

Opening Scene

A disruptive audience member at an improv show can sometimes, through a carefully worded suggestion or a persistent, manipulative request, coax a performer into breaking character entirely — saying or doing something well outside what the show’s format and standards were meant to allow. Skilled performers are trained to recognize and resist this, but a sufficiently clever or persistent attempt can occasionally still succeed. Language models face a strikingly similar, genuinely serious risk.

In Plain English

Prompt injection and jailbreaking are techniques for crafting input that manipulates a model into ignoring its intended behavior — its RLHF-trained guardrails from Article 13, or explicit system instructions — to produce output it was specifically designed not to produce. Prompt injection specifically refers to malicious instructions hidden within content a model processes (like a document it’s summarizing); jailbreaking refers to directly crafted prompts designed to circumvent a model’s safety training through clever framing, role-play scenarios, or other manipulation techniques.

The Old Way

Before this had specific names in the LLM context, the underlying risk of an automated system being manipulated through crafted input was already familiar in computer security:

SQL injection attacks manipulate a database system by embedding malicious code within what looks like ordinary input data, a well-established security concept that prompt injection directly parallels.
Social engineering attacks manipulate humans through carefully crafted, persuasive requests, exploiting trust and normal cooperative behavior rather than any technical vulnerability.
Earlier automated systems, covered in Article 3, were generally simpler and had fewer sophisticated “intended behaviors” to actually circumvent, since they had less nuanced guardrails in the first place.

The specific LLM vulnerabilities are genuinely new, but they draw directly on well-established security concepts from earlier computing eras.

What’s Changing (and Why AI Is the Reason)

As LLMs have been deployed with real access to tools, data, and consequential actions, connecting directly to this content library’s dedicated AI agents series, the practical stakes of prompt injection and jailbreaking have grown considerably beyond a simple chatbot producing embarrassing output.
This has become an active, ongoing security research area, with both attack techniques and defensive measures continuing to evolve in something of a genuine arms race, similar to other areas of computer security.
This connects directly to this content library’s dedicated series on AI governance and responsible AI, since robust defense against these techniques is a genuine, practical requirement for any organization deploying LLMs with real access or authority.

The Metaphor, Fully Extended

The Improv Scene	Prompt Injection & Jailbreaking Concept
A disruptive audience member coaxing a performer to break character	A carefully crafted prompt manipulating a model into ignoring its guardrails
A manipulative suggestion hidden within an otherwise normal request	Malicious instructions hidden within content a model is asked to process
A performer’s training to recognize and resist manipulation attempts	A model’s RLHF-based safety training designed to resist these techniques
An ongoing, evolving contest between manipulation tactics and performer training	An ongoing, evolving contest between attack techniques and defensive measures

For Beginners: What to Actually Do

Learn the basic distinction between prompt injection (malicious content embedded in processed data) and jailbreaking (direct manipulation through crafted prompts).
Understand this as a genuine, active security concern, not a rare theoretical edge case, especially for any LLM application with real access or authority.
Practice recognizing common jailbreaking patterns, like role-play framings designed to bypass safety guidelines, as a basic security literacy skill.

For Practitioners and Leaders: The Deeper Layer

Treat prompt injection and jailbreaking as genuine security risks requiring dedicated defensive measures, particularly for any LLM application with real tool access or consequential authority.
Connect this concern directly to this content library’s AI agents series, since agentic systems with real-world action-taking capability face meaningfully higher stakes from these attacks.
Stay current on this genuinely active, evolving area of both attack and defense research, since best practices continue to develop.

Quick Recap

Prompt injection embeds malicious instructions within content a model processes; jailbreaking directly manipulates a model into bypassing its safety training.
Both draw on well-established computer security concepts, applied to the specific new context of language models.
The practical stakes have grown as LLMs gain real access to tools, data, and consequential actions.
This remains an active, ongoing arms race between attack techniques and defensive measures.

Where This Fits in the Series

Article 15 covered a genuine risk to a capable model’s intended behavior. Article 16 covers a related, more resource-conscious topic: models deliberately built smaller than the largest possible.