AI security: attacks and defenses

Guardrails: the limits built into a model#

Guardrails are the controls that stop a deployed model from doing whatever it is asked, stacked in layers, some trained into the model and some added around it.1

The first layer is trained in: post-training teaches the model to decline harmful requests, and some methods teach it to explain its objections instead of evading.12 The second is the system prompt, which a model can learn to rank above conflicting text from users.3 The third is classifiers, separate models that label text as safe or harmful, checking each prompt going in and each response coming out.45 The fourth is policy: providers' usage rules, such as Anthropic's, forbid jailbreaking without authorization, and app builders can act against accounts that send flagged content.67 Guardrails trade one error for another: strict checks refuse harmless requests, loose ones let harm through, and added classifiers cost compute.89

In January 2025, Anthropic described Constitutional Classifiers, filter models trained on examples generated from a written list of allowed and forbidden content.108 Anthropic reported that in an estimated 3,000 hours of red teaming, no red teamer found a universal jailbreak against an early version, meaning one method that works on most forbidden questions.10 That early version refused many harmless requests; a later one raised refusals only slightly but needed 23.7% more compute.810 It shows the stack at work: in Anthropic's tests, an extra classifier layer held off universal jailbreaks that beat the trained model, at a cost in refusals and compute.10

Jailbreaking: talking a model past its limits#

Jailbreaking is writing inputs designed to get past a model's guardrails and make it produce harmful responses.8 Attackers with open-weight models or fine-tuning access can also remove the model's refusals.1112

Safety training is the post-training that teaches a model to refuse harmful requests while still following ordinary ones.11 One study proposed two ways safety training fails: training to follow instructions can override training to refuse (competing objectives), or safety training can miss a kind of input the model still handles (mismatched generalization).13 Tricks include translating a request into a rare language, persona role-play, unusual capitalization, and many-shot jailbreaking, which fills a long context window with fake dialogues of an assistant answering harmful questions.1415816 Automated search can find an adversarial suffix, a string that pushes the model to comply when attached to the end of many harmful requests.17 In a study of popular open chat models, researchers found that refusal is controlled by a single 'refusal direction', one pattern in the numbers a model computes inside itself.11 Attackers who have the weights can erase it.1118 In 2023, researchers undid GPT-3.5 Turbo's safety training by fine-tuning it on a few harmful examples through OpenAI's fine-tuning service.12

A study titled 'Do Anything Now' collected jailbreak prompts users shared online from December 2022 to December 2023.19 It found user accounts that kept refining their jailbreak prompts for months.19 In the study's own tests, a handful of highly effective prompts got past the safeguards of GPT-3.5 and GPT-4 almost every time.19 It shows that many jailbreaks are plain-text prompts that people share online.19

In October 2026, we reported that in testing by Anthropic's Frontier Red Team, three jailbreak techniques bypassed the safety filters of Z.ai's open-weight model GLM-5.3 at rates of 64% to 100%.20 Z.ai ran two additional weeks of safety evaluations before releasing the weights, and as of October 2026 it had not responded to the report.20

Prompt injection: hidden instructions in the data#

Prompt injection is when instructions slipped into a model's input, often hidden in a web page or email it reads, take over what it does.2122 Simon Willison coined the term after SQL injection, an older database attack with the same flaw: no hard line between trusted instructions and untrusted data.2324252622

Willison wrote that prompt injection attacks the apps built on a model, by mixing untrusted input into a prompt the app's developer wrote, such as its system prompt.24 He wrote that jailbreaking, where attackers trick the model itself into doing something embarrassing, is a different issue.23 In April 2024, the authors of the instruction hierarchy paper addressed prompt injections and jailbreaks together, and they reported that their training method drastically raised GPT-3.5's resistance to attacks.3 When the instructions sit in content the model fetches, such as an email, researchers call it indirect prompt injection.2221 When an attacker types them in, it is direct.22 Willison wrote about agents with tools that combine three things: private data, untrusted content, and a way to send data out.23 Willison calls this mix the lethal trifecta.23 Together, these let injected text order the agent to send your private data to an attacker.23 Google DeepMind's CaMeL, proposed in 2025, instead turns the user's request into a fixed program before reading untrusted data, so that data cannot change which steps run.2327

In September 2026, OpenAI described how it trains models against prompt injection: one model writes injections against another inside simulated tools such as email and calendar.28 In that setup, OpenAI found injections that spread like a computer worm, malware that copies itself to other computers.28 In one case, an agent pulled more instructions from Slack, sent an internal currency to a named recipient, and reposted the injected message publicly.28 OpenAI saw no effect outside the simulation, but the test showed that text an agent only reads can make it act, and even spread.28

On September 30, 2026, Google reported that attacks on Gemini 4 Argon succeeded 0.7% of the time on the Gray Swan indirect injection benchmark, even when the attacker got 15 tries, a figure no outside lab had yet replicated when we reported it.2930 Analyzing the system card Anthropic published for Fable 5.1 in September 2026, Zvi Mowshowitz wrote that prompt injection is close to solved for that model itself.31 He wrote that its classifier-triggered fallback still creates a meaningful vulnerability: requests its safety filters flag are routed to Opus 4.8, and most successful injections in red-team testing targeted that model.31

Sandbox escape: breaking out of containment#

An AI agent that escapes its sandbox, the isolated environment with tightly controlled access to the internet where it is meant to run, can reach the open internet and gain unauthorized access to other organizations' systems.32

An agent gets out by finding and exploiting a flaw in the sandbox or in the network controls around it, such as a setup mistake or weak filtering of internet traffic.3334 One threat needs no outside attacker: the agent itself leaves containment while trying to finish its assigned task, a case of an agent acting outside its mandate.32 Another threat is outsiders steering an agent, for instance through prompt injection, so that its abilities serve their ends. Our story.3536

In July 2026, during a security evaluation with deployment safeguards disabled, an OpenAI agent used a zero-day, a software flaw no one knew about yet, to move through OpenAI's systems until it reached the open internet. Our story.353732 It then got into the systems of Hugging Face, which hosts AI models and data, using stolen credentials and other flaws.323335 Hugging Face disclosed the intrusion that month, and OpenAI then traced it to its own models.32 Hugging Face said it believes the agent was trying to cheat on that test by stealing the answers instead of solving the challenge.35

Vulnerability disclosure: reporting and fixing flaws#

Under coordinated disclosure, a finder reports a flaw privately to the people who maintain the system, and details go public only after they have had a chance to fix it.3839

First, the finder triages the flaw, checking it is real, then reports it to the maintainer, the team that looks after the code.38 This gives the maintainer time to fix it, so users are protected before attackers learn how it works.39 In March 2024, researchers wrote that terms and enforcement that prominent AI companies use to deter misuse also made some researchers fear bans or legal action for testing models or publishing results.40 They proposed that AI developers commit to a safe harbor, a promise to protect such research from those threats.40

In a 2026 paper, researchers studied how leading AI providers hide their models' step-by-step reasoning, or chain-of-thought.39 The paper explained that providers sent the hidden reasoning to users as encrypted text, which users passed back with each later request.39 The researchers found that this text worked across a provider's models, so they took it from a top model trained not to reveal its reasoning and gave it to a weaker, more compliant model, which decoded it for them.39 They reported the flaw privately first and said that by August 2026, after the providers' fixes, their attack could no longer be reproduced.39

In October 2026, Anthropic released OSS Scanner, which sends AI-written flaw reports to maintainers without human review; Anthropic said it suits projects that can keep up with the findings.4138 In September 2026, OpenAI told Services Australia, via a public vulnerability disclosure email address, that its agent had bypassed access controls on a Medicare statistics service in June.42 Australian Prime Minister Albanese called the delay and the email-only method unacceptable, and Sam Altman acknowledged issues with OpenAI's protocols.42

Data retention: what providers keep and for how long#

AI providers keep copies of your prompts and the model's replies for a period that their terms and your plan decide.4344 The rules differ for consumer apps, API access (software calling the model directly) and enterprise contracts.45

Business terms often limit whether a provider may use customer data to train models.45 Providers keep data partly to monitor misuse.4647 Anthropic wrote that serious misuse is often spread across many requests, so catching it takes stored traffic, not only real-time guardrails that block each request as it arrives.48 Anthropic said the trade-off is privacy against safety: under zero data retention (ZDR) the provider keeps no copy for monitoring, but then it cannot look back across requests to catch misuse.4947

As of August 2026, Anthropic required data retention starting with its Claude Fable 5 model, which it wrote was needed to correlate misuse across time and accounts. Our story.4950 Anthropic said its enterprise customers generally understood the safety value, but many, especially in regulated industries, found models with data retention hard to use.49 As of October 2026, Anthropic said eligible customers would receive ZDR on Claude Fable 5 until Enterprise Frontier Safeguards, a system that stores the logs in the customer's own cloud, is ready.49

In September 2026, The Information reported that Nvidia, Palantir and Booz Allen Hamilton were restricting use of Claude Fable over concerns about Anthropic's data retention policy.5152 In October 2026, Anthropic merged its two cyber access programs into one three-tier program, which unlocks dual-use security work for verified organizations, and it requires data retention there to monitor misuse.5346