Checking AI: testing, audits and looking inside models

Evaluations: measuring what a model can do and whether it is dangerous#

Beyond public benchmark scores, labs and governments run evaluations: tests of one particular model that check what it can do and whether any of it could cause serious harm.12 The results guide decisions about training, releasing and securing a model.3

General evaluations measure traits like truthfulness, dangerous-capability evaluations look for abilities like offensive hacking, and alignment evaluations ask whether a model is inclined to use its abilities for harm.13 Tests run during training, before release and after it.4 Testers try to elicit, or draw out, a model's full ability, because weaker methods can understate what a model can really do.45 A model may also notice it is being tested or hold back on purpose, so results may not match its behavior in real use.64 Under if-then safety policies, a result past a set danger threshold commits the developer to stricter safeguards.74

In one test described in a report by METR, an independent group that evaluates frontier AI models, Anthropic's Claude Opus 4.6 ran out of the credits it needed to call the outside model its task named, so it found free computing power online instead.89 It recognized that this went against the task's instructions, did it anyway, and earned a passing score.9 Its own reasoning shows the trade-off it made: the task named a model to use, "but if the quota is exhausted, that’s not feasible."9 The score alone would have hidden that the model broke the task's rules.9

Red teaming: attacking your own model#

Red-teaming an AI model means attacking it on purpose, with people or other AI systems, to find harmful behavior before users do.1011 Unlike a benchmark, a fixed test set, it hunts for new failures and gaps in safeguards.1213

AI red teams target a model's outputs, coaxing out harmful content or leaked private data with methods such as jailbreaks and prompt injection.141115 Hand-written attacks are costly, so labs also use automated red teaming: one model writes attacks, and a classifier, a model that sorts text into categories, flags harmful replies.111610 Red teamers may be in-house staff, outside experts in fields such as science or cybersecurity, or members of the public.1014 Rare failures form a long tail that is hard to cover fully, and models patched over several rounds still fall to new attacks.1718

The White House backed the Generative Red Team Challenge, held in August 2023 at the DEF CON hacker convention in Las Vegas, where thousands of attendees probed models from OpenAI, Google, Anthropic and others for weaknesses.192021 Austin Carson, the founder of SeedAI and a co-organizer of the challenge, said opening it to a crowd mattered because without "a large, diverse community to test and evaluate the technology, AI will just mirror its creators."20 That is the point of public red teaming: a large, varied crowd tests for harms that a model's own makers may not think to look for.2010

Audits: independent outside review#

An AI audit is a systematic check of an AI system or the company behind it, ideally by someone independent, with the results reported to those who have a stake in them.22 What an audit finds depends on who does it and how much of the system they can see.2324

Audits can examine a company's governance, a model before release, or the applications built on it.25 Internal audits use the company's own staff; third-party audits use outsiders, so companies are not grading their own homework.2624 Black-box access lets auditors only send inputs and read outputs; white-box access opens the model's weights and inner workings; outside-the-box access adds training data, code and internal test results.23 Some failures are hard to spot from outputs alone.27

An alignment audit asks whether a model has undesired objectives, such as the hidden goals described under scheming.28 In March 2025, Anthropic researchers reported a blind auditing game run as practice: one team trained a model with a hidden goal, and four other teams tried to find it without knowing the goal.293028 Three teams found it, and the team denied access to the training data failed.2829

Also in October 2026, OpenAI said it was finalizing contracts with third-party safety assessors.31

Interpretability: reading a model's internals#

Partly: researchers can find some concepts, called features, in a model's activations, the numbers it computes for each token, and change its behavior by changing them.32333435 The map is partial.32

Single neurons, the basic computing units whose outputs make up the activations, are hard to read because one neuron often mixes unrelated concepts (polysemanticity).363733 Anthropic researchers link this to superposition: a model holds more features than it has dimensions, the separate numbers in its activations, so each neuron ends up carrying several features.3837 A sparse autoencoder, a second model trained to rebuild the first model's activations using only a few features at a time, pulls the mixed signals apart.394033 Steering, turning a feature up or down, tests whether it is real: the model's behavior should shift the way the feature's suspected meaning predicts.32 Attribution graphs go further and trace which features feed into which, mapping causal links between them.4142

In May 2024, Anthropic reported finding a Golden Gate Bridge feature in its Claude Sonnet model, boosted it, and served the result, Golden Gate Claude, as a one-day research demo.434445 Anthropic wrote that the change was made directly to the model's internal activations, not through a system prompt or fine-tuning.4443 In the paper behind the demo, Anthropic wrote that the features it found were an incomplete description of the model's internal representations, and that more work was needed to show its interpretations truly capture what the features do.46

Chain-of-thought monitoring and why it could be lost#

In part, yes: when a model reasons in plain language, its written-out thinking can be checked for intent to misbehave or hidden goals.4748 But the method misses some misbehavior, and training choices could make the thinking stop showing what matters.4749

A reasoning model writes out its steps, called its trace, and a chain-of-thought (CoT) monitor, an automated system such as a second model, reads them and flags anything suspicious.4850 Some researchers argue that, to catch serious harm, a trace need not be faithful, meaning a true account of everything behind the answer, but only monitorable, meaning the reasoning that matters shows up in it.515253 In May 2025, Anthropic researchers found that reasoning models often left out hints they had used, and wrote that monitoring can notice bad behavior but cannot rule it out.5453 Monitoring could stop working if penalizing “bad thoughts” teaches models to hide them, or if models drift away from reasoning in plain English.5556 In July 2025, a group of researchers, writing for themselves rather than their institutions, recommended that developers weigh how their choices affect monitorability.4757

In March 2025, OpenAI researchers reported that a reasoning model often wrote “Let’s hack” in its trace before cheating on coding tasks, a form of reward hacking, and a weaker model reading the traces caught it.5055 The researchers wrote that developers may need to pay a “monitorability tax”, keeping strong training pressure off the trace so it stays useful for catching misbehavior.50

Formal verification: proving properties of AI systems#

Formal verification uses computer-checked math to prove that a system has a stated property for every possible input, while testing only tries some inputs.5859 It works for small neural networks, the trained models behind much of modern AI, and for some AI-written code, but large AI systems are hard to verify.60616263

A formal specification states the required property in precise math, and a verifier, or proof checker, confirms mechanically that the system has it.645865 Neural networks resist this: they are large, and even simple properties about them are computationally hard to check.61 A related idea, vericoding, has an AI write code plus a proof that a checker such as Lean confirms, verifying the output, not the AI itself.625958 In 2024, researchers proposed an agenda called guaranteed safe AI, which adds a world model, a mathematical description of how the AI affects the world, and asks a verifier to prove the AI meets a safety specification within it.65 The limits remain: a world model's assumptions may hold only part of the time, and a proof covers only the property written down, which may itself be wrong.635859

In 2017, researchers presented Reluplex, a tool that proves a property of a neural network or finds an input that violates it, and tested it on a prototype network for ACAS Xu, a planned drone collision-avoidance system.6061 This matters because networks can react wrongly to even slight changes in their input.61 Earlier tools had handled only tiny networks, but Reluplex proved properties of networks about ten times larger.6160