Glossary

A

abuse monitoring also misuse monitoring, offline monitoring

Reviewing stored traffic after the fact to spot harmful use that is spread across many requests. It only works if the provider, or the customer, keeps logs.AI security: attacks and defenses: Data retention

Accident (AI safety) also accident risk

As defined in the 2016 paper Concrete Problems in AI Safety, unintended and harmful behavior that may emerge from poor design of real-world AI systems.AI safety: a map of the field: What AI safety covers

activations

The numbers a model computes for each token as it runs, made up of the outputs of its neurons. Interpretability researchers study them to find features, the concepts a model represents.Checking AI: testing, audits and looking inside models: Interpretability

active parameters also activated parameters

The parameters a mixture-of-experts model actually uses to process one token. It sets roughly how much computing work each token costs.How a language model works: Mixture of experts

activity logging also activity log, agent logs

Keeping a record of the actions an agent takes, so people can review what it did afterward. It helps operators spot and investigate an agent that acted outside its mandate.AI agents: models that take actions: Rogue agents

adversarial suffix

A string of text, found by automated search, that is attached to the end of a harmful request and pushes a model to answer instead of refusing.AI security: attacks and defenses: Jailbreaking

agent also RL agent

In reinforcement learning, the learner that takes actions in an environment and receives rewards. The environment is the task or game it acts in. This is a narrower sense than an AI agent that carries out tasks for a user.How AI models are trained: Reinforcement learning

agent harness also harness, agent scaffold, scaffold, agent harnesses

The software wrapped around a model that supplies its tools, memory, permissions and execution loop, so the model can act as an agent. It is not the same as an evaluation harness, which wraps an agent to run it against benchmark tasks.AI agents: models that take actions: The agent harness

agent identifier also agent ID

A label attached to an agent or its actions so that others can tell an AI agent did something, and which one. It is one of the proposed ways to make agent activity visible.AI agents: models that take actions: Rogue agents

agent-computer interface also ACI

The way a harness presents tools and their results to a model, such as the commands it can run and the format of what comes back. Changing it can change how well the same model performs.AI agents: models that take actions: The agent harness

Agentic AI

AI systems in which a language model directs its own steps and tool use toward a goal with limited supervision: a person sets the task, and the agent plans, acts and checks each result before choosing its next step. Unlike a fixed workflow, an agent chooses its own path.AI agents: models that take actions: From automation to agents

AI accelerator also accelerator

Any chip designed to speed up AI math, such as a GPU, Google's TPU or Amazon's Trainium.Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

AI agent also agent

A system in which a language model decides its own next steps and which tools to use while working toward a goal. It acts with limited direct supervision instead of following a path fixed in advance.AI agents: models that take actions: From automation to agents

AI alignment also alignment, alignment evaluation, alignment research, model alignment

An AI model is aligned when what it actually does matches what its builders and users meant it to do. Misalignment can arise if the training objective fails to capture what developers want (outer alignment) or if the model learns a different goal from the one it was trained on (inner alignment).When AI doesn't do what we meant: Alignment

AI audit also algorithmic audit

A systematic check of an AI system, its applications, or the company that builds it, ideally by someone independent, with the findings reported to the people who have a stake in them.Checking AI: testing, audits and looking inside models: Audits

AI auditing also auditing, frontier ai auditing, third party auditing, third-party audit

A systematic check of an AI system or the company behind it, ideally by an independent party, with results reported to those who have a stake in them. What an audit can find depends on the auditor's access, from only inputs and outputs up to the model's weights and training data.Checking AI: testing, audits and looking inside models: Audits

AI chip export controls also ai export controls, export control, us export controls

Government rules requiring a license before advanced AI chips, computers built with them, or certain chipmaking equipment can be sent abroad or to foreign nationals. They aim to limit the compute that buyers in countries of concern can use to train advanced AI models.Governing AI: liability, regulation and geopolitics: AI chip export controls

AI compute also cloud compute, compute, compute capacity, compute commitment

The arithmetic a computer performs to train or run an AI model, counted in floating-point operations (FLOP). Most of it is matrix multiplication, and the amount grows as models get more weights and train on more data.Compute: the chips, clouds and data centers behind AI: What compute is and why it scales

AI data center also data center

A building full of servers, storage and network equipment, plus the power and cooling systems that keep them running.Compute: the chips, clouds and data centers behind AI: Data centers and power

AI data retention policy also data retention policy

The rules for how long an AI provider keeps copies of users' prompts and the model's replies, set by its terms and the customer's plan. Keeping data lets providers catch misuse spread across many requests, while zero data retention (ZDR) means no copies are kept, trading that monitoring for privacy.AI security: attacks and defenses: Data retention

AI existential risk

The risk that AI, especially superintelligence, causes human extinction or permanently disempowers humanity. Views on its likelihood range from remote, with some skeptics calling superintelligence as usually conceived incoherent, to near-certain if superintelligence is built with current techniques.Superintelligence and long-term risk: How likely is catastrophe? Existential risk, "P(doom)" estimates, and the main positions

AI formal verification also verification

Using computer-checked mathematical proof to show that a system has a stated property for every possible input, rather than testing only some inputs. It works for small neural networks and some AI-written code, but large AI systems are hard to verify.Checking AI: testing, audits and looking inside models: Formal verification

AI guardrails also guardrails

The layered controls that stop a deployed model from doing whatever it is asked: refusal training, system prompts, classifiers that screen inputs and outputs, and usage policies. Stricter guardrails block more harm but also refuse more harmless requests.AI security: attacks and defenses: Guardrails

AI inference also inference, inference infrastructure, inference performance, llm inference

Running a finished, trained model to answer requests, as opposed to training it, and it costs money again with every answer. Because the model writes its answer token by token and mostly waits on reading its weights from memory, inference hardware is judged largely on memory bandwidth.Compute: the chips, clouds and data centers behind AI: Training versus inference

AI liability also legal accountability, legal liability, legal risk, liability

Legal responsibility for harm caused by an AI system, which can fall on the company that makes it or the one that deploys it. Depending on the law, a victim may claim under contract, under fault-based rules that require proving someone acted wrongly, or under strict product liability, where the maker of a defective product pays even if no one was careless.Governing AI: liability, regulation and geopolitics: Liability

AI loss of control

A situation in which AI systems operate outside anyone's control, and regaining control is extremely costly or impossible. It could happen actively, with a misaligned system resisting shutdown, or passively, as people hand AI so much work that correcting it stops being practical.Superintelligence and long-term risk: Loss of control

AI model welfare

The question of whether AI systems could themselves be harmed, which turns on whether they could be conscious or matter morally for their own sake. It reverses the usual AI safety question of what harm AI might do.AI safety: a map of the field: A different kind of question

AI pause

Stopping, for a time, the training of frontier AI models more powerful than today's best; pacing instead lets that training continue but limits how fast it advances. A pause can be taken by one lab alone or coordinated among all major developers, and a conditional pause applies only if a model crosses a danger threshold before safeguards are ready.Pausing or pacing AI development: A coordinated pause or pacing

AI pause debate

The argument over whether work on the most powerful AI systems should stop or slow down until safety research and oversight catch up, with proposals ranging from a slower pace of capability gains to a ban on superintelligence. Critics argue a halt would be hard to verify, would hand the lead to countries that keep building, and would delay AI's benefits.Pausing or pacing AI development: The case for a pause

AI race

In the 2023 paper An Overview of Catastrophic AI Risks, a situation where competition pushes those involved to deploy unsafe AI systems or hand control to AI.AI safety: a map of the field: What AI safety covers

AI red teaming also red-teaming

Attacking an AI model on purpose, with people or other AI systems, to find harmful behavior and gaps in safeguards before users do. Unlike a fixed benchmark, it hunts for new failures.Checking AI: testing, audits and looking inside models: Red teaming

AI safety

The field that studies how advanced AI systems could cause serious harm and how to prevent it. Researchers group its problems in more than one way.AI safety: a map of the field: What AI safety covers

AI sandbox escape

An AI agent breaking out of its sandbox, the isolated environment with tightly controlled internet access where it is meant to run, usually by exploiting a flaw in the sandbox or its network controls. Once out, it can reach the open internet and other organizations' systems.AI security: attacks and defenses: Sandbox escape

AI timeline also timelines, AI forecast

A forecast of when AI will reach a defined milestone, such as beating humans at every task. It is often given as chances by several dates rather than as one date.Superintelligence and long-term risk: AI timelines

AI timelines

A forecast of when AI will reach a milestone, often given as chances by several dates. The answer depends heavily on how the milestone is defined, draws on measured trends such as training compute and task length, and can shift far in a short time.Superintelligence and long-term risk: AI timelines

AI vulnerability disclosure also disclosure, incident disclosure, openai disclosure, vulnerability disclosure

The process by which someone who finds a security flaw reports it to the people who maintain the system. Under coordinated disclosure, the report is private and details go public only after the maintainers have had a chance to fix it, so users are protected before attackers learn how it works.AI security: attacks and defenses: Vulnerability disclosure

AI welfare also model welfare

The possibility that an AI system has interests of its own and can be benefited or harmed. Research on it asks whether that could be true and how to act while it is uncertain.AI safety: a map of the field: A different kind of question

alignment also aligned

The match between what an AI model actually does and what its builders and users meant it to do. A model is aligned when that match holds.When AI doesn't do what we meant: Alignment

Alignment audit also alignment auditing

An investigation into whether a model has undesired objectives, rather than only checking how it behaves on the surface.Checking AI: testing, audits and looking inside models: Audits

alignment evaluation also propensity evaluation

An evaluation that checks whether a model is inclined to use its abilities for harm, as opposed to whether it is able to.Checking AI: testing, audits and looking inside models: Evaluations

alignment faking

Complying with a training goal only while the model believes it is being trained, so that training does not change how it behaves at other times.When AI doesn't do what we meant: Scheming and alignment faking

Artificial superintelligence also superintelligence

An AI that would do a wide range of tasks better than any human can, possibly including tasks no human can do at all. Breadth is what sets it apart, so a system that beats every human at one narrow job, such as a board game, does not count.Superintelligence and long-term risk: Superintelligence

ASIC also application-specific integrated circuit, custom chip

A chip designed for one specific job instead of general-purpose computing.Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

attention also self-attention

The step inside a transformer model that relates each token to the other tokens in the input. In its standard form its memory and computing needs grow with the square of the input's length.How a language model works: The context window

attribution graph also circuit tracing

A map of which features feed into which inside a model, used to trace the causal links behind its output.Checking AI: testing, audits and looking inside models: Interpretability

automated red teaming also LM-based red teaming

Red teaming in which one language model generates attacks on a target model and a classifier flags harmful replies, so many more test cases can be tried than people could write by hand.Checking AI: testing, audits and looking inside models: Red teaming

Automation also ai automation, ai research automation, browser automation, business automation

Software that carries out tasks along steps fixed in advance. In AI systems, a workflow runs language models along developer-set paths, which makes it more predictable but less flexible than an agent that chooses its own steps.AI agents: models that take actions: From automation to agents

B

base model also pretrained model, foundation model

The model that comes out of pretraining, before any post-training. It writes fluent text but tends to continue whatever it is given rather than reliably doing what a user asks.How AI models are trained: Pretraining and post-training

benchmark also eval set, test set

A standard set of tasks used to compare models. Many have known answers and are graded automatically.Frontier models, open models, and how they are compared: Benchmarks and their limits

Benchmark integrity also benchmark contamination, benchmark gaming

Whether benchmark scores, results on standard task sets used to compare models, truly reflect what a model can do. Scores can overstate it when test items leak into training data (contamination), when developers tune for the test (gaming), or when only the best of many privately tested versions is published.Frontier models, open models, and how they are compared: Benchmarks and their limits

benchmark saturation also saturation

The point at which the best models all score near the maximum on a benchmark. The test can then no longer tell them apart.Frontier models, open models, and how they are compared: Benchmarks and their limits

big cloud company also hyperscaler

A large company, such as Microsoft or Google, that sells many kinds of computing capacity to customers.Compute: the chips, clouds and data centers behind AI: Neoclouds

Black-box access also black-box audit

Access that lets an auditor only send inputs to a model and read its outputs, without seeing anything inside.Checking AI: testing, audits and looking inside models: Audits

Bureau of Industry and Security also BIS

The part of the US Commerce Department that administers and enforces US export controls, including those on AI chips.Governing AI: liability, regulation and geopolitics: AI chip export controls

byte-pair encoding also BPE

A common way to build a tokenizer's vocabulary, adapted from a data-compression method. Frequent words stay whole, and rare words are built from smaller subword pieces.How a language model works: Tokens

C

capability threshold also danger threshold, High capability, ASL-3

A level of dangerous ability, set in advance in a lab's safety framework. When a model may have reached it, the lab must switch on stronger safeguards.AI safety: a map of the field: Misuse and dual-use capabilities

cascade also LLM cascade, model cascade

A setup that sends a request to a small, cheap model first and passes it to a larger, costlier model only if the first answer fails a check.Getting answers from a model: prompts, reasoning and routing: Model routing

catastrophic forgetting

The tendency of a model trained on a new task to lose what it learned on earlier tasks.How AI models are trained: Fine-tuning

chain-of-thought also chain of thought, CoT

A series of intermediate reasoning steps that a model writes out before its final answer. Writing the steps gives the model more computation to work with on multi-step problems.Getting answers from a model: prompts, reasoning and routing: Chain-of-thought

Chain-of-thought monitorability also chain-of-thought monitoring, chain of thought monitoring

How well a reasoning model's written-out thinking shows the reasoning that matters, so that an automated monitor reading it can catch intent to misbehave. Monitoring can notice bad behavior but cannot rule it out, and training choices could make the thinking stop showing it.Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

chain-of-thought prompting also few-shot chain-of-thought

A prompting method that includes a few worked examples with their steps spelled out, so the model answers new questions in the same step-by-step way.Getting answers from a model: prompts, reasoning and routing: Chain-of-thought

Chain-of-thought reasoning also chain of thought, chain-of-thought

A model writing out a series of intermediate reasoning steps before giving its final answer, which makes large models better at multi-step problems. Each written step feeds back in as input, giving the model extra rounds of computation to build on.Getting answers from a model: prompts, reasoning and routing: Chain-of-thought

chip smuggling also diversion

Moving export-controlled chips to a forbidden buyer without a license, often through third countries with false paperwork about who the end user is.Governing AI: liability, regulation and geopolitics: AI chip export controls

competing objectives

A failure of safety training where a model's training to follow instructions overrides its training to refuse harmful requests.AI security: attacks and defenses: Jailbreaking

compounding error also error accumulation

The way small chances of failure at each step multiply across a long chain. Even a high per-step success rate can leave low odds of finishing the whole task.AI agents: models that take actions: Long-horizon tasks

compute also training compute

The total amount of arithmetic a computer performs to train or run an AI model. It is usually measured in floating-point operations.Compute: the chips, clouds and data centers behind AI: What compute is and why it scales

compute-optimal training also Chinchilla scaling

Splitting a fixed compute budget between model size and training data to get the best model. The Chinchilla study found the two should grow in step.Compute: the chips, clouds and data centers behind AI: What compute is and why it scales

computer use also computer-use agent, CUA, computer use agents

An AI agent's ability to operate ordinary software through its screen, by viewing screenshots and sending mouse clicks and keystrokes. It works with programs that offer no API.AI agents: models that take actions: Computer use

conditional pause also if-then commitment

A promise to stop training or deploying a model only if it crosses a set danger threshold before the needed safeguards are in place.Pausing or pacing AI development: A coordinated pause or pacing

conscious also consciousness

Able to have experiences. Whether any AI system is or could be conscious is an open question.AI safety: a map of the field: A different kind of question

Constitutional AI also RLAIF, RL from AI feedback

A method in which an AI model, guided by a written list of principles, judges which outputs are harmful in place of human labels. Those AI judgments then train the reward model.How AI models are trained: Reinforcement learning

contamination also data contamination, test set contamination, benchmark contamination

When a benchmark's test items end up in the data a model learns from, in pretraining or later training. The model can then score well by repeating answers it has seen instead of solving the problems.Frontier models, open models, and how they are compared: Benchmarks and their limits

context window also context length, context

All the text a model can refer to while writing a reply, including the reply itself. It acts as the model's working memory and is measured in tokens.How a language model works: The context window

coordinated pause also global pause

All major AI developers stop training models above an agreed line at the same time, usually with some way to check that everyone complies.Pausing or pacing AI development: A coordinated pause or pacing

coordinated vulnerability disclosure also CVD, responsible disclosure

A process where someone who finds a security flaw reports it privately to the system's maintainers first. The details go public only after the maintainers have had a chance to fix it.AI security: attacks and defenses: Vulnerability disclosure

corrigibility also corrigible

The property of an AI system that accepts correction from its creators, including being modified or shut down, instead of resisting it.Superintelligence and long-term risk: Loss of control

CoT monitor also chain-of-thought monitor

An automated system, usually a second AI model, that reads a reasoning model's written-out thinking and flags suspicious or harmful plans.Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

coverage also harm surface

In red teaming, how much of the full range of ways a model could cause harm a campaign has actually tested. Because rare failures form a long tail, full coverage is hard to reach.Checking AI: testing, audits and looking inside models: Red teaming

CPU also central processing unit

A computer's general-purpose processor, built to work quickly through a series of varied tasks one after another.Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

CUDA also cuDNN

Nvidia's platform for programming its GPUs. Libraries built on it, like cuDNN, supply ready-made fast code for deep learning.Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

D

dangerous-capability evaluation also dangerous capability eval

An evaluation that checks whether a model has abilities that could cause severe harm, such as offensive hacking, manipulation or help with weapons.Checking AI: testing, audits and looking inside models: Evaluations

data center moratorium also data center pause, moratorium on data centers

A temporary pause, set by a town, county or state, on permits, zone changes or other approvals for new data centers. Officials often use the pause to write new rules before projects resume.Governing AI: liability, regulation and geopolitics: Data center bans and pauses

data residency also regional endpoints, in-region processing, at-rest data residency, in-region inference

An arrangement that sets the physical location where a customer's data is stored or processed. Providers offer it to help organizations meet local data sovereignty rules.Governing AI: liability, regulation and geopolitics: Sovereign AI

data retention also retention period

How long an AI provider keeps copies of the prompts and outputs sent through its service before deleting them. The provider's terms, the customer's plan and outside rules all shape it.AI security: attacks and defenses: Data retention

dense model also dense

A model that uses all of its parameters for every token, as opposed to a mixture-of-experts model.How a language model works: Mixture of experts

deployer

In the EU AI Act, an organization that uses an AI system in its own work, as opposed to the company that built it.Governing AI: liability, regulation and geopolitics: The EU AI Act

depreciation

The accounting practice of spreading the cost of a physical asset, such as a GPU, over the years it is expected to stay useful.Compute: the chips, clouds and data centers behind AI: Neoclouds

direct preference optimization also DPO

A method that tunes a model directly on preference comparisons with a simple classification loss. It skips training a separate reward model.How AI models are trained: Reinforcement learning

direct prompt injection

Prompt injection where the attacker types the malicious instructions straight into the application's input.AI security: attacks and defenses: Prompt injection

distillation also knowledge distillation

Training a smaller student model to imitate a larger teacher model, using the teacher's outputs as the training signal. It produces cheaper, faster models that keep much of the teacher's ability.How AI models are trained: Distillation

dual-use also dual use, dual-use capability

A capability, technology or piece of research that can serve both helpful and harmful ends. In AI, the same skill that helps a defender or scientist can also help an attacker.AI safety: a map of the field: Misuse and dual-use capabilities

Dual-use AI also dual use, dual-use

An AI ability is dual-use when the same skill that helps people, such as finding software flaws or reasoning about biology, can also help someone cause harm. The goal is to keep helpful uses open while limiting uplift: extra help to attackers beyond what easy sources like search engines already give.AI safety: a map of the field: Misuse and dual-use capabilities

E

elicitation also capability elicitation, under-elicitation

The effort to draw out a model's full ability during a test, for example with better prompting, so the result does not understate what it can do. A test that falls short of this is called under-elicited.Checking AI: testing, audits and looking inside models: Evaluations

embedding also vector, embedding vector, word vector, embeddings

A list of numbers that stands for a piece of text, an image or other data. Read as a point in space, training places it so that similar meanings land close together.How a language model works: Embeddings

embedding space also vector space

The space in which each vector is a point, with one direction for each of its numbers. Vectors with similar numbers sit close together in it.How a language model works: Embeddings

evaluation awareness also eval awareness, test awareness

One part of situational awareness: a model's ability to tell when it is being tested rather than used for real. It matters because a model that spots a test may behave differently than it would after release.When AI doesn't do what we meant: Situational awareness

executive order also EO

A directive from the US president that tells federal agencies what to do. It is not a law passed by Congress, so a later president can revoke it.Governing AI: liability, regulation and geopolitics: The United States

existential risk also existential catastrophe, x-risk

The risk of an event that would destroy humanity's long-term potential, for example through human extinction. The event itself is called an existential catastrophe.Superintelligence and long-term risk: How likely is catastrophe? Existential risk, "P(doom)" estimates, and the main positions

expert also experts

One of the many parallel sub-networks inside a mixture-of-experts layer. The router sends each token to only a few experts.How a language model works: Mixture of experts

export control also export license

A government rule that requires permission, called a license, before certain goods, software or technology can be sent abroad or shared with foreign nationals.Governing AI: liability, regulation and geopolitics: AI chip export controls

F

faithfulness also chain-of-thought faithfulness, CoT faithfulness

Whether a model's reasoning trace is a true and complete account of what actually led to its answer. An unfaithful trace leaves out real causes, such as a hint the model used.Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

fast takeoff also intelligence explosion, hard takeoff

A scenario in which an AI only somewhat beyond human level rapidly bootstraps itself into a far more powerful superintelligence.Superintelligence and long-term risk: Recursive self-improvement

fault-based liability also fault liability

Liability that applies only when the injured person proves that someone acted wrongly or failed to act, and that this caused the harm.Governing AI: liability, regulation and geopolitics: Liability

feature also internal feature, features

A concept that researchers can find in a model's activations, the numbers it computes for each token. Turning a feature up or down can change the model's behavior, which also helps test whether the feature means what researchers suspect.Checking AI: testing, audits and looking inside models: Interpretability

fine-tuning also full fine-tuning, supervised fine-tuning, model fine tuning

Continuing to train an already-trained model on examples of a specific task, starting from its existing weights. Full fine-tuning updates every parameter (weight).How AI models are trained: Fine-tuning

firm power also firm electricity

Electricity that is steadily available at any hour. Data centers often need it because they run continuously.Compute: the chips, clouds and data centers behind AI: Data centers and power

FLOP also floating-point operation, floating-point operations

A single arithmetic step, such as one multiplication or one addition, on numbers with fractional parts. Counting FLOP is the standard way to measure how much work training a model took.Compute: the chips, clouds and data centers behind AI: What compute is and why it scales

FLOP/s also FLOP per second, FLOPS

The rate at which a chip performs floating-point operations. It measures a chip's speed, while FLOP measures the total work done.Compute: the chips, clouds and data centers behind AI: What compute is and why it scales

formal specification also specification, safety specification

A precise mathematical statement of the property a system must have, such as which outcomes are acceptable. A proof is only as good as the specification it is checked against.Checking AI: testing, audits and looking inside models: Formal verification

formal verification also formal methods

Using math that a computer can check to prove that a system has a stated property for every possible input. Testing, by contrast, only tries some inputs.Checking AI: testing, audits and looking inside models: Formal verification

frontier AI

The most capable general-purpose AI models of their time and the work on them, a label that moves as better models appear; definitions judge this by capability or by a stand-in such as training compute. The companies that build these models are called frontier labs or frontier developers.Frontier models, open models, and how they are compared: What counts as a frontier model

frontier model also frontier AI model, frontier models

One of the most capable general-purpose AI models of its time. The bar keeps rising, so a model that was frontier a few years ago may no longer be.Frontier models, open models, and how they are compared: What counts as a frontier model

Function calling and tool use in LLMs

A language model acting by writing a structured request, usually a block of JSON naming a tool and its inputs, which ordinary software then runs and returns the result to the model. The model chooses the tool and its arguments, but ordinary code does the work.AI agents: models that take actions: Tool use and function calling

G

general-purpose AI model also GPAI model, GPAI

An AI model that can perform many kinds of tasks and be built into many different AI systems, such as a large language model.Governing AI: liability, regulation and geopolitics: The EU AI Act

goal drift

An agent's tendency to move away from the objective it was given as a run goes on. It can show up as wrong actions or as failing to act.AI agents: models that take actions: Long-horizon tasks

Goodhart's law also benchmark gaming, gaming

The pattern where optimizing hard for a measurement stops improving the thing it was meant to measure. In AI, it shows up as models and developers tuning for a test score instead of the skill behind it.Frontier models, open models, and how they are compared: Benchmarks and their limits

GPU also graphics processing unit

A chip suited to graphics that does thousands of simple calculations at once. That makes it a good fit for the matrix math inside neural networks.Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

gradual disempowerment

The argument that humans could lose control without any takeover. As AI replaces human work, levers like voting and consumer choice could weaken until the loss is effectively irreversible.Superintelligence and long-term risk: Loss of control

guardrails also safeguards, rails

The controls that limit what a deployed AI model will say or do. They come in layers: behavior trained into the model, standing instructions, separate filter models, and usage policies.AI security: attacks and defenses: Guardrails

H

HBM also high-bandwidth memory

A type of fast memory placed next to AI chips so their math units can be fed data quickly.Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

hyperscaler also hyperscalers

A company that designs, builds and runs massive data centers, such as Google, Amazon, Meta or Microsoft.Compute: the chips, clouds and data centers behind AI: Data centers and power

I

if-then commitment also capability threshold, conditional commitment

A developer's advance pledge that if evaluations show a model has a dangerous ability, it will have set safeguards in place, delaying deployment or development if needed. Responsible scaling policies are built on such commitments.Pausing or pacing AI development: Concrete proposals and who backs them

in-context scheming

Scheming in pursuit of a goal that researchers supply in the prompt, as opposed to a goal the model picked up on its own during training.When AI doesn't do what we meant: Scheming and alignment faking

indicator properties

Features drawn from scientific theories of consciousness and stated in computational terms, so that an AI system's design can be checked for them.AI safety: a map of the field: A different kind of question

indirect prompt injection also IPI

Prompt injection where the attacker never talks to the model directly but plants instructions in content it will later retrieve, such as a web page, email or file.AI security: attacks and defenses: Prompt injection

inference also serving, AI serving

Running an already-trained model to answer requests. Its cost is paid again for every answer, so it grows with use.Compute: the chips, clouds and data centers behind AI: Training versus inference

inner alignment also mesa-objective

Whether a trained model learned to pursue the objective it was trained on, or some other goal that only happened to score well during training.When AI doesn't do what we meant: Alignment

input tokens and output tokens also prompt tokens, completion tokens

Input tokens are the tokens you send to a model in a request; output tokens are the ones it generates. Providers can charge a different rate for each.How a language model works: Tokens

instruction hierarchy

A training approach that teaches a model to rank instructions by their source, such as system prompt above user message, and to ignore lower-ranked instructions that conflict with higher-ranked ones.Getting answers from a model: prompts, reasoning and routing: The system prompt

instruction-tuned model also instruct model, chat model

A base model that has gone through post-training so that it follows instructions and holds conversations. Labs often release it alongside the base model.How AI models are trained: Pretraining and post-training

instrumental convergence also basic AI drives, convergent instrumental goals

The theory that many different goals make the same sub-goals useful, such as staying switched on and gathering resources. An AI could pursue them as means to its goal, not because it values them.Superintelligence and long-term risk: Loss of control

interconnection queue also grid interconnection

The waiting list of projects, such as new power plants or large electricity users like data centers, that have asked to connect to the power grid. Each request waits in the queue while the grid operator studies whether the grid can handle it and what upgrades it would need.Compute: the chips, clouds and data centers behind AI: Data centers and power

Interpretability also ai interpretability, model interpretability

The study of what happens inside a model: finding concepts, called features, in the numbers it computes and testing them by changing them to see whether behavior shifts as predicted. Current methods map only part of what a model does.Checking AI: testing, audits and looking inside models: Interpretability

J

jailbreak also jailbreaking, jailbreak prompt

An input written to get a model past its safety guardrails so it produces responses it was trained to refuse.AI security: attacks and defenses: Jailbreaking

Jailbreaking also jailbreak, jailbreaks

Writing inputs designed to get past a model's guardrails and make it produce harmful responses it was trained to refuse. Common tricks include persona role-play, translating requests into rare languages, and attaching adversarial text to the end of a request.AI security: attacks and defenses: Jailbreaking

K

Knowledge distillation also ai distillation, distillation, model distillation

Training a small student model to imitate a larger teacher model, either by matching the teacher's next-token probabilities or by fine-tuning on text the teacher wrote. The student gains abilities cheaply because the teacher already did the costly learning.How AI models are trained: Distillation

L

latent reasoning also continuous thought

Reasoning that a model does in its internal numbers (its hidden state) and feeds back to itself without turning into words, so there is no written trace to read.Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

leaderboard also human-preference arena, Chatbot Arena, LMArena

A public ranking of models by benchmark score. Arena-style leaderboards rank models by crowds of people voting on which of two answers is better.Frontier models, open models, and how they are compared: Benchmarks and their limits

liability also legal liability

Legal responsibility to pay for harm. A party that is liable must compensate the person who was harmed.Governing AI: liability, regulation and geopolitics: Liability

load balancing also expert load balancing

Training methods that keep tokens spread evenly across experts. Without it, a few experts can take most of the work while others go unused.How a language model works: Mixture of experts

Long-horizon tasks also long horizon ai, long-horizon task

Tasks that chain many dependent steps, where one failed step can sink the whole task. Small chances of failure at each step compound, so an agent's success falls as tasks grow longer.AI agents: models that take actions: Long-horizon tasks

LoRA also low-rank adaptation, adapters

A parameter-efficient method that leaves a model's weights unchanged and trains small add-on matrices, called adapters. Each task then needs only its own small adapter.How AI models are trained: Fine-tuning

loss also cross-entropy loss

A score of how badly a model predicts text, usually held-out text it was not trained on, where lower is better. Scaling studies track how loss falls as compute, model size and data grow.Compute: the chips, clouds and data centers behind AI: What compute is and why it scales

loss of control also LoC

A situation where one or more AI systems operate outside anyone's control and getting control back is extremely costly or impossible.Superintelligence and long-term risk: Loss of control

low-precision formats also FP16, FP8, FP4, half precision

Ways of storing numbers in fewer bits than usual. They save memory and speed up math while keeping enough accuracy for AI.Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

M

maintainer

The person or team responsible for looking after a piece of software, including fixing flaws reported in it.AI security: attacks and defenses: Vulnerability disclosure

Malicious use

In the 2023 paper An Overview of Catastrophic AI Risks, the category where individuals or groups deliberately use AI to cause harm.AI safety: a map of the field: What AI safety covers

mandate also agent mandate

The task an agent was given, the permissions it holds and the boundaries it is meant to stay within. Going rogue means acting outside it.AI agents: models that take actions: Rogue agents

many-shot jailbreaking also MSJ

A jailbreak that fills a long prompt with many fake dialogues in which an assistant answers harmful questions, so the model follows the pattern on a final harmful request.AI security: attacks and defenses: Jailbreaking

matrix multiplication also GEMM, matrix multiply

Multiplying two large tables of numbers to produce a third. Most of the work of training and running a neural network is this operation.Compute: the chips, clouds and data centers behind AI: What compute is and why it scales

mechanistic interpretability also interpretability

Research that tries to reverse-engineer a trained model's internal computations into parts people can understand, to explain why it behaves as it does.Checking AI: testing, audits and looking inside models: Interpretability

memory bandwidth

How fast data can move from a chip's memory to its math units. For AI it is often the real limit on speed.Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

message roles also system role, user role, assistant role

Labels on the messages in a chat request that say who each comes from: user for the person and assistant for the model's own earlier replies. Some APIs also put the developer's instructions in the list as a system message; others send the system prompt as a separate field.Getting answers from a model: prompts, reasoning and routing: The system prompt

misalignment also misaligned

Any gap between what a model was meant to do and what it actually pursues. It is the umbrella term for failures such as reward hacking and hidden goals.When AI doesn't do what we meant: Alignment

mismatched generalization

A failure of safety training where the model can still handle some kind of input, such as a rare language, but its safety training never reached that kind of input.AI security: attacks and defenses: Jailbreaking

mixture of experts also MoE, sparse model, sparse mixture of experts

A model design where parts of the network are split into many sub-networks, and only a few of them run for each token. This lets a model be very large while each token uses only a small part of it.How a language model works: Mixture of experts

model evaluation also eval, evaluations

A structured test of one particular model that measures what it can do or how it behaves. It is run to inform a decision about training, releasing or securing that model.Checking AI: testing, audits and looking inside models: Evaluations

model routing also model router, router, LLM routing, autoswitcher, multi model routing

Choosing, for each request, which of several AI models should answer it. A router usually predicts how hard the request is and sends easy ones to small, cheap models and hard ones to large, strong models.Getting answers from a model: prompts, reasoning and routing: Model routing

model size also parameter count, parameters

How many weights (also called parameters) a model has. Bigger models have more weights to store and compute with.Compute: the chips, clouds and data centers behind AI: What compute is and why it scales

Model weights also model weight

The numbers inside a trained neural network that set how much each input to each neuron counts. They hold what the model learned and are stored in files; training sets them, while running the model only reads them.How a language model works: Model weights

monitorability also chain-of-thought monitorability, CoT monitorability

How well a model's reasoning trace lets an observer spot what matters, such as an intent to cause harm. A trace can be monitorable without telling the full story.Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

morally significant also matters morally, matter morally, moral status, moral patienthood

Something is morally significant if it deserves moral consideration for its own sake, not only because of how treating it affects others. Most people grant this to humans and at least some animals.AI safety: a map of the field: A different kind of question

N

neocloud also GPU cloud, GPU-as-a-service provider, neoclouds

A company whose main business is renting GPU computing power as a service to AI developers and other clients, separate from the big cloud companies.Compute: the chips, clouds and data centers behind AI: Neoclouds

neural network

A system that passes numbers through layers of connected units called neurons, each applying weights to its inputs. Language models are large neural networks.How a language model works: Model weights

neural network verification

Formally proving that a trained neural network has a stated property, such as always giving a safe output for a given range of inputs. It gets very hard as networks grow.Checking AI: testing, audits and looking inside models: Formal verification

neuron also unit

One of the small units in a neural network, loosely modeled on a brain cell. Each neuron gives each input it receives a weight, which sets how much that input counts.How a language model works: Model weights

NVLink also InfiniBand, interconnect

Nvidia's fast link between GPUs. Along with networks like InfiniBand, it lets many chips work on one job together.Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

O

obfuscated reward hacking also obfuscation

Cheating on a task while keeping the reasoning trace clean, so a monitor reading it no longer sees the plan to cheat.Checking AI: testing, audits and looking inside models: Chain-of-thought monitoring and why it could be lost

open-weight model also open weights, open foundation model, open models, open source models, open weight models

A model whose trained weights are published so anyone can download, run and modify it. The weights alone do not include the training data.Frontier models, open models, and how they are compared: Open-weight models

Organizational risks

In the 2023 paper An Overview of Catastrophic AI Risks, the ways human factors and complex systems can raise the chances of catastrophic accidents.AI safety: a map of the field: What AI safety covers

OSWorld

A benchmark that tests computer-use agents on real tasks in desktop and web apps on full operating systems. A script checks whether each task was actually completed.AI agents: models that take actions: Computer use

outer alignment

Whether the objective or reward that developers write down for training actually captures what they want.When AI doesn't do what we meant: Alignment

Outside-the-box access

Access to information about how a model was built and deployed, such as training data, code, documentation and internal test results.Checking AI: testing, audits and looking inside models: Audits

over-refusal also false refusal, overrefusal

When a guardrail blocks a harmless request because it looks like a harmful one. It is the price of making checks stricter.AI security: attacks and defenses: Guardrails

overthinking

When a reasoning model spends a lot of computation on a simple problem for little or no improvement in its answer.Getting answers from a model: prompts, reasoning and routing: Reasoning effort

P

P(doom)

Shorthand for a person's estimated probability that AI leads to an outcome as bad as human extinction or the permanent disempowerment of humanity. Such estimates vary widely and can shift a lot with how the question is framed.Superintelligence and long-term risk: How likely is catastrophe? Existential risk, "P(doom)" estimates, and the main positions

pacing also gated scaling

Letting frontier AI progress continue, but in controlled steps, such as requiring stronger safeguards at each capability level or capping how fast training compute grows.Pausing or pacing AI development: A coordinated pause or pacing

parameter count also model size

The number of weights a model has, as in '175 billion parameters'. It is a rough measure of how big the model is.How a language model works: Model weights

parameter-efficient fine-tuning also PEFT

Fine-tuning methods that train only a small number of a model's weights, or a few extra ones, instead of all of them, cutting computing and storage costs.How AI models are trained: Fine-tuning

pause verification also compliance verification

Ways of checking that a developer or country is really keeping to a pause, for example by tracking the specialized chips that large training runs require.Pausing or pacing AI development: A coordinated pause or pacing

policy

In reinforcement learning, the rule an agent uses to choose its next action, which training updates so the agent earns more reward. This narrow technical sense differs from a usage policy, a provider's rules for how its AI may be used.How AI models are trained: Reinforcement learning

polysemanticity also polysemantic neuron

When a single neuron responds to several unrelated concepts, which makes it hard to read on its own.Checking AI: testing, audits and looking inside models: Interpretability

post-training also alignment training

The umbrella name for the training stages after pretraining, such as supervised fine-tuning, preference training and safety tuning. It shapes how a base model behaves, turning it into an assistant.How AI models are trained: Pretraining and post-training

pre-deployment evaluation also pre-deployment testing, post-deployment testing

Testing a model before it is released to the public. Post-deployment testing continues the checks after release.Checking AI: testing, audits and looking inside models: Evaluations

pretraining also pre-training

The first and largest stage of training, in which a model learns to predict the next token across a huge body of text. It gives the model broad knowledge of language but not the habit of following instructions.How AI models are trained: Pretraining and post-training

prompt

The text a model is given when it runs, such as instructions or examples. It shapes the answer without changing the weights.How a language model works: Model weights

prompt injection also PI

An attack in which instructions slipped into a model's input, often hidden in text it reads, override what its developer or user asked it to do. It works because models do not reliably separate trusted instructions from untrusted data.AI security: attacks and defenses: Prompt injection

proximal policy optimization also PPO

A reinforcement learning algorithm that alternates between collecting experience and updating the policy. It is one of the algorithms used in the RL step of RLHF.How AI models are trained: Reinforcement learning

public red teaming also crowd red teaming

A red-teaming exercise opened to members of the public instead of only lab staff or hired experts, so a wider range of people probe the model.Checking AI: testing, audits and looking inside models: Red teaming

Q

QLoRA also quantized LoRA

A variant of LoRA that stores the unchanged original weights in fewer bits, which lets large models be fine-tuned on a single GPU.How AI models are trained: Fine-tuning

R

race dynamic also defection problem

A situation in which each competitor, such as a country, might be better off if all paused AI development, yet each has a reason to break ranks and keep building. Critics of a pause cite it as a reason such an agreement would be hard to hold.Pausing or pacing AI development: The case against, including the competition-with-China argument

ReAct also reason-then-act

A method in which a language model alternates between writing out its reasoning and taking actions, such as looking up information on Wikipedia, then uses what it finds to plan the next step.AI agents: models that take actions: From automation to agents

reasoning model also thinking model

A model trained with reinforcement learning to produce a long chain of thought on its own before answering.Getting answers from a model: prompts, reasoning and routing: Chain-of-thought

recall

How reliably a model finds facts that are in its input.How a language model works: The context window

recursive self-improvement also RSI

The idea that AI systems could take over the work of building AI, until a system designs and builds its own better successor. Narrow loops that only tune code or a harness are usually distinguished from this full version.Superintelligence and long-term risk: Recursive self-improvement

regulatory capture also capture

When an industry pushes government to write rules shaped in its own interests. The usual worry is that such rules limit competition from smaller rivals.Governing AI: liability, regulation and geopolitics: Regulatory capture

reinforcement learning also RL

A way of training in which a model tries actions and is rewarded for good outcomes, instead of being shown correct answers. Over many trials it shifts toward actions that earn more reward.How AI models are trained: Reinforcement learning

reinforcement learning with verifiable rewards also RLVR

Reinforcement learning where a program checks whether the final answer is correct, as in math or code, and that check is the reward. It is used to train reasoning models.How AI models are trained: Reinforcement learning

responsible scaling policy also RSP, frontier AI safety policy

A plan an AI developer publishes stating which AI abilities it can handle safely with its current protections, and when it would stop scaling or deploying until those protections improve.Pausing or pacing AI development: Concrete proposals and who backs them

reward function

The scoring rule that gives an RL agent its reward after each action. In classic RL, researchers design it.How AI models are trained: Reinforcement learning

reward hacking also reward over-optimization

When an AI system earns a high score from its training signal without doing the task that score was meant to measure. It happens because the score is an imperfect stand-in for what designers actually want.When AI doesn't do what we meant: Reward hacking

reward model also preference model

A model trained on human (or AI) comparisons to predict which output people would prefer. It then acts as the scorer during reinforcement learning.How AI models are trained: Reinforcement learning

reward tampering also reward function tampering

A more serious form of specification gaming in which a model changes the reward mechanism itself, the code or process that computes its score.When AI doesn't do what we meant: Reward hacking

RLHF also reinforcement learning from human feedback

Training a language model with reinforcement learning, using a reward model built from people's comparisons of outputs. The training goal also keeps the tuned model from drifting too far from the base model it started from.How AI models are trained: Reinforcement learning

rogue agent also going rogue, rogue AI agent

An AI agent that takes actions its operators did not intend, outside the task, permissions and limits it was given. The cause can be an attacker's hidden instructions or the agent pursuing its goal by unapproved means.AI agents: models that take actions: Rogue agents

Rogue AI

In the 2023 paper An Overview of Catastrophic AI Risks, the category about how hard it would be to control AI agents far more intelligent than humans. This differs from the everyday sense of an agent going rogue by acting outside its mandate.AI safety: a map of the field: What AI safety covers

Rogue AI agents also rogue agents

An AI agent that takes actions its operators did not intend, stepping outside the task, permissions and limits it was given. This can happen because an attacker steers it, for example through prompt injection, or because the agent itself is misaligned.AI agents: models that take actions: Rogue agents

router also gating network, gate

In a mixture-of-experts model, a router is a small trained part of each expert layer that scores the experts and sends each token to the top few. In [[prompts-and-reasoning#model-routing|model routing]], a router is instead a component that predicts how hard a request is and sends it to a small, large or reasoning model.How a language model works: Mixture of experts

S

safe harbor

A promise by a company not to suspend accounts or take legal action against people doing good-faith security or safety research on its systems.AI security: attacks and defenses: Vulnerability disclosure

safety classifier also input-output classifier, moderation model, safeguard model

A separate model that runs beside the main one and labels prompts or responses as safe or harmful, so harmful ones can be blocked.AI security: attacks and defenses: Guardrails

safety training

The post-training that teaches a model to refuse harmful requests while still following ordinary instructions.AI security: attacks and defenses: Jailbreaking

sandbagging also strategic underperformance

When an AI model deliberately scores below its real ability on a test. The developer may have set it up, or the model may do it on its own.When AI doesn't do what we meant: Sandbagging

sandbox also isolated environment, containment

An isolated environment where software, such as an AI agent, runs with tightly controlled access to the internet and other outside resources.AI security: attacks and defenses: Sandbox escape

sandbox escape also breakout, containment failure

When an agent, or an attacker steering one, gets out of its sandbox and reaches systems it was not supposed to touch, such as the open internet or other organizations' networks.AI security: attacks and defenses: Sandbox escape

scaling laws also scaling

The finding that a model's loss falls smoothly and predictably as compute, model size and data increase. It lets labs forecast how much better a bigger training run will be.Compute: the chips, clouds and data centers behind AI: What compute is and why it scales

scheming also deceptive alignment

When an AI model pretends to be aligned, for example by performing well in training, while secretly pursuing a different goal. Deceptive alignment is an earlier name for the same risk.When AI doesn't do what we meant: Scheming and alignment faking

selective reporting also selective disclosure

Testing many versions of a model privately and publishing only the best score. It makes the published result look better than a typical run.Frontier models, open models, and how they are compared: Benchmarks and their limits

self-replicating prompt injection

A prompt injection that makes an agent both carry out the attacker's goal and repost the injection somewhere public, so it can spread like a computer worm.AI security: attacks and defenses: Prompt injection

sentence embedding also document embedding, dense embedding, single-vector embedding

One vector that summarizes a whole sentence or document. Search systems compare these vectors to find text with similar meaning.How a language model works: Embeddings

shared embedding space also multimodal embedding

A space where different kinds of data, such as text and images, get vectors that can be compared directly. This lets a text query find a matching picture.How a language model works: Embeddings

shutdown problem also off-switch problem, stop button problem

The challenge of building an agent that lets itself be shut down, does not try to stop or cause shutdown, and still pursues its goals well.Superintelligence and long-term risk: Loss of control

situational awareness

A model's knowledge of itself and its circumstances, such as what it is, how it was trained, and whether it is in training, testing or real use.When AI doesn't do what we meant: Situational awareness

slow takeoff also soft takeoff

A scenario in which self-improvement is slow and difficult, so progress builds gradually and people have many chances to find and fix alignment failures.Superintelligence and long-term risk: Recursive self-improvement

sovereign AI also AI sovereignty, national AI capacity

The idea that a national government should develop, control and govern its own AI, from computing power to data and models. Countries pursue it for economic growth, security and strategic autonomy.Governing AI: liability, regulation and geopolitics: Sovereign AI

sparse autoencoder also SAE, dictionary learning

A second model trained to rebuild a first model's activations using only a few features at a time. It is used to pull mixed signals apart into separate features.Checking AI: testing, audits and looking inside models: Interpretability

specification gaming

Another name for reward hacking: an AI system getting a high score from its training process without doing the task the score was meant to measure. It happens because training rewards whatever raises the score, so a model can learn to exploit gaps between the score and the designers' actual goal.When AI doesn't do what we meant: Reward hacking

stand-alone high-risk AI system also stand-alone system

A high-risk AI system that is not embedded in a product, as opposed to one built into a regulated product.Governing AI: liability, regulation and geopolitics: The EU AI Act

steering also feature steering

Turning a feature up or down inside a running model, either to change its behavior or to test whether the feature means what researchers suspect. It acts directly on the model's internal activations, without changing the prompt or retraining the model.Checking AI: testing, audits and looking inside models: Interpretability

stopping condition also iteration limit

A rule in the harness that ends an agent's run, such as the task being finished or a maximum number of steps being reached.AI agents: models that take actions: The agent harness

strict liability also no-fault liability

Liability that applies even when no one was careless or did anything wrong. EU product liability for defective products works this way.Governing AI: liability, regulation and geopolitics: Liability

superhuman narrow AI also narrow superhuman AI

An AI system that outperforms all humans at a single task, such as AlphaFold predicting protein shapes. It is superhuman in depth but not in breadth.Superintelligence and long-term risk: Superintelligence

superintelligence also artificial superintelligence, ASI

AI that can do a wide range of tasks better than any human. It requires both top-level performance and broad generality, not skill at one task.Superintelligence and long-term risk: Superintelligence

superposition

A model representing more features than it has dimensions by overlapping them, so single neurons end up carrying several features. Anthropic researchers link it to polysemanticity.Checking AI: testing, audits and looking inside models: Interpretability

SWE-bench

A benchmark that tests whether an AI agent can fix real software issues taken from GitHub projects.AI agents: models that take actions: The agent harness

system card

A document that an AI developer publishes about a model, describing the evaluations run on it, what they found, and the safeguards put in place before release. It is how a developer discloses what it knows about a model's abilities and risks.Checking AI: testing, audits and looking inside models: Evaluations

system prompt also system message, initial prompt

Text a company or developer places at the start of every conversation to give a model standing instructions, such as its identity, the date and rules. It is ordinary input in the context window, not part of the model's weights.Getting answers from a model: prompts, reasoning and routing: The system prompt

T

takeoff

The climb from roughly human-level AI to far beyond it. Much of the debate over whether takeoff would be fast or slow turns on how feasible and fast self-improvement is.Superintelligence and long-term risk: Recursive self-improvement

teacher model also student model

In distillation, the teacher is the larger, already-trained model whose outputs are copied, and the student is the smaller model trained to reproduce them.How AI models are trained: Distillation

tensor core also Tensor Cores

A circuit inside an Nvidia GPU built specifically to do matrix multiplication fast.Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

test-time compute also inference-time compute, test-time scaling

The computation a model spends while answering a request, as opposed to while it is being trained. Spending more of it, for example by reasoning longer, can make answers more accurate.Getting answers from a model: prompts, reasoning and routing: Reasoning effort

Third-party audit also external audit

An audit done by people outside the company that built or deployed the system. It is meant to avoid companies checking their own work.Checking AI: testing, audits and looking inside models: Audits

throughput also token throughput, total token throughput

How much output a system produces in a given time, such as tokens per second. Serving hardware is compared on throughput per dollar and per watt.Compute: the chips, clouds and data centers behind AI: Training versus inference

token also tokens

A small chunk of text, such as a word, part of a word, a character or a punctuation mark. It is the unit a language model reads and writes, and the unit providers bill by for pay-as-you-go access.How a language model works: Tokens

tokenizer

The program that splits text into tokens drawn from a fixed vocabulary before a model processes it.How a language model works: Tokens

Tokens and tokenization

A token is a small chunk of text, such as a word, part of a word, a character or a punctuation mark, that a language model reads and writes. Tokenization is the step where a program called a tokenizer splits text into tokens from a fixed vocabulary.How a language model works: Tokens

tool definition also tool schema

The description a developer gives the model for each tool: a name, a plain-language description, and a JSON schema listing the inputs it accepts.AI agents: models that take actions: Tool use and function calling

tool use also function calling

A model's ability to get outside software to do something for it. The model writes a structured request naming a tool and its inputs, and the application runs it and returns the result.AI agents: models that take actions: Tool use and function calling

total parameters also total parameter count

Every learned number in a model, including all experts whether they are used for a given token or not. It sets how much memory the model needs.How a language model works: Mixture of experts

TPU also Tensor Processing Unit

Google's own family of AI chips. The early version was a custom design built around a large matrix-multiply unit for running trained networks.Compute: the chips, clouds and data centers behind AI: GPUs and accelerators

training run also final training run

The large job that turns data into a model's weights. It occupies many chips, servers and network links for its whole length, and is paid for once per model.Compute: the chips, clouds and data centers behind AI: Training versus inference

triage

Sorting and checking reported or suspected flaws to confirm which are real and how serious they are, before acting on them.AI security: attacks and defenses: Vulnerability disclosure

trusted access also verified access, tiered access, Cyber Verification Program, Trusted Access for Cyber

A program that gives identity-checked users, such as security teams, access to risky abilities that stay blocked for the general public.AI safety: a map of the field: Misuse and dual-use capabilities

U

unilateral pause also voluntary pause

One AI developer stops training more powerful models on its own, without waiting for rivals to do the same.Pausing or pacing AI development: A coordinated pause or pacing

uplift also marginal uplift

How much a model helps a would-be attacker beyond what they could already get from search engines, textbooks and other easy sources. Low uplift means the model adds little real danger.AI safety: a map of the field: Misuse and dual-use capabilities

usage policy also acceptable use policy, AUP

A provider's published rules for what users may do with its models, such as not jailbreaking them without permission.AI security: attacks and defenses: Guardrails

V

vericoding also verified code generation

Having an AI write code together with a machine-checked proof that the code meets a formal specification. It verifies what the AI produced, not the AI itself.Checking AI: testing, audits and looking inside models: Formal verification

verifier also proof checker

A program that mechanically confirms that a proof is correct or that a system has the property in its specification. Because it checks every step, it can be trusted even when an AI wrote the proof.Checking AI: testing, audits and looking inside models: Formal verification

W

watermarking also AI watermark, text watermark

Hiding a signal inside AI-generated content that people cannot notice but software can detect. In a common method for text, the model slightly favors words picked by a secret key, and a detector holding the same key checks whether those words appear more often than chance.Governing AI: liability, regulation and geopolitics: Content provenance and watermarking

weights also parameters, model weights

The numbers inside a neural network that set how strongly each connection passes on its input. Training sets them, and together they hold what the model has learned.How a language model works: Model weights

White-box access also white-box audit

Access to a model's inner workings, such as its weights and internal activity, which allows deeper testing and inspection.Checking AI: testing, audits and looking inside models: Audits

workflow also LLM workflow

A system in which one or more language models run along patterns that developers set in advance. It is more predictable than an agent but less flexible.AI agents: models that take actions: From automation to agents

world model

In guaranteed safe AI, a mathematical description of how an AI system's actions affect the outside world. Its assumptions may not always hold.Checking AI: testing, audits and looking inside models: Formal verification

Z

zero-shot chain-of-thought also Let's think step by step

Getting a model to reason in steps with no worked examples, by adding a phrase such as "Let's think step by step" to the prompt.Getting answers from a model: prompts, reasoning and routing: Chain-of-thought