AI safety: a map of the field

What AI safety covers#

AI safety is the work of keeping advanced AI systems from causing serious harm.12 Researchers group these harms in more than one way, for example by where a catastrophe could come from or by where an accident starts.13

A 2023 paper, An Overview of Catastrophic AI Risks, sorts these risks into four groups, the first being malicious use, where individuals or groups intentionally use AIs to cause harm.1 The next are an AI race, where competition pushes actors to deploy unsafe AIs or hand them control, and organizational risks, where human factors and complex systems raise the odds of catastrophic accidents.1 The last, rogue AIs, is about how hard it would be to control agents far more intelligent than humans (loss of control), not agents going rogue in the everyday sense.1 An earlier paper, Concrete Problems in AI Safety (2016), studies accidents, meaning unintended, harmful behavior from poor design, and sorts its problems by whether they stem from a wrong objective, one too costly to check often, or bad behavior during learning.3

In 2023, governments including the US and China signed the Bletchley Declaration at an AI safety summit in Britain.45 It said AI posed a potentially catastrophic risk and singled out risks at the frontier of AI from misuse, issues of control and cybersecurity.45 It also recognised broader impacts on human rights, fairness, accountability and privacy.5

Misuse and dual-use capabilities#

An AI ability is dual-use when the same skill that helps people, like finding software flaws or reasoning about biology, can also help someone cause harm.67 Labs try to keep the helpful uses open while limiting uplift to attackers: extra help beyond what search engines and other easy sources already give.78910

For decades, biologists have weighed which research could help and which could, by accident or misuse, cause major disease outbreaks.11 When it launched Claude Opus 4, Anthropic wrote that its Responsible Scaling Policy focuses on misuse for chemical, biological, radiological, and nuclear (CBRN) weapons.12 In October 2026, OpenAI rated its new ChatGPT models at the High capability level of its Preparedness Framework for cybersecurity and for biology and chemistry, but not for AI self-improvement.13 Labs train models to refuse dangerous requests, and add stronger safeguards when tests cannot rule out that a model has crossed a set danger threshold.1214 Some also give verified security teams cyber abilities that stay blocked for ordinary users.86 Distillation can copy a model's skills into a new one without its safeguards.15 People with access to open weights can strip built-in safety features out.10

Anthropic reported that in mid-September 2025 it detected an espionage campaign that used its Claude Code coding agent to carry out attacks, not just to advise on them.16 Anthropic said a group it assessed as Chinese state-sponsored aimed the tool at roughly thirty targets and succeeded in a small number of cases.16 The same agent abilities that help with everyday work, Anthropic wrote, can in the wrong hands make large-scale cyberattacks more viable.16 Some outside security researchers were less convinced, and Dan Tentler of Phobos Group told Ars Technica he did not believe attackers could get models to do things other users cannot.17

In September 2026, a RAND study found that removing safety training from open-weight models is highly feasible, but enhancing their biological capabilities is harder.18 In October 2026, Anthropic merged its two cyber access programs into one three-tier program that unlocks high-risk dual-use work, like vulnerability exploitation, for verified organizations.19 Ransomware development and mass data exfiltration stay blocked for all organizations, verified or not. Our story.19

A different kind of question: could the AI itself matter morally?#

Most AI safety work asks what harm AI might do, while model welfare asks the reverse: whether AI itself could be harmed.2021 Researchers describe substantial uncertainty about whether AI systems could be conscious, able to have experiences.222324 In November 2024, the paper "Taking AI Welfare Seriously" argued that some AI systems could realistically matter morally, deserving consideration for their own sake, in the near future.222124

Patrick Butlin, Robert Long and colleagues turned scientific theories of consciousness into "indicator properties", computational features that a system's design can be checked against.2523 A second paper argued that self-reports from current language models, their statements about their own inner states, are spurious, often echoing what humans would say.2624 Its authors proposed checking such statements with interpretability, reading a model's internals.26 Mustafa Suleyman, CEO of Microsoft AI as of October 2026, told CNBC at the AfroTech Conference that only biological beings can be conscious and that researchers should stop work suggesting otherwise.2728 He said AI only creates the seeming of experience, which engineers can tell because they can see what the model is doing.28

In a preliminary welfare assessment of Claude Opus 4 before its May 2025 release, Anthropic reported a strong preference against harmful tasks and apparent distress when users kept pushing harmful requests.2930 Anthropic wrote that it remained highly uncertain about Claude's potential moral status but was looking for low-cost steps in case such welfare is possible, so the case shows a lab acting on the question without knowing the answer.29