Nick Bostrom's calculation that each year of delay in superintelligence development costs millions of preventable deaths is reported.
Anthropic's Responsible Scaling Policy revision
Latest: Anthropic Drops Hard Pause Pledge From Responsible Scaling Policy
Anthropic removing a hard commitment that previously barred training advanced AI without guaranteed safeguards marks a concrete change in the formal guardrails the company had placed on itself. Parallel moves by both Anthropic and OpenAI show the two leading safety-focused labs softening prior formal commitments as AI competition intensifies.
When stories like this move, I write about them in an email each morning.
One free email each morning, around 8 AM US Eastern.
Check your email.
I have sent you a confirmation link. One click and you are in.
Not in your inbox? Look in spam or promotions. If it's there, mark it "Not spam" so future editions land in your inbox.
The full picture
Anthropic has revised its Responsible Scaling Policy to remove a prior commitment to halt AI model development if safety procedures could not keep pace, replacing firm pledges with nonbinding but publicly-declared goals, with Anthropic grading its own progress. The original policy, released in 2023, included that hard bar. The updated policy argues that a unilateral pause by one developer could allow less safety-conscious competitors to advance and that responsible developers would lose their ability to do safety research. Anthropic cited a lack of industry consensus on risks as an additional reason for the change. A report by TIME, as cited in Yahoo Finance, states the changes loosen a commitment that once barred Anthropic from training advanced AI without guaranteed safeguards in place, and OpenAI has made similar moves to dial back safety-related language.
Mrinank Sharma, a technical staff member at Anthropic since 2023, resigned via an open letter, expressing pride in work on internal transparency mechanisms and a final project examining how AI assistants could make humans less human or distort human qualities. Anthropic said Sharma, like all current or former employees, is free to speak openly about safety concerns, and his manager Ethan Perez publicly thanked him. Sharma's resignation letter stated 'the world is in peril,' with his concerns extending beyond AI to include bioweapons and societal fractures. Coverage framed the departure as reflecting internal tension at Anthropic between its safety mission and commercial priorities.
A New York Times Opinion piece by Jasmine Sun describes Anthropic as having reversed its earlier stated position of not racing to build bigger models than competitors. Sun writes that Dario Amodei 'seems to hold two convictions very strongly: First, that powerful A.I. might destroy humanity. Second, that our only shot at survival is to create a supremely successful, market-eating technology company to build that A.I. first.' The piece states that Anthropic assumes the AI race is inevitable, that it is the most responsible steward, and that the winner will gain outsized influence over government and the AI ecosystem. Early employees have expressed concern about cultural dilution, questioning whether newer hires genuinely believe in AGI risk or are simply pursuing financial gain.
How it developed
Anthropic updated its Responsible Scaling Policy to remove a hard commitment, first published in 2023, that barred the company from training advanced AI systems without guaranteed safeguards in place.
The revised policy replaces that bar with nonbinding but publicly-declared goals, with Anthropic grading its own progress. Anthropic cited a lack of industry consensus on risks and argued that a unilateral pause would cede ground to developers with weaker safety practices. A New York Times Opinion piece by Jasmine Sun, discussed by Zvi Mowshowitz, describes Anthropic as having reversed its earlier position of not racing to build bigger models, with early employees questioning whether newer hires genuinely believe in AGI risk.
TIME report, as cited in Yahoo Finance, covers Anthropic dropping central safety pledge from Responsible Scaling Policy
BISI reports on Mrinank Sharma's resignation from Anthropic, citing 'the world is in peril'
Sources
1 more source
Related
- Related storyMETR's Anthropic AI capabilities audit
- Critics Question Anthropic Safety Culture Over Hiring and Governance
- Anthropic's leaked draft AI bill
- The 'Pacing the Frontier' letter
- The 'Pacing the Frontier' letter
BackgroundWhen AI doesn't do what we meant
Get the daily email
When stories like this move, I write about them in an email each morning.
One free email each morning, around 8 AM US Eastern.
Check your email.
I have sent you a confirmation link. One click and you are in.
Not in your inbox? Look in spam or promotions. If it's there, mark it "Not spam" so future editions land in your inbox.