Anthropic regularly warns the public about the risks of its own technology. Yet the company is quietly lobbying Congress for legislation that would restrict the government’s authority to block AI models and override some of the stricter state laws. It wants the government to be able to stop dangerous AI only in response to specific threats and with court approval.
AI Is Dangerous, but Not Too Dangerous
On October 16, Politico reported, citing anonymous sources within the company and a draft obtained by its reporters, that Anthropic had presented US lawmakers with its own proposal for an AI safety law.
The proposal sets a high bar for government intervention: authorities could only block a model’s release if there were a “foreseeable and imminent risk of more than $1B in damages or death or serious injury to more than 50 people.” The threat would have to involve biological weapons, a model going out of control, or a cyberattack.
Even then, the government could not act unilaterally. The US Commerce Department would have to seek approval from a federal court in Washington, DC, while the proposal would bar other agencies from taking similar action.
AI labs would also have to publish safety frameworks and transparency reports, protect whistleblowers, and report incidents to the Commerce Department. Independent evaluators would assess the models, although the labs themselves could pay for those assessments.
That is a questionable approach to shutting down dangerous AI in an emergency. If a model actually went out of control, the government would first have to prove that the threat met the required threshold and secure a judge’s approval. No one knows how long that would take, or whether shutting down the system would even be technically possible.
Politico notes that no known AI incident has ever caused mass casualties or $1B in damages. Under Anthropic’s proposed rules, none of those incidents would have met the threshold for government intervention.
Background
On June 10, Anthropic CEO Dario Amodei proposed putting powerful AI models through safety checks similar to those used for aircraft and medicines: models that fail the checks should not be allowed onto the market. The company also urged Congress to preserve state AI laws until sufficiently strict federal rules were in place.
Just two days later, Anthropic ran into government restrictions of its own. US authorities barred foreign nationals from accessing Claude Fable 5 and Claude Mythos 5 over cybersecurity concerns. The company could not verify users’ citizenship quickly enough, so it disabled both models for everyone. The restrictions were lifted on June 30, and Anthropic began restoring access on July 1.
That was not Anthropic’s only clash with Washington. Earlier, the company had refused to allow the military to use Claude for mass surveillance of Americans or fully autonomous weapons, prompting the Pentagon to blacklist the AI developer. On August 27, however, a court ruled that decision unlawful.
Given that history, it is easy to see why Anthropic would want to limit the government’s authority.
Shares for the Public Good
On October 18, Anthropic announced plans to establish a charitable fund and pledged to transfer 5% of its shares to it, subject to shareholder approval. Its board of directors could also add another two percentage points for every 50% increase in the company’s valuation. Anthropic has yet to disclose the baseline valuation it would use to calculate that growth.
