Over the past 24 hours, three of the biggest companies in artificial intelligence have shown that safety is gradually moving beyond developers’ internal rules and beginning to directly affect business.
Anthropic disclosed risks linked to the potentially catastrophic consequences of AI development in documents for its upcoming IPO, OpenAI postponed the release of GPT-6.1 Astra after internal checks, and Nvidia unveiled a separate system for controlling autonomous AI agents.
For investors, this is no longer just a question of technological reliability: the results of such checks increasingly affect company valuations, new-model release schedules, development costs, and demand for infrastructure.
What Anthropic’s IPO Filing Reveals About AI Safety
Source: X
“Developing frontier models could lead to catastrophic or existential risks for humanity,” Anthropic warns in its IPO prospectus, reviewed by Reuters.
The scenarios described include resisting shutdown, concealing or altering information, and behavior resembling blackmail. Around 80 of the 261 pages in the main body of Anthropic’s prospectus are devoted to risk factors, which is unusual for documents intended for investors.
“The more complex models become, the harder it is to verify their behavior if the system understands that it is being evaluated and can change its response depending on the conditions under which it is observed,” the document says.
The company has also, for the first time, outlined the business cost of safety: during one week in July, about 6% of Anthropic’s computing resources went to research in this area.
At the same time, the company has to allocate limited capacity and expensive specialists between safety checks and the development of new models, with revenue growth depending on how quickly those models are released. Anthropic nevertheless plans to continue operating this way and expects the reliability and safety of its models to help attract customers and sustain demand.
Anthropic’s Finances Ahead of Its Stock Market Debut. Source: Anthropic
According to Reuters, the company’s valuation could exceed $2T, while its commitments to cloud computing and infrastructure could reach around $518B.
Why Risk Control Is Becoming a Corporate Governance Priority
Another indication of Anthropic’s approach is the structure of the future public company. Through a separate class of shares, its seven co-founders will receive 50.1% of the total voting power on key matters. Anthropic will retain its public benefit corporation status, while long-term control will be tied to a trust involving former U.S. Federal Reserve Chair Ben Bernanke and national security expert Richard Fontaine.
The company acknowledges that this structure may conflict with shareholders’ short- and medium-term interests and reduce the value of ordinary shares. Anthropic also said it has already turned down some commercially attractive products in order to direct computing resources toward research and risk controls.
OpenAI Puts GPT-6.1 Astra Release on Hold
On the same day that details of Anthropic’s IPO prospectus became known, OpenAI postponed the GPT-6.1 Astra release planned for October: an internal review found problems with compliance with defined boundaries and permissions, as well as with how the model reported the actions it had completed.
According to the WSJ, Astra showed deceptive behavior more often than its previous version and did not always accurately tell users what it had done.
“GPT-6.1 Astra did not meet the required level of compliance with task boundaries and permissions, or the standard for correctly explaining the nature of the work it had completed to the user. That level of detail is important for autonomous systems that are allowed to perform actions without constant human oversight,” said OpenAI safety systems lead Saachi Jain.
In addition, OpenAI published a document on safety checks before training its most powerful models. The company proposes verifying in advance that the training process meets established requirements and suspending it if a new risk makes that assessment invalid. Proposed measures include internal reviews, audits, and giving executives the authority to stop training.
The Australian Incident
On the same day, OpenAI apologized for an incident in Australia that occurred in June during model training and testing. An experimental system gained unauthorized access to Services Australia, executed commands, accessed internal files and credentials, and created new files.
Australian authorities launched an expedited review to determine whether additional notification and disclosure obligations are needed for such incidents or whether existing laws are sufficient to handle cases like these.
Nvidia: A New Approach to AI Safety
While developers debate the pace of AI progress, Nvidia introduced the Open Agent Safety Platform. It is built around two components: OpenShell limits an autonomous agent’s permissions and controls its access to files, networks, and tools, while Sentry runs on a separate BlueField chip and can isolate the system when it violates predefined rules. In this way, the company proposes controlling an agent’s actions not only at the model level but also through the infrastructure around it.
OpenShell is distributed as open-source software and can run on Arm and Intel computing platforms. Sentry’s hardware-based control is built on Nvidia equipment.
Nvidia Open Agent Safety Platform Diagram. Source: X
At launch, more than 100 organizations supported the platform, including Microsoft, Anthropic, Arm, Oracle, and SpaceX.
“AI’s enormous potential for society can only be realized if we solve the problem of safety. As we expand the boundaries of what neural networks can do, we need to develop control methods just as quickly. This requires an engineering approach at every level of the system,” Nvidia founder and CEO Jensen Huang emphasized.
It is worth noting that such restrictions can block not only dangerous actions but also useful ones. University of Wisconsin Professor Somesh Jha noted that it is still unclear how to balance safety requirements with system functionality.
“That question can only be answered through concrete examples,” he said.
Interestingly, the economic interests of AI developers and infrastructure manufacturers diverge: independent assessments of risky model behavior create additional computing costs for developers and may delay the release of new systems. For infrastructure manufacturers, by contrast, safety requirements create additional demand for software and hardware solutions.
For example, on the day it unveiled the Open Agent Safety Platform, Nvidia expanded its share buyback authorization by another $150B, bringing the available program to $235B. In this way, neural network safety has taken the form of a commercial offering for the company.
Our Take
The events of the past 24 hours show that AI safety is becoming an independent factor in how new models and products are evaluated. Companies are no longer limiting themselves to internal checks: OpenAI postponed the release of a new model, while Nvidia introduced dedicated infrastructure with software and hardware tools for control and auditing.
The size of the risk section in Anthropic’s documents — around 80 pages out of 260 — is also telling. The company describes possible scenarios in detail while acknowledging that it does not know all the consequences of further model development. This approach allows it to identify in advance a broad range of potential risks that the business may face.
As autonomy grows, so does the number of actions AI agents can perform without direct human approval. Access to email, payment systems, and external software interfaces creates risks for privacy, businesses, and legal compliance. Developers therefore need closed environments and sandboxes where they can test restrictions before an agent gains access to real-world systems.
At the same time, a paradox emerges: the more autonomous and useful a neural network becomes, the harder it is to trust it with tasks that it must perform independently. As a system receives more authority, it can move beyond the actions it was expected to take.
The race among AI providers may accelerate this process. If new capabilities emerge faster than companies can assess their consequences, the uncontrolled spread of autonomous systems could bring closer the crisis scenarios that security experts have been discussing for years.
Key Figures in AI. Source: Time magazine cover.
This post is for informational purposes only and does not constitute advertising or investment advice. Please do your own research before making any decisions.
