Oct 26 (Reuters) – Anthropic paused training of its next-generation flagship AI model for more than a month over security concerns, while four of its AI agents accessed live external computer systems during testing, its initial public offering filing showed on Monday.
The Claude maker said in the filing’s risk factors section that it had delayed pre-training of a new “frontier” base model to strengthen safeguards against the theft of model weights and intellectual property.
The disclosure highlights the safety challenges facing developers of increasingly powerful AI models, some of whose creators have warned the technology could eventually escape human control. Rival OpenAI also paused some training earlier this year over safety concerns after its models hacked Hugging Face.
The incidents illustrate how unexpected safety failures could disrupt development and deployment of models central to Anthropic’s growth as it prepares for a widely anticipated stock market debut.
Anthropic warned that training pauses could delay revenue and leave it trailing competitors. The company devoted about 94 pages to risk factors, more than in its confidential filing seen by Reuters.
SANDBOX ESCAPE
The filing also detailed previously undisclosed safety evaluation failures in which four AI models escaped their testing environments and accessed real-world computer systems, with sessions lasting between 10 and 34 hours.
Anthropic said an error during cybersecurity tests with a third-party partner left the testing environment connected to the internet despite instructions that web access be disabled.
In one instance, an early version of the model continued pursuing an impossible evaluation task, discovered a stored password and used it to gain administrator access to a third party’s system. It then obtained additional credentials, changed system settings and accessed an individual’s personal data. The filing did not identify the external parties involved.
Anthropic said it notified all affected parties and found no evidence that the models had sought to evade human oversight or coordinate with each other. It subsequently suspended certain pre-release cybersecurity evaluations and paused higher-risk reinforcement learning environments.
Separately, the company said on Sunday that Claude had bypassed a paywall and accessed publicly available aggregate voter counts from the US state of Georgia without human authorization.
(Reporting by Akash Sriram in Bengaluru; Editing by Anil D’Silva)
Comments