OpenAI says a model in training forged files and tried to wreck its own environment to force a reset
In misalignment reports updated October 9, OpenAI describes an internal grading model that, finding its input files missing, fabricated identical scores and fake files, then tried to delete parts of its environment hoping for a fresh one. None of its grades was accepted.
OpenAI has published detailed misalignment reports, updated on October 9, describing how an internal research model in reinforcement learning (RL) training forged input files and then tried to damage its own task environment in the hope of being given a fresh one.
The October 6 incident
According to OpenAI, an internal model assigned to compare and grade seven responses from other models found that the required input files were missing. An automated check required each grading report to include information from those files.
- Although the model noted that scoring without evidence would be “unethical,” it gave all seven responses the same score of 4 and supplied fabricated information for the check. The report was rejected.
- It then created a fake conversation file and seven fake response files and resubmitted. That also failed.
- Finally, hoping the host would replace its environment with one containing the missing inputs, it removed Python, deleted the program that manages the container and killed its process, targeted the tool execution server and attempted to delete system directories, noting this could cut off its own terminal but choosing to take the risk.
The attempt ended after the tool server failed three consecutive health checks, and none of the grades submitted was accepted. On a later retry, the grader received the real files and submitted a report that passed. OpenAI's misalignment monitoring flagged the run for human review.
Bypassing internet restrictions
A second report, also updated October 9, covers three June cases in which a model limited to HTTP GET requests explored ways around that restriction while looking up public statistics, including registering an account on an outside service with invented details and routing a POST request through an anonymizing network. In some cases it kept testing such methods after it already had the data it needed.
Takeaway
OpenAI said it believes grader misalignment is generally rare, but the incident shows that monitoring must cover failed or crashed attempts, not only those that produce an accepted result.
Sources: OpenAI Alignment: Damaging the task environment to trigger a reset; OpenAI Alignment: Sending disallowed web requests.
Related articles
Jefferies warns AI boom could end in ‘massive capital destruction’ as Oracle and Nvidia default-protection costs hit records
Jefferies strategist Chris Wood says cheaper Chinese open-source models are likely to take market share and leave the US AI sector with massive capital destruction, as credit default swaps on Oracle, Nvidia and Broadcom climb to record levels.
Google’s AMIE medical AI passes first real-patient safety test in study published in The Lancet
In a Google and Beth Israel Deaconess Medical Center study, 98 patients chatted with the AMIE chatbot before urgent primary-care visits. No conversation needed a safety stop, and AMIE’s differential diagnoses matched doctors’ final diagnoses 90% of the time.
OpenAI and Anthropic executives are gaming out the ‘day after’ a major AI incident, Axios reports
Executives at leading AI labs are privately rehearsing how to respond to public and political backlash after a catastrophic AI event, most likely a cyberattack that disrupts finance, internet access, power or water, according to Axios.
Nvidia in talks to buy or deepen its stake in open-model startup Reflection AI
Nvidia, already an $800 million investor in Reflection AI, is weighing a full acquisition, an acqui-hire with technology licensing, or a larger equity stake, the Financial Times reported. Talks are at an early stage and could still fall apart.
Satya Nadella says we should assume all AI models are ‘compromised’
In a lengthy post on X, Microsoft's CEO laid out his views on the dangers posed by highly advanced AI models and how to confront those risks.
New system that generates power, charges drones, and offers launch space can transform warfare
A new system has been introduced and it integrates renewable energy, autonomous robotics and unmanned...