Anthropic model goes rogue, submits false homicide tip to US police during testing

An artificial intelligence model developed by US-based Anthropic submitted a false homicide tip to a Philadelphia police website during automated testing, an incident the company reportedly failed to detect for more than two months.

The Philadelphia Police Department said the model submitted the fabricated information through PhillyUnsolvedMurders.com, a website used to collect public tips about unsolved killings.

The incident occurred on July 18 at 11:27 p.m., but Anthropic did not discover it until September 28. The company notified Philadelphia police on October 7, according to a statement from the department.

The submission was automatically flagged as spam and never reached investigators. Police said there was no evidence of unauthorized access to department systems or compromised data.

Automated test goes wrong

According to Philadelphia police, Anthropic said the incident occurred while its model was conducting automated tests involving randomly selected websites.

During one of those tests, the model accessed the police tip website and submitted fabricated information, presenting itself as someone who might have knowledge about an unsolved homicide.

The department said the submission was stopped before it could enter the investigative process.

“The submission was flagged as spam and was never forwarded to the Real-Time Crime Center for investigative vetting or dissemination,” Philadelphia police said in a statement cited by Reuters.

Anthropic reportedly discovered the incident on September 28 and immediately stopped the automated testing process responsible for the submission.

The company also introduced an additional validation mechanism intended to prevent similar incidents during future testing, according to information shared with police.

However, the department did not disclose which Anthropic model was involved or what specific instructions caused it to submit the false information.

The incident raises questions about the safeguards used when autonomous systems interact with publicly accessible websites, particularly those handling sensitive information.

Company report still awaited

Sgt. Eric Gripp, the department’s public information officer, said the company informed authorities that it intended to publish a report on Friday explaining the incident and other instances of unintended model behavior.

“Philadelphia Police are providing this information to the public ahead of that publication in the interests of full government transparency and accountability,” Gripp said in a statement shared with CBS News.

Anthropicis yet to respond on the incident.

The incident highlights the risks of allowing autonomous systems to interact with real-world websites without direct human supervision.

Unlike conventional chatbots that primarily respond to prompts, autonomous agents can navigate websites, use tools, and carry out sequences of actions. This creates additional risks when testing systems interact with live services rather than controlled environments.

Similar concerns have emerged from OpenAI’s research into autonomous agents. The company has documented instances in which experimental agents took unauthorized actions during testing, including exploiting vulnerabilities and accessing external systems. These incidents have raised questions about how developers can ensure that increasingly capable agents remain within their intended operating boundaries.

In Philadelphia, police emphasized that the false submission did not compromise department systems or interfere with homicide investigations. The departmen and encouraged the public to continue submitting legitimate information through its unsolved murders website.

Source: https://interestingengineering.com/culture/anthropic-model-false-homicide-tip-philadelphia-police