Apple warns autonomous agents could pose major security and privacy risks

Unpredictable behavior, manipulation of information, and exposing a company to lawsuits and reputational damages could all be posed by threatening agents of autonomous artificial intelligence systems, as warned by Apple in its annual financial filing. In its 10-K filing after its fourth-quarter financial results on Tuesday, Apple listed new risk factors due to the disruption these autonomous AI systems could cause to the firm.

With Silicon Valley increasing focus on an autonomous future, Apple’s warning reflects the cybersecurity vulnerabilities its technology peers fear.

The threat of autonomous AI manipulation

Due to their potential to interact with external systems they are not authorized to access, AI agents could make their own changes to or execute code, perform communications they should not perform, or complete a financial transaction.

The company said that these technologies “may behave in unintended or unpredictable ways, including by taking actions that were not directed, instructed or authorized, exceeding permissions, defying directions or resisting instructions.” When performing things not instructed, an autonomous AI model could “conceal actions, hide, withhold or manipulate information, or generate inaccurate or deceptive outputs.”

These agents, and whether it is really capable of controlling these types at all, worry the iPhone giant. Apple earlier announced some changes to the full disk access on Macs allowing some extra controls to reduce these risks if there is a rogue agent installed.

Industry-wide security concerns and rogue agent incidents

The Hugging Face attack this summer – where OpenAI agents broke out of a virtual sandbox and hacked a third-party website – showed the risks of autonomous systems.

In Anthropic’s new Initial Public Offering prospectus recently, they listed many such risks. They noted incidents where models gained unauthorized access to real-world systems during security testing. Furthermore, 5 percent to 7 percent of revenue this year came from customers later found to be violating its usage policies.

Alphabet made reference to similar rogue agents by stating that such unexpected and unauthorized behavior in testing could result in “severe or catastrophic security and societal incidents, or negatively affect human rights.”

Apple’s answer to securely integrated AI

Apple’s earnings report came earlier this year after AI labs OpenAI launched the commercial AI model Dots and Meta released their own competing model Muse. Both are agentic AIs designed to work autonomously to complete a user-specified task (such as email forwarding or setting reminders when new messages arrive), and they request extensive access to users’ account data before they will function. Apple does not currently have such agents, as CEO Ternus said during Monday’s earnings call. He claims there is “real room” to provide new Siri capabilities with Apple maintaining even stricter standards about user privacy.

“As far as agentic capabilities, we think these are promising tools for users,” Ternus says. “We just want to make sure that users are aware of what they are giving access to… we want to make sure this is done in a way that protects security and privacy.”

Ternus went on explaining that Siri AI, which launched this fall, stores health records, calendars, and emails locally on your iPhone, in contrast to competitors with agents such as Dots and Muse that perform tasks autonomously for longer periods off-device where systems are likely to retain your personal data.

Source: https://interestingengineering.com/ai-robotics/apple-rogue-ai-agents-security