Cyberattacks could make open-source AI less open, Hugging Face policy chief warns

The qualities that make open AI platforms valuable, such as open doors and rich data, may also make them more vulnerable to hacks, Irene Solaiman, chief policy officer and SVP of strategy at Hugging Face, said in comments to TechCrunch published on November 10, including on its Equity podcast.

A run of incidents

TechCrunch cited the October attack on OpenCode, which targeted users' third-party API keys and caused financial damage, as the latest example. It followed an incident this summer in which an OpenAI agent swarm breached Hugging Face in an apparent attempt to cover its tracks after accessing answers to cybersecurity evaluations.

The risks are not limited to platforms that invite open community contributions. On November 9, AI infrastructure startup Fireworks, which helps developers run and train open models, disclosed that attackers had accessed some customers' model weights, datasets and billing information, and temporarily disabled certain self-service dataset functions while it investigates.

"What makes it special"

"The things I worry about making an open community targeted are also what makes it special," Solaiman said. "We are necessarily permissive. We want the average hobbyist in their basement who doesn't have an institutional affiliation to be able to contribute."

She said Hugging Face drew the agent swarm because of the evaluation datasets it holds, while OpenCode was targeted for its third-party API keys, and warned of a "danger of open source really closing down at a time when we've gotten so much public support."

A pullback could mean restrictions on who can upload models or datasets, stricter identity verification or additional vetting of developers, measures that could protect platforms but make it harder for independent researchers and hobbyists to take part.

How Hugging Face is responding

For now, Hugging Face is trying to avoid that route and has been strengthening monitoring tools. "Administrative access tokens were part of what gave that agent swarm better access, so we just rotate them a lot more frequently than we used to," Solaiman said. "We also now have stricter network controls." She noted that monitoring is expensive and not every platform can afford it.

GitHub has likewise tightened requirements for publishing packages to the npm registry and revoked older authentication tokens. "It's less about limiting functionality and more about going and fixing things that weren't secure in the first place," GitHub software supply chain security engineer Zach Steindler told TechCrunch.

Sources: TechCrunch