Fireworks AI breach exposes Notion customers' AI chat transcripts

Fireworks AI, a platform developers use to run open AI models, has confirmed that a cyberattack exposed some customer data. At least one affected customer, the documents and notes service Notion, says chat transcripts from its Notion AI assistant were among the stolen data.

How the breach happened

According to Fireworks' security advisory, a third-party cloud provider hosting its research and development machines suffered a security incident on October 23. The next day, the attacker used that access to reach Fireworks development machines and run an unauthorized workload, exposing some customers' billing information, resource metadata, model weights and datasets.

Fireworks says the evidence points to a financially motivated group aiming to sell tokens from unauthorized access to large language models to illicit routing gateways, a practice known as "LLMjacking." The company also suffered a separate breach in early October in which API keys and tokens for a limited number of accounts were accessed.

Impact on Notion users

Notion said Fireworks first notified it on November 2 that only synthetic testing data was involved. By the morning of November 3, Notion had disconnected its systems from Fireworks and deleted all data held there. On November 8, Fireworks confirmed a final impacted dataset that included Notion AI chat transcripts used for the Insights feature.

Notion says fewer than 1% of its workspaces were affected. Most of the transcripts were unattributable, meaning they were not labeled with the author or workspace and did not contain tool-call results. Still, Notion advises affected customers to treat the data as exposed and to rotate any passwords, API keys or other credentials pasted into Notion AI during the period. It has disabled Insights and stopped serving open-weight models through Fireworks.

A supply-chain warning for AI

TechCrunch reports that Fireworks serves tens of thousands of customers, including legal tech company Harvey and AI company Cursor. VentureBeat notes that Fireworks offers zero data retention by default, but features such as batch jobs and training do store customer content, and that stored side is what the attackers reached. The incident shows how a breach at an AI infrastructure vendor can reach end users of software built on top of it.

Sources: TechCrunch, Notion, BleepingComputer, VentureBeat