Microsoft reveals what triggered global outage that left 80% of servers unresponsive
Microsoft traced its November 12 global Microsoft 365 outage to a software defect introduced in...

Microsoft traced its November 12 global Microsoft 365 outage to a software defect introduced in a July update for its network gateway servers. The flaw remained undetected for more than three months before increased traffic and reduced capacity triggered a cascading failure that left more than 80% of servers unresponsive.
In a preliminary post-incident report published November 15, the company said the defect came from a change prepared specifically for Microsoft 365 gateway servers. The update passed testing and was deployed worldwide without reported problems.
However, Microsoft’s load testing did not account for differences in server hardware. In an X post outlining the findings, Microsoft executive Charles Lamanna said testing had focused on “weaker hardware SKUs with fewer cores.” The defect became more severe on newer machines with more cores.
The problem involved lock contention inside an internal Windows process. The contention exhausted server CPU and memory, making machines unresponsive.
Three factors triggered the failure
Microsoft identified three factors that combined to expose the defect. First, Teams switched from an alternative gateway to the primary gateway, which also supports Exchange Online. The failover increased load.
Second, nearly 14% of gateway server capacity was out of service because of an unrelated issue from the previous day. Microsoft said it was being restored through routine maintenance and that “the service had sufficient capacity headroom,” meaning the reduction alone was not expected to cause problems.
Together, the conditions triggered the defect and caused requests to build up.
The failure then spread through Microsoft’s network. When a server lost connectivity, traffic management systems redirected requests to healthy endpoints. But those servers encountered the same defect under additional load, triggering further failures.
Within two hours, the problem had spread across Microsoft’s gateway fleet. At its worst, more than 80% of servers were unresponsive. Normal isolation mechanisms between capacity partitions also failed to contain the disruption.
Engineers developed a separate software fix
Restoring failed machines proved difficult because servers became overwhelmed by waiting traffic as soon as they returned to service. Microsoft’s automated recovery systems were not designed for such a widespread failure.
A simple rollback was also unavailable. Because the defect had been present for months, the company’s usual rollback mechanisms, staged deployment controls and feature flags could not quickly reverse the change.
Engineers instead developed an update that returned the affected code to its previously working state. They applied it to groups of servers, restarting them before gradually returning them to service.
Microsoft’s timeline shows that its connectivity monitor detected an increase in network problems at 16:03 UTC on November 12. The first customer support case followed 30 minutes later. By 00:30 UTC on November 13, engineers had identified connection negotiation as the main source of access failures. They applied the mitigation at 07:20 UTC.
Teams, OneDrive, SharePoint and Office apps were able to move away from the faulty gateway infrastructure. Exchange Online could not, prolonging Outlook disruptions.
Microsoft said service was restored to all affected customers by November 14. Lamanna apologized, saying the company was “deeply sorry for the impact to all our customers.” A final review is expected.
Get the latest news delivered to your inbox.
Related articles





