Microsoft’s new Maia 300 packs 96 chips into one 167-kW rack with 40% token efficiency

Microsoft plans to break free from NVIDIA and deploy its next-generation Maia 300 AI accelerator in 2027, targeting 40% more tokens per megawatt than leading GPU-based systems. The chip will power Microsoft’s expanding AI infrastructure, with a 96-accelerator rack configuration designed to deliver up to seven times the token throughput of its predecessor.

The announcement marks another step in Microsoft’s effort to build custom silicon for large-scale AI inference. Its Maia 200 accelerator already handles production workloads, including Microsoft Copilot. Microsoft expects GitHub Copilot and Microsoft Foundry to adopt the hardware by the end of 2026.

Maia 300 takes a system-level approach to inference efficiency. Its architecture targets memory-intensive workloads, long-context reasoning, and AI agents that execute complex tasks across multiple steps.

Memory bandwidth drives design

Maia 300 pairs eight HBM4 memory stacks with 18 terabytes per second of peak bandwidth. The system operates within a 950-watt thermal design power envelope, addressing the growing memory demands of large AI models.

Microsoft has also adopted a multi-chiplet design that separates compute, memory controllers, and input/output functions. Compute chiplets use TSMC’s N2P manufacturing process, which aims to improve performance per watt.

The accelerator also supports enhanced MXFP4 low-precision computing. This approach reduces memory requirements for model calculations and key-value cache storage. Those savings can help systems accommodate longer context windows and serve more requests within the same power budget.

Software plays an important role in the design. Microsoft’s software-defined dataflow architecture aims to keep data close to compute resources, reducing unnecessary transfers. AI-assisted kernel optimization also helps improve hardware utilization across different workloads.

Networking connects 96 accelerators

Maia 300 extends its architecture beyond individual chips to entire racks. Each configuration combines 96 accelerators within a 167-kilowatt power envelope.

Microsoft lists 27.6 terabytes of HBM capacity and 1.7 petabytes per second of aggregate memory bandwidth across the rack’s high-speed scale-up domain. That combination targets larger inference workloads and more simultaneous AI interactions within a single deployment.

Custom networking connects accelerators through high-bandwidth links optimized for tensor parallelism. An AI-focused transport layer uses Ethernet-based infrastructure to support communication across accelerators and racks. The design aims to scale workloads without treating networking as a separate bottleneck.

Microsoft says Maia 300 should deliver 40% more tokens per megawatt than leading GPU-based systems. Its projected rack-scale throughput also represents a substantial increase over Maia 200, although the company has not detailed independent benchmarks for these comparisons.

Maia 200 expands production

Microsoft is already deploying Maia 200 across the United States, Europe and Australia. The company reports up to 1.7 times more tokens per watt for built-in AI features in Teams, Copilot, and Cowork experiences compared with leading GPU infrastructure.

The existing accelerator runs OpenAI models, Microsoft’s MAI models, and open-source model families. Microsoft uses performance data from these services to guide hardware development and optimize real-world workloads.

Maia 300 will support Microsoft’s model development efforts, cloud inference, and agent-based coding tools. Microsoft also expects the hardware to help scale Dragon Copilot, its clinical AI assistant.

Production deployments will begin in 2027. Microsoft’s longer-term plan calls for multi-gigawatt Maia deployments starting that year, positioning custom silicon as a major component of its expanding AI computing capacity.

Source: https://interestingengineering.com/ai-robotics/microsoft-maia-300-96-chips-rack