Microsoft announced that its Azure Maia family will scale to multi‑gigawatt deployments starting in 2027. The new Maia 300 accelerator is built for inference work that needs long context and heavy decoding, not for training models. Microsoft says Maia 300 will deliver 40% more tokens per megawatt than leading GPU‑based systems and up to seven times the rack‑scale token throughput of its predecessor, Maia 200.
What Maia 300 looks like
Each Maia 300 accelerator packs eight HBM4 memory stacks, giving 288 GB of fast memory and 18 TB/s bandwidth inside a 950‑watt power envelope. A rack holds 96 accelerators across 24 trays, consuming 167 kW and offering 27.6 TB of HBM and 1.7 PB/s aggregate bandwidth. The chips are built on TSMC’s N2P process and use a NUMA memory layout so software can keep data close to the compute cores.
The system’s networking uses a custom AI transport on Ethernet. Accelerators can talk directly to any peer in a rack‑level “superpod” without involving a host CPU or separate NIC. Up to 128 racks can be linked, forming a superpod of 12,288 accelerators. Microsoft says this fabric lets a spare tray replace a failed one transparently.
Reaction to the efficiency claim
The headline metric – tokens per watt – has been called “absurd” by commenters who note that token definitions vary and the metric ignores factors like latency or cost per token. Critics also ask what context lengths were used for the 40% improvement claim, pointing out that long‑context workloads are often limited by memory rather than compute.
Why it matters
For customers of Microsoft’s own services, the new hardware could mean faster responses and lower energy costs if the claimed efficiency holds. However, Microsoft has not said whether external customers will be able to rent Maia capacity, so the benefit may stay inside Microsoft products for now.
