Deep dive into Azure Maia 300 architecture
Microsoft unveils Azure Maia 300, a third-generation AI inference system co-designed across silicon, software, and datacenter infrastructure to improve efficiency and performance for large-scale agentic workloads.
Azure Maia 300 represents the third generation of Microsoft’s AI inference system, built as a unified platform integrating software, silicon, rack infrastructure, networking, and datacenters to optimize for cloud-scale agentic workloads. The system addresses the growing computational demands of multi-step execution, long-context decoding, and tool use by prioritizing data locality, memory capacity and bandwidth, and minimizing data movement within fixed power and cooling constraints. Microsoft highlights its direct experience with these workloads through products like Copilot and GitHub Copilot, which inform the design choices needed to support next-generation AI applications and agents.
Maia 300 advances the Software Defined Locally Accessed Dataflow Architecture (SDLA) introduced in Maia 200, evolving it for long-context and reasoning-heavy inference. Key improvements include shifting from uniform memory access to a locality-aware non-uniform organization across chiplets, introducing narrow-precision formats with hardware-accelerated conversion, and scaling the rack to 96 accelerators with a superpod configuration of 12,288 accelerators. These changes enable more efficient data placement, reduced data movement, and higher performance per watt compared to its predecessor.
The system’s network is integrated as a native extension of its software-defined dataflow architecture, using a custom AI transport based on Ethernet to build a two-tier high-performance superpod. This design allows Maia 300 to present a single communication model to software, enabling direct data movement to any accelerator within the superpod. The result is a scalable, efficient infrastructure that supports a unified programming model from kernel to datacenter scale.
Maia 300 delivers up to 7 times greater rack-scale token throughput and more than 2.5 times the performance per watt compared with Maia 200, while increasing memory bandwidth per watt and doubling the number of accelerators per rack. The system’s end-to-end co-design across software, silicon, rack infrastructure, networking, and datacenters aims to make advanced AI capabilities more accessible and affordable by improving efficiency and performance for cloud-scale agentic inference.