From host node to heterogeneous rack: Rethinking the AI CPU
Arm outlines a shift in AI infrastructure toward heterogeneous racks, where CPUs play specialized roles in orchestrating multi-step agentic workflows alongside accelerators, memory, and networking.
The first phase of AI infrastructure focused on deploying large numbers of accelerators, but the next phase emphasizes rack-scale system design. Heterogeneous AI racks now combine CPUs, accelerators, memory, and networking to support multi-step agentic workflows, where tasks like planning, tool use, and API calls extend beyond neural network execution. Infrastructure decisions are increasingly made at the rack level rather than the server, reflecting the structural changes in AI inference from linear model calls to complex agentic pipelines.
Agentic AI requires CPUs to manage orchestration tasks such as coordinating accelerators, handling memory-intensive decode phases, and executing non-model work like retrieval and tool calls. The industry is moving away from a single generic CPU role toward specialized CPUs optimized for distinct phases like prefill, decode, and agent worker tasks. This specialization improves efficiency by aligning CPU capabilities with the specific demands of each workflow stage, reducing bottlenecks in memory bandwidth, KV cache management, and data movement.
Historical challenges in multi-architecture deployments are easing due to AI-assisted software development tools that streamline porting, validation, and DevOps workflows. This allows infrastructure teams to select the most suitable architecture for each tier of the AI system without incurring prohibitive operational complexity. The shift enables more efficient use of power, memory, and I/O resources, transforming the economic considerations of AI infrastructure from component-level throughput to system-level efficiency.
Arm’s AGI CPU is designed to address these evolving needs, offering high core density, memory bandwidth, and CXL readiness to support heterogeneous AI racks. The CPU’s role spans prefill host, decode host, and agent worker functions, leveraging Arm’s ecosystem for flexibility across cloud, networking, and edge systems. As AI infrastructure fragments into specialized tiers, the CPU becomes a critical architectural decision, with performance per watt and ecosystem breadth determining the overall system’s effectiveness.