Amazon SageMaker AI inference now supports GPU partitioning for real-time endpoints

Amazon SageMaker AI inference now supports GPU partitioning through NVIDIA Multi-Instance GPU (MIG) technology, enabling customers to divide a single GPU into up to seven isolated partitions. Each partition operates independently with dedicated memory and compute resources, allowing multiple models to run on one accelerator. Previously, each model required a dedicated GPU, leading to underutilized capacity and higher costs for serving small or mid-size models such as fine-tuned variants or classifiers.

The new capability lets customers configure GPU partitions when creating an inference endpoint, matching partition size to each model's requirements. Hardware-enforced isolation ensures that a busy model cannot consume resources from neighboring partitions, maintaining performance predictability. This approach also makes newer instance generations, which may have been oversized for single-model workloads, more cost-effective by enabling efficient resource sharing.

Customers can invoke models by name without modifying application code, as the partitioning configuration is managed at the endpoint level. The feature supports P4d, P5, G7, and G7e instances across multiple AWS regions, including US East (N. Virginia), US East (Ohio), and Asia Pacific (Singapore).

Amazon SageMaker AI deployment documentation provides guidance on configuring GPU partitioning for inference endpoints. The feature is designed to optimize GPU utilization while maintaining isolation, addressing common challenges in serving multiple smaller models efficiently.