Amazon SageMaker HyperPod now supports model caching for faster inference autoscaling and reduced cold starts
Amazon SageMaker HyperPod introduces model caching to reduce inference cold starts by pre-loading model weights and container images, cutting scale-out times by up to 60% for large models.
Amazon SageMaker HyperPod now supports model caching, an optimization that pre-loads model weights and container images onto cluster nodes to accelerate pod startup. This addresses cold start delays that can last minutes when deploying large models for tasks such as chat assistants or document analysis. By storing weights locally on NVMe and pre-pulling container images, pods avoid repeated downloads from remote storage, significantly reducing initialization time.
The feature introduces two independent caching mechanisms: a weights cache that stores model weights on local NVMe for fast access, and an image cache that pre-pulls container images to skip ECR downloads. If a pod lands on a node without a warm cache, it automatically falls back to the original source, ensuring reliability without manual intervention.
Benchmarks across models ranging from 57 GB to 145 GB demonstrate around 60% faster scale-out times, with the image cache reducing image-pull delays by over two minutes—a 97% reduction. The performance gains increase with model size, while maintaining the robustness of the original source path for fallback scenarios.
Customers can enable model caching through the HyperPod Inference Operator by adding a modelCacheConfig section to their InferenceEndpointConfig or JumpStartModel resource. The operator manages the full lifecycle, eliminating manual setup or cleanup. Model caching is now generally available in all regions where SageMaker HyperPod is offered.