Generative AI Inference Recommendation for Amazon SageMaker now available in the SageMaker AI Studio
Amazon SageMaker AI Studio now offers Generative AI Inference Recommendations to help users select optimal inference configurations for production workloads with minimal manual effort.
Amazon SageMaker AI Studio has introduced Generative AI Inference Recommendations, a low-code, no-code feature designed to simplify the process of selecting the best inference configuration for generative AI models. This tool addresses the challenge of deploying models in production, where finding the right balance between instance type, serving container, and optimization strategy typically requires extensive manual benchmarking and trial-and-error. The new experience aims to reduce this process from weeks to hours by automating configuration selection based on user-defined priorities such as latency, throughput, or cost.
The feature allows users to describe their workload and optimization goals within SageMaker AI Studio, where the system benchmarks multiple configurations on real GPU infrastructure using NVIDIA AIPerf. It applies techniques like speculative decoding for throughput or kernel tuning for latency, then returns ranked, production-ready recommendations with measured performance data. Users can select from predefined use-case profiles (Interact, Generate, Summarize, or Custom) and choose an optimization goal before deploying directly to a SageMaker real-time endpoint.
Recommendations are ranked by metrics such as time-to-first-token (TTFT), inter-token latency, throughput, and cost, enabling users to compare options visually before deployment. The feature is accessible under the Jobs section in SageMaker AI Studio, where users can also select models from JumpStart, S3, Model Registry, or existing SageMaker models. No additional cost is incurred for generating recommendations, though standard compute costs apply for optimization jobs and endpoints used during benchmarking.
The capability is currently available in multiple AWS regions, including US East (N. Virginia), US West (Oregon), US East (Ohio), Europe (Ireland), Europe (Frankfurt), Asia Pacific (Singapore), and Asia Pacific (Tokyo). For further details, users can refer to the official blog post or documentation provided by Amazon Web Services.