Supercharge your auto scaling for generative AI inference – Introducing Container Caching in SageMaker Inference

Supercharge Your Generative AI Scaling: Amazon SageMaker Introduces Game-Changing Container Caching

At AWS re:Invent 2024, Amazon Web Services (AWS) unveiled a groundbreaking feature for Amazon SageMaker: Container Caching. This innovation is set to revolutionize the way generative AI models scale for inference, slashing latency times and boosting efficiency. With up to a 56% reduction in latency when scaling a new model copy and up to 30% when adding a model copy on a new instance, this feature is a game-changer for developers and businesses leveraging SageMaker’s Deep Learning Containers (DLCs).

Container Caching is compatible with a wide range of frameworks, including Large Model Inference (LMI), Hugging Face Text Generation Inference (TGI), PyTorch (powered by TorchServe), and NVIDIA Triton. By addressing the critical bottleneck of container startup times, this feature ensures that end-users experience minimal latency, even during traffic spikes.

The Growing Challenge of Scaling Generative AI Models

Generative AI models are becoming increasingly complex, with some models now boasting hundreds of billions of parameters. This growth has made scaling these models for inference a daunting task. Traditionally, when SageMaker scaled up an inference endpoint, it had to pull the container image—often tens of gigabytes in size—from the Amazon Elastic Container Registry (Amazon ECR). This process could take several minutes, creating significant delays during traffic surges.

Scaling involves multiple steps, including:

• Provisioning new compute resources
• Downloading the container image
• Loading the container image
• Loading model weights into memory
• Initializing the inference runtime
• Shifting traffic to serve new requests

The cumulative time for these steps could range from several minutes to tens of minutes, depending on the model size and infrastructure. This delay often led to suboptimal user experiences and potential service degradation during high-demand periods.

How Container Caching Solves the Problem

Container Caching eliminates the need to download container images during scaling events by pre-caching them. This innovation brings several key benefits:

• Faster Scaling: Pre-cached containers significantly reduce the time required to scale inference endpoints, enabling systems to adapt quickly to traffic spikes.
• Quick Endpoint Startup: Accelerated startup times allow for more frequent model updates and faster deployment cycles.
• Improved Resource Utilization: By minimizing idle time on GPU instances, computational resources can focus on inference tasks immediately.
• Cost Savings: Faster deployments and efficient scaling reduce overall inference costs, making it possible to handle increased demand without proportional infrastructure costs.
• Enhanced Compatibility: The feature ensures quick access to the latest SageMaker DLCs, optimized for cutting-edge AI technologies.

Real-World Impact: Performance Evaluation

In tests conducted by AWS, Container Caching demonstrated remarkable improvements in scaling times for generative AI models. For instance, when running the Llama3.1 70B model, the end-to-end (E2E) scaling time on an available instance dropped from 379 seconds (6.32 minutes) to 166 seconds (2.77 minutes), a 56% reduction. Similarly, scaling by adding a new instance saw a 30% reduction in E2E scaling time, from 580 seconds (9.67 minutes) to 407 seconds (6.78 minutes).

These results highlight the feature’s ability to handle sudden traffic spikes efficiently, providing more predictable scaling behavior and minimal impact on end-user latency. The following table summarizes the improvements:

Getting Started with Container Caching

Container Caching is automatically enabled for popular SageMaker DLCs, including LMI, Hugging Face TGI, NVIDIA Triton, and PyTorch. To use cached containers, simply ensure you’re using a supported SageMaker container. No additional configuration is required.

The following table lists the supported DLCs:

Framework-Specific Benefits

Hugging Face TGI

Hugging Face TGI is optimized for high-performance text generation and supports a wide range of open-source LLMs. With Container Caching, scaling performance improves automatically, requiring no additional configuration. Philipp Schmid, Technical Lead at Hugging Face, remarked, “We are excited to see Container Caching speed up auto scaling for users, expanding the reach and adoption of open models from Hugging Face.”

NVIDIA Triton

NVIDIA Triton Inference Server supports multiple frameworks and model formats. Container Caching enhances Triton’s ability to scale efficiently, especially for large-scale inference workloads. Eliuth Triana, Global Lead Amazon Developer Relations at NVIDIA, stated, “This feature perfectly complements Triton’s advanced serving capabilities by reducing deployment latency and optimizing resource utilization during scaling events.”

PyTorch and TorchServe

Powered by TorchServe, SageMaker’s PyTorch DLC now integrates with the vLLM engine for advanced inference capabilities. Container Caching significantly reduces scaling times, making it ideal for large models during auto-scaling events.

LMI Containers

LMI containers offer high-performance serving solutions with features like token streaming and speculative decoding. Container Caching further accelerates scaling, making it a powerful tool for large-scale LLM deployments.

Conclusion

Container Caching is a transformative feature for SageMaker, addressing critical bottlenecks in scaling generative AI models. By reducing latency, improving resource utilization, and cutting costs, it empowers businesses to build more responsive and scalable AI systems. AWS encourages users to explore this feature and share their feedback to help shape the future of machine learning on the platform.

For more information, visit the Amazon SageMaker Documentation or explore examples on the AWS GitHub repository.