Run AI workloads on any cloud, store on Hugging Face: zero-egress storage with SkyPilot

Hugging Face Storage now functions as a first-class backend for SkyPilot, enabling teams to mount datasets and models directly from the Hugging Face Hub or Buckets into SkyPilot jobs using a single hf:// URL. This integration eliminates the need to transfer data across clouds, as Hugging Face charges no egress fees for reads, regardless of where the compute runs. Authentication is handled via the existing HF_TOKEN, simplifying access across multiple cloud providers and on-premises clusters.

SkyPilot mounts Hugging Face Storage using the hf-mount FUSE backend, which presents the bucket or repo as a local filesystem path. The system fetches only the data accessed by the job, reducing network traffic, and maintains an on-disk cache to speed up repeat reads. This lazy loading approach allows training or inference tasks to begin immediately, even while large files are still streaming, minimizing GPU idle time and associated costs.

The integration supports both MOUNT and MOUNT_CACHED modes, with the latter caching data locally for faster access. For operations requiring full downloads, the COPY mode is available, which uses the huggingface_hub client to transfer data upfront. The setup requires no migration of existing data, as most teams already store models and datasets on the Hugging Face Hub, and the token-based authentication remains consistent across all supported clouds.

A benchmark fine-tuning task for the Qwen/Qwen3.5-4B model demonstrated the system’s cross-cloud flexibility, running successfully on AWS, GCP, and Lambda with the same SkyPilot YAML configuration. Hugging Face Buckets, built on Xet technology, use content-defined chunking to deduplicate checkpoints and model variants, reducing storage and transfer costs. The integration is open source, with contributions from Nikhil Jha and upstream fixes from Hugging Face, and is supported on SkyPilot’s 20+ cloud and on-premises backends.