Record, train, and deploy from one place with Strands Agents, LeRobot, and Hugging Face Storage Buckets
Hugging Face introduces a continuous data loop for robotics training using Strands Agents, LeRobot datasets, and Storage Buckets, enabling efficient recording, training, and deployment without repeated data transfers.
Hugging Face has unveiled a workflow that integrates Strands Robots, an open-source SDK from AWS, with LeRobot datasets and Hugging Face Storage Buckets to create a continuous loop for recording, training, and deploying robot policies. The system records demonstrations directly into a Storage Bucket, avoiding redundant uploads by using byte-level deduplication via Xet technology. This approach allows users to collect episodes throughout the day, train policies on growing datasets, and deploy updated models back to hardware without repeatedly transferring large files.
The Storage Bucket serves as an intermediate storage layer within the Hugging Face workspace, enabling in-place updates and overwrites without versioning overhead. This contrasts with versioned dataset repositories, where even minor changes can trigger full file re-uploads. Xet’s content-defined chunking ensures that only changed bytes are transferred during syncs, reducing data transfer by up to four times in Hugging Face’s measurements. For example, modifying 1% of a 500 MB file results in only a 5.5 MB upload, compared to a full 500 MB transfer without deduplication.
Training occurs by streaming datasets directly from the Hub to GPUs, eliminating the need to download entire datasets. LeRobot’s StreamingLeRobotDataset and Strands Robots’ stream_dataset() function enable this by reading batches as byte-range requests over large shards, such as Parquet and MP4 files. This method minimizes local storage usage and reduces GPU idle time, as training can begin immediately without waiting for full dataset downloads. The streaming approach also supports shuffling and parallel decoding, aligning with standard PyTorch training workflows.
The companion notebook, examples/notebooks/05_streaming_data_loop.ipynb, demonstrates the full loop: recording a LeRobotDataset, syncing it to a Storage Bucket, streaming it for training, and deploying the trained policy back to hardware. The process maintains the same LeRobot dataset format throughout, ensuring compatibility with existing tools. Users can switch between mock and real policies with minimal code changes, and the loop can run on a laptop, making it accessible for iterative development and testing.