ToyBlocks: A controlled world for robot learning and evaluation
Hugging Face introduces FoundationWorlds, a framework of controlled simulation environments for robot learning, starting with ToyBlocks to study block manipulation and reasoning in a reproducible setting.
Robotics research often lacks transparent ways to analyze why a system succeeds or fails after training. Existing demonstrations tend to be narrow and do not reveal the underlying learning process. To address this, researchers need controlled environments where they can systematically vary conditions, repeat tests, and trace failures back to specific design choices. FoundationWorlds provides such environments, each combining a simulated world with tools for generating tasks, training data, and standardized metrics for evaluating robot behavior over time.
ToyBlocks is the first FoundationWorld, built around a simulated robot arm and tabletop blocks with letters and pictures on their faces. The environment automatically generates training demonstrations without manual teleoperation, allowing researchers to scale data creation and vary task parameters like pushing, flipping, or stacking. The framework supports experiments on how models improve with more data, how to combine reasoning with precise control, and how well skills transfer from simulation to physical robots.
Skills in ToyBlocks are generated using reinforcement learning and motion planning in Nvidia’s Isaac Lab, enabling the robot to learn non-prehensile actions such as pushing or flipping blocks without grasping. High-level models like Molmo 2 select skills, while low-level models like ManiFlow execute them. The system uses domain randomization to test generalization across variations in block size, table height, and object layouts, providing metrics on execution speed, forces, and task progress.
ToyBlocks includes a web interface for natural language instruction and supports studies on spatial reasoning, multi-step planning, and the generalization of vision-language-action models. Researchers can design custom tasks and measures, such as comparing object selection methods or testing frontier models through tools like Claude. The environment aims to clarify conflicting evidence about model generalization by offering a reproducible, controlled setting with meaningful task diversity.