NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation

NVIDIA’s Cosmos3-H-Dreams advances interactive generative simulation by training on clinical procedure videos from the CMR Surgical Versius platform, enabling real-time control of surgical instruments within complex in-body scenes. Unlike traditional numerical simulation, it learns visual dynamics directly from synchronized video and robot kinematics, capturing tissue deformation, reflections, and occlusion in real time. The system is designed to handle the procedural complexity of clinical surgery, where small instrument movements can significantly alter the scene. Powered by Cosmos 3 and accelerated with FlashDreams, it runs interactively on a single NVIDIA RTX PRO 6000 GPU.

Cosmos3-H-Dreams builds on a 16-billion-parameter model pretrained on Open-H-Embodiment, specializing a bidirectional teacher on curated cholecystectomy video and kinematics. The model uses 28 action dimensions, including pose and gripper information for both arms, energy controls, and thumbstick inputs, with actions aligned at 10 Hz. Training employs 832 x 480 video, and the system supports long autoregressive rollouts of up to 72 frames (7.2 seconds) during training to learn surgical dynamics. Two student checkpoints are provided: a four-step model for higher visual quality and a two-step model for faster inference, allowing developers to balance performance and speed.

The simulator introduces a teacher-to-student training pipeline with Self-Forcing distillation, enabling the model to learn from its generated history while maintaining temporal consistency. FlashDreams provides accelerated streaming inference, and an adapted FadeMem module manages the model’s attention history during long rollouts. Demonstrations show Cosmos3-H-Dreams generating different cholecystectomy scenes from the same initial frame, conditioned on distinct instrument action sequences, illustrating physically accurate tissue deformations and coherent instrument responses.

Cosmos3-H-Dreams is positioned as a research tool, not a validated clinical system, offering a foundation for studying surgical interaction in increasingly realistic environments. The system can be adapted to other datasets and embodiments with synchronized video and compatible action representations, and NVIDIA provides a step-by-step guide for training real-time student models on custom datasets. Controller integrations must align with the training action conventions and normalization, emphasizing research directions rather than established clinical capabilities.