Into the Omniverse: How Developers Turn Ideas Into Simulations With Frontier AI Agents
Developers are using frontier AI models with NVIDIA Omniverse to create interactive simulations for robotics, autonomous vehicles, and design validation through natural-language instructions.
Frontier AI models like GPT-6 Astra are being integrated with NVIDIA Omniverse libraries to help developers build interactive simulations for exploring scenarios, investigating failures, and improving designs. These simulations rely on GPU-accelerated physics, rendering, and sensor capabilities provided by Omniverse, enabling developers to direct AI agents using natural-language instructions and review results. Projects include gamified robot control and interactive warehouse simulations, demonstrating the practical applications of combining AI with simulation tools.
Frank DeLise, Omniverse product manager at NVIDIA, used Astra to transform a SimReady warehouse and humanoid robot into an interactive simulator with first- and third-person views. DeLise directed Astra to connect Omniverse libraries for physics, scene updates, rendering, and user interface, while also using SimReady to create the physical scene. Astra then generated animation and application code to integrate these capabilities, showcasing how AI can streamline simulation development.
Doyub Kim, a manager on NVIDIA’s simulation technology team, directed Astra to build Zero to Alpamayo, a reusable simulation environment based on San Francisco’s Market Street. Kim mapped the workflow to connect asset creation, traffic, Omniverse RTX sensor simulation, and Alpamayo driving in stages, checking each integration. The prototype served as a testing ground for comparing models and tracing how scene or sensor changes affect downstream driving behavior, with experiments varying weather and lighting conditions.
Ashley Reid, working on RTX sensor validation at NVIDIA, directed Astra and Claude Fable 5 agents to compare Omniverse RTX camera and raw LiDAR outputs with recorded data. The agents created two digital twins from scratch and improved two existing ones over about three days, guiding iterative workflows to address missing objects, geometry, and materials based on camera and LiDAR metrics.