MirroS Unveils 'Code-as-World': Real Videos to Executable MuJoCo Physics

MirroS has unveiled 'Code-as-World,' a groundbreaking paradigm capable of transforming real-world videos into editable, executable MuJoCo physics programs, marking a significant leap beyond traditional pixel-based world representations. This novel system, whose technical report was released on August 27, 2026, represents physical scenes not as raw visual data, but as structured, executable code, specifically `scene.json` files compatible with the advanced MuJoCo physics engine. The core innovation lies in an "agentic discovery loop" that leverages abductive reasoning, allowing an AI agent to iteratively propose, execute, render, verify, and refine these executable world representations (EWRs) from real footage, typically within five rounds. This process recovers explicit physical composition, dynamic evolution, and visual appearance as code, which can then be verified against the original video and subsequently edited or re-simulated.

This development matters immensely because it fundamentally shifts how AI perceives and interacts with the physical world. Existing video-based world models, while capable of predicting plausible future frames, often fail to represent the underlying physics of mass, contact, or gravity, thereby limiting their ability to understand causality and maintain persistent consequences in open-ended simulations. Code-as-World, by contrast, provides a "quantitatively grounded and controllable abstraction" of reality, generating explicit states, dynamics, and mechanisms that capture the underlying world that produced the observed video, not just its surface appearance. This translates directly into "scalable physical supervision," providing high-fidelity training data with exact physical labels that are absent in raw video, which is crucial for training advanced vision-language models. For instance, the Code-as-World-VL-9B model achieved a 55.4 MRA score on QuantiPhy-validation, outperforming Gemini-3.1 Flash at 54.8 MRA and surpassing the strongest open-weight baseline by approximately 15 points, with an even more capable Code-as-World-VL-27B variant demonstrating further improvements. The ability to edit these recovered worlds—changing physical quantities, initial conditions, or camera perspectives for re-simulation—offers unprecedented control and debuggability for researchers and developers.

MuJoCo (Multi-Joint dynamics with Contact), the chosen physics engine, is central to this breakthrough. Acquired by Google DeepMind in October 2021 and open-sourced in May 2022, MuJoCo is a high-performance, general-purpose simulator optimized for robotics, biomechanics, and machine learning research. Its design prioritizes model-based optimization through contacts, offering exceptional speed and accuracy for continuous control environments and complex multi-body systems, making it an ideal foundation for physically realistic simulations. The engine's use of human-readable XML (MJCF) for scene description further enhances the editability of the generated code. This focus on an executable, physically accurate representation contrasts with prior generations of video-to-simulation approaches that often relied on less robust inverse graphics or 3D reconstruction methods, which, while providing strong constraints, did not yield inherently executable and editable world models. Another recent, related development, the "Code World Model" framework, also explores separating world evolution from visual realization using a coding agent and a video model. However, Code-as-World specifically addresses the inverse problem: inferring the underlying physical code from *real-world video*, rather than primarily generating visual observations from pre-defined code.

Looking ahead, Code-as-World promises to accelerate the development of truly intelligent agents and robotic systems. By providing a direct bridge between visual observation and a physically grounded, editable simulation, it significantly enhances the "sim-to-real" transfer problem, a persistent challenge in robotics. Robots can now be trained in simulations derived directly from real-world scenarios, leading to more robust and adaptable behaviors in actual deployments. Agentic AI is already being explored to orchestrate robot control and navigation, and Code-as-World provides a richer, more accurate environment for these agents to learn and refine their physical reasoning. The open-source release of the Code-as-World GitHub repository and its checkpoints (Code-as-World-VL-4B and Code-as-World-VL-9B), fine-tuned from Qwen3.5 models and available under Apache 2.0, ensures broad accessibility for researchers. While challenges remain in scaling this system to extremely complex, dynamic scenes and ensuring the robustness of the agentic loop across highly diverse real-world footage, Code-as-World represents a pivotal step towards general-purpose physical intelligence, enabling AI systems to not just see the world, but truly understand and interact with its underlying mechanics.