In this interview, DUSt3R researcher Jérome Revaud shares his thoughts on Real2Sim (Reality-to-Simulation) and the future of robotics. Real2Sim refers to the process of transforming the real world into a digital simulation environment where robots can learn and train.
For years, robotics research has focused primarily on Sim2Real (Simulation-to-Reality): transferring intelligence learned in simulation into the physical world. More recently, however, attention has shifted toward the opposite direction—how quickly and accurately we can bring reality into simulation. No matter how much training happens in simulation, it cannot be effectively transferred to real robots if the simulated environment fails to faithfully represent the real world.
The challenge is that building such simulations has traditionally been both complex and time-consuming. DUSt3R has emerged as a promising breakthrough by reconstructing 3D environments directly from just a few images, without requiring camera calibration or complex preprocessing. We spoke with Jérome Revaud about the possibilities this unlocks and his vision for the future.
The following article summarizes key insights from our interview.
Why Is Reality-to-Simulation So Difficult?
One of the most powerful approaches for training robots is reinforcement learning. The challenge is that reinforcement learning often requires millions of trials and errors. In the real world, neither time nor robots are unlimited, making such large-scale experimentation impractical.
This is where the idea of Real2Sim comes from. Instead of training entirely in the real world, researchers train robots in simulation. However, for policies learned in simulation to work reliably in reality, the simulated environment must closely resemble the real world. Building such a high-fidelity simulation is exactly what Real2Sim aims to achieve.
The problem is that this process is far more difficult than it sounds. It requires reconstructing the geometry of the environment, reproducing realistic appearance and textures, modeling accurate physical parameters and physics engines, and even representing sensors and actuators. For a long time, creating such simulations demanded significant effort, time, and expertise.
How DUSt3R Reduces Real2Sim Bottlenecks
DUSt3R opens up new possibilities across three major challenges in the Real2Sim pipeline.
① Instantly Reconstructing the 3D Geometry of the World
The first challenge is recreating the geometry of the world—and this is precisely what DUSt3R was designed for.
With only a handful of images or a short video sequence, DUSt3R can reconstruct highly accurate 3D representations of real environments. More importantly, 3D reconstruction can happen almost simultaneously with data collection. This dramatically reduces one of the largest bottlenecks in traditional Real2Sim pipelines, which often require complex processing and significant time before a usable 3D model is available.
.gif)
One area where DUSt3R particularly excels is reconstructing textureless objects. A white table, for example, is notoriously difficult for traditional 3D reconstruction methods because it lacks visual features. Having learned from vast amounts of data, DUSt3R has seen countless examples of tables and understands their underlying geometry, allowing it to generate clean and simulation-ready 3D structures.
② Understanding Materials and Appearance Beyond Geometry
The second challenge is photorealism.
Robots perceive the world through cameras, which means simulated environments must also look realistic. The appearance of the virtual world needs to be close enough to reality for robots to learn effectively.
Recent advances such as NeRF and 3D Gaussian Splatting have made impressive progress in realistic rendering. However, these approaches are often better at reproducing appearance than recovering the accurate 3D geometry required for robotic interaction. In other words, there is still no single approach that can simultaneously capture both geometry and appearance with high accuracy.
Just as DUSt3R learns geometry from large-scale data, it could eventually be extended to learn and predict material properties and reflectance. A model that has seen millions of tables, for example, may develop prior knowledge about the materials and surface characteristics that tables typically have. This could help create simulation environments that are visually much closer to the real world.
③ Digitizing the Physical Properties of the Real World
The third challenge is physical properties.
To build a realistic simulation, it is not enough to reconstruct geometry and appearance. The simulation must also behave like the real world. This requires modeling physical properties such as mass distribution, inertia, friction coefficients, and damping.
One way to think about this is to view the 3D points generated by DUSt3R as tiny atoms that compose the world. Today, these points mainly contain information such as position or object category. In the future, however, they could also carry richer attributes, including properties like friction coefficients.
As each point becomes associated with more information, it may become possible to build a more complete digital world—one that contains not only geometry and appearance, but also the physical properties required for simulation.
Completing the Real2Sim2Real Loop
Today, many robotics researchers are exploring how foundation models like DUSt3R can be integrated into Real2Sim pipelines.
We are still far from a future where a single button instantly generates a perfect simulation. Nevertheless, by dramatically simplifying the process of converting reality into simulation, DUSt3R is bringing the vision of a complete Real2Sim2Real loop closer to reality.
If such a loop can eventually be closed, robots could be trained at massive scale with minimal cost. Highly capable robots could be deployed from the start without extensive fine-tuning, and tasks learned in simulation could transfer directly into the real world.
The moment the bottleneck in Real2Sim disappears, the boundary between reality and simulation may begin to blur. Robotic learning could accelerate at a pace unlike anything seen before, fundamentally changing how robots acquire skills and adapt to the world.
It is a future some describe as a “Singularity for Robotics.” And perhaps, we are already taking the first steps toward it.