← Blog

November 2026

Sim to real is a data problem, not a physics problem

There is a standard explanation for why policies trained in simulation fail on real robots. The explanation goes like this: the physics in the simulator is not accurate enough. Friction coefficients are wrong. Actuator delays are not modeled. Contact dynamics are simplified. The real world is messier than the simulation, so the policy breaks when it meets the real world.

This explanation is true in a narrow sense and misleading in a broader one. Yes, physics gaps exist. But in my experience the physics gap is rarely what kills a policy. What kills a policy is usually something much simpler: the training data did not cover the situation the robot encountered, or the policy was never shown what failure looks like, so it has no idea what to do when things start going wrong.

Think about how most robot learning pipelines are set up. You collect demonstrations of successful task completions. You train a policy on those demonstrations. You evaluate the policy in the simulator and it looks fine. Then you deploy it on the real robot and it fails in the first thirty seconds. Everyone assumes the simulator is to blame.

But ask yourself: did the training data include any near failures? Did it include the moment where the grasp was slightly off and the robot had to recover? Did it show the policy what a deteriorating situation looks like before it becomes a full failure? In most pipelines the answer is no. The demonstrations are clean successes collected by an expert who corrected every mistake before it became visible in the data. The policy learned to succeed in ideal conditions. The real world does not offer ideal conditions.

This is a data problem. Not a physics problem.

The fix is not a better simulator. The fix is more intentional data collection. You need demonstrations that include recovery behaviors. You need data that was collected in variable conditions, not just the pristine setup that happened to work. And critically, you need to label your failures. When a policy fails, you need to know which step broke, whether it was perception, planning, or execution, and feed that signal back into training.

Right now almost nobody does this systematically because the tooling makes it too hard. Correlating a real world failure with the specific training episode that might have caused it requires pulling data from multiple systems and doing the analysis manually. So most teams skip it and blame the simulator instead.

Better physics simulation is worth pursuing. I am not saying otherwise. But if you are spending your time tuning friction coefficients while your failure data sits unlabeled in a log file, you are optimizing the wrong thing. Fix the data first. The physics gap is smaller than you think.