Sim-to-Real: Model Uncertainty and Action Delay · Learning
English translation
Execution conditions
- Required packages: gymnasium, stable-baselines3, and PyTorch.
- SB3 policy:
MlpPolicy. - Gymnasium
step: 5 return values. - Public environment visualization: Add
gymnasium[classic-control].
If the required package is missing, run %pip install control==0.10.2 gymnasium==1.3.0 stable-baselines3==2.9.2 in a Colab or Jupyter code cell.
Topics covered
Even a policy that works well in a simulator can fail when physical parameters and delays differ. Quantify these differences before transferring to real hardware.
Reality gap
Differences in physical parameters, sensor noise, action delay, and normalization can change a simulator policy's behavior on real devices. Vary one factor at a time to identify vulnerabilities.
Identification and randomization
System identification estimates a model from observations; domain randomization trains across a range of possible variations. Design the evaluation range separately from the training range.
Worked example
Use the baseline motor model and P control. Compare reference angular velocities of 0.5, 1, and 1.5.
"""Synthetic motor; not a hardware interface. dt=0.1 s, scalar speed and bounded input."""
import numpy as np
class Motor:
def reset(self, seed=0, reference=1., noise=.01):
self.rng = np.random.default_rng(seed)
self.omega, self.t = 0., 0
self.reference, self.noise = reference, noise
return np.array([self.omega / 5, reference / 2], dtype=np.float32)
def step(self, action):
u = 2 * float(np.clip(np.asarray(action).item(), -1, 1))
self.omega += .1 * (-.4 * self.omega + .8 * u + self.rng.uniform(-self.noise, self.noise))
self.t += 1; error = self.reference - self.omega
reward = -(error**2 + .05 * u**2)
return np.array([self.omega / 5, self.reference / 2], dtype=np.float32), reward, abs(self.omega) > 5, self.t >= 100, {'error': error, 'u': u}
def motor_evaluate(policy, references=(.5, 1., 1.5)):
rows=[]
for reference in references:
env=Motor(); obs=env.reset(100, reference)
errors=[]; inputs=[]; total=0.
while True:
obs,r,terminated,truncated,info=env.step(policy(obs))
errors.append(info['error']);inputs.append(info['u']);total+=r
if terminated or truncated:break
rows.append([reference, np.sqrt(np.mean(np.square(errors))),np.mean(np.square(inputs)),int(terminated),total])
return np.array(rows)
def proportional(obs):return np.clip((2 * obs[1] - 5 * obs[0]), -1, 1)
print('reference / RMSE / mean u² / safety termination / Return')
print(motor_evaluate(proportional))
reference / RMSE / mean u² / safety termination / Return [[ 0.5 0.12626501 0.07338515 0. -1.96121101] [ 1. 0.25292968 0.29432127 0. -7.86894852] [ 1.5 0.3970162 0.6158097 0. -18.84123509]]