Sim-to-Real: Model Uncertainty and Action Delay

Learning objectives

Even a policy that works well in a simulator can fail when physical parameters and delays differ. Quantify these differences before transferring to real hardware.

Execution conditions

• Required packages: gymnasium, stable-baselines3, and PyTorch.
• SB3 policy: MlpPolicy.
• Gymnasium step: 5 return values.
• Public environment visualization: Add gymnasium[classic-control].

If the required package is missing, run %pip install control==0.10.2 gymnasium==1.3.0 stable-baselines3==2.9.2 in a Colab or Jupyter code cell.

Applications

Drive-system tracking performance and input cost

A synthetic 1st-order motor model that tracks a constant reference angular velocity in each experiment. Separate training at target 1 from evaluation at 0.5, 1, and 1.5.

Observation and state: angular velocity $\omega/5$, target $r/2$.

Action: Continuous $a\in[-1,1]$, actual input $u=2a$.

Reward and constraints: $-((r-\omega)^2+0.05u^2)$. Termination at $|\omega|>5$; time limit at 100 steps.

Comparison baseline: P control with the same input limits.

Metrics to check: Tracking RMSE, mean u², and number of safety terminations. Evaluate separately for each target.

HVAC and thermal process control

A design case for control that reduces both temperature deviation and energy use.

Observation and state: temperature, target, ambient conditions, and recent inputs. Include history if delays are large.

Action: Bounded continuous heating/cooling output.

Reward and constraints: Temperature error and energy cost. Record safety limits as separate metrics as well as in the reward.

Comparison baselines: Thermostat and PI control.

Metrics to check: Error, energy, and time spent violating limits. Evaluate separately by outside-air condition.

Baseline control on the nominal model

Starting conditions

Use the baseline motor model and P control. Compare reference angular velocities of 0.5, 1, and 1.5.

Procedure

1. Record the model's input limits, time interval, and observation units, and run the baseline controller.
2. Compare RMSE, input cost, and return for each target.

Results to retain

• Evaluation table by target and representative tracking curves.

In the synthetic model, action $a\in[-1,1]$ corresponds to model input $u=2a\in[-2,2]$. Physical units for $u$ are unspecified.

"""Synthetic motor; not a hardware interface. dt=0.1 s, scalar speed and bounded input."""
import numpy as np
class Motor:
    def reset(self, seed=0, reference=1., noise=.01):
        self.rng = np.random.default_rng(seed)
        self.omega, self.t = 0., 0
        self.reference, self.noise = reference, noise
        return np.array([self.omega / 5, reference / 2], dtype=np.float32)
    def step(self, action):
        u = 2 * float(np.clip(np.asarray(action).item(), -1, 1))
        self.omega += .1 * (-.4 * self.omega + .8 * u + self.rng.uniform(-self.noise, self.noise))
        self.t += 1; error = self.reference - self.omega
        reward = -(error**2 + .05 * u**2)
        return np.array([self.omega / 5, self.reference / 2], dtype=np.float32), reward, abs(self.omega) > 5, self.t >= 100, {'error': error, 'u': u}

def motor_evaluate(policy, references=(.5, 1., 1.5)):
    rows=[]
    for reference in references:
        env=Motor(); obs=env.reset(100, reference)
        errors=[]; inputs=[]; total=0.
        while True:
            obs,r,terminated,truncated,info=env.step(policy(obs))
            errors.append(info['error']);inputs.append(info['u']);total+=r
            if terminated or truncated:break
        rows.append([reference, np.sqrt(np.mean(np.square(errors))),np.mean(np.square(inputs)),int(terminated),total])
    return np.array(rows)

def proportional(obs):return np.clip((2 * obs[1] - 5 * obs[0]), -1, 1)
print('reference / RMSE / mean u² / safety termination / Return')
print(motor_evaluate(proportional))

Record results and self-check

Objective:

RMSE:

Mean u²:

Return:

Points to check

• Did you distinguish and record the mapping between action ranges and model input ranges?
• Did you distinguish steady-state error from transient error?

Where performance breaks down when the model is wrong

Starting conditions

Apply gain .8→.6 or input delay 0→0.2 s, one at a time. Compare P control and briefly trained SAC under the same conditions.

Procedure

1. Identify and mark only the terms that the provided StressMotor changes from the nominal model.
2. Evaluate nominal, reduced-gain, and input-delay conditions, and compare the metrics.
3. Find one condition under which each controller is vulnerable and connect it to quantities to measure on a real device.

Results to retain

• Policy-by-condition table and tracking plot for a vulnerable condition.
• Record location: In the existing while loop in stress_test, immediately after each step.
• Values to preserve: env.t*0.1, env.omega, env.reference. Distinguish by policy, condition, and seed.
• Plot: For the vulnerable condition selected in the table, show the tracking responses of both policies and the target line using the existing seed=100.
• Axes: Time [s]–Angular velocity [rad/s]. Use only existing evaluation records.

# Run the Queue or Motor definition first. Install gymnasium and stable-baselines3 if absent.
import gymnasium as gym
from gymnasium import spaces
from stable_baselines3 import A2C, PPO, SAC
class QueueGym(gym.Env):
    def __init__(self):
        self.model=Queue();self.action_space=spaces.Discrete(3)
        self.observation_space=spaces.Box(0.,1.,(1,),dtype=np.float32)
    def reset(self,seed=None,options=None):
        super().reset(seed=seed)
        return np.array([self.model.reset(seed)/20],dtype=np.float32),{}
    def step(self,a):
        q,r,te,tr,info=self.model.step(a)
        return np.array([q/20],dtype=np.float32),r/20,te,tr,info
class MotorGym(gym.Env):
    def __init__(self):
        self.model=Motor();self.action_space=spaces.Box(-1.,1.,(1,),dtype=np.float32)
        self.observation_space=spaces.Box(-np.inf,np.inf,(2,),dtype=np.float32)
    def reset(self,seed=None,options=None):
        super().reset(seed=seed)
        return self.model.reset(seed),{}
    def step(self,a):return self.model.step(a)

def fit_sb3(algorithm='PPO',seed=0,steps=2048):
    cls={'A2C':A2C,'PPO':PPO,'SAC':SAC}[algorithm]
    env=MotorGym() if algorithm=='SAC' else QueueGym()
    options={'policy_kwargs':dict(net_arch=[32,32]),'seed':seed,'verbose':0,'device':'cpu'}
    if algorithm=='PPO':options.update(n_steps=128,batch_size=64,n_epochs=3)
    if algorithm=='A2C':options.update(n_steps=32)
    if algorithm=='SAC':options.update(learning_starts=100,buffer_size=5000,batch_size=64)
    model=cls('MlpPolicy',env,**options);model.learn(total_timesteps=steps)
    return model

motor_model=fit_sb3('SAC')
# Run the Motor baseline first. These are sensitivity tests, not real-robot results.
from collections import deque
class StressMotor(Motor):
    def __init__(self,damping=.4,gain=.8,delay=0):
        self.damping,self.gain,self.delay=damping,gain,delay
    def reset(self,seed=0,reference=1.,noise=.01):
        obs=super().reset(seed,reference,noise)
        self.pending=deque([0.]*self.delay)
        return obs
    def step(self,action):
        command=2*float(np.clip(np.asarray(action).item(),-1,1))
        self.pending.append(command);u=self.pending.popleft()
        self.omega+=.1*(-self.damping*self.omega+self.gain*u+self.rng.uniform(-self.noise,self.noise))
        self.t+=1;error=self.reference-self.omega
        return np.array([self.omega/5,self.reference/2],dtype=np.float32),-(error**2+.05*u**2),abs(self.omega)>5,self.t>=100,{'error':error,'u':u}
def stress_test(policy):
    rows=[]
    for name,params in [('nominal',{}),('gain loss',dict(gain=.6)),('delay 0.2s',dict(delay=2))]:
        for seed in range(100,103):
            env=StressMotor(**params);obs=env.reset(seed);errors=[];energy=[];total=0.
            while True:
                obs,r,te,tr,info=env.step(policy(obs));errors.append(info['error']);energy.append(info['u']**2);total+=r
                if te or tr:break
            rows.append([name,seed,total,float(np.sqrt(np.mean(np.square(errors)))),float(np.mean(energy)),te])
    return rows
for row in stress_test(proportional):print(row)

for row in stress_test(lambda o:motor_model.predict(o,deterministic=True)[0]):print('SAC',row)

Record results and self-check

Policy:

Condition:

seed:

Return:

RMSE:

Input cost:

Points to check

• Did you avoid changing multiple physical parameters simultaneously?
• Did you avoid claiming that robustness in a synthetic model establishes real-device performance?

Define the reality gap in your research

Starting conditions

Choose the one of the three preceding conditions closest to your research.

Procedure

1. Choose one priority among physical parameters, sensor noise, and delays to identify.
2. Create a table of nominal values, measurement ranges, training ranges, and separate evaluation conditions.
3. If measured data are available, describe the estimation method; otherwise, state the minimum observations needed.

Results to retain

• Experiment plan table for domain randomization or system identification.

Record results and self-check

Uncertainty:

Measurement method:

Training range:

Evaluation scope:

Points to check

• Did you avoid unconditionally using the same ranges for training and evaluation?
• Did you define safety limits and metrics to check before applying to real equipment?

Official examples and data

• Isaac Lab · Robot learning tutorial