Sim-to-Real: Model Uncertainty and Action Delay

Even a policy that works well in a simulator can fail when physical parameters and delays differ. Quantify these differences before transferring to real hardware.

• Required packages: gymnasium, stable-baselines3, and PyTorch.
• SB3 policy: MlpPolicy.
• Gymnasium step: 5 return values.
• Public environment visualization: Add gymnasium[classic-control].

If the required package is missing, run %pip install control==0.10.2 gymnasium==1.3.0 stable-baselines3==2.9.2 in a Colab or Jupyter code cell.

Drive-system tracking performance and input cost

A synthetic 1st-order motor model that tracks a constant reference angular velocity in each experiment. Separate training at target 1 from evaluation at 0.5, 1, and 1.5.

Observation·State

Angular velocity $\omega/5$, target $r/2$.

Action

Continuous $a\in[-1,1]$, actual input $u=2a$.

Reward·Constraint

$-((r-\omega)^2+0.05u^2)$. Termination at $|\omega|>5$; time limit at 100 steps.

Comparison baseline

P control with the same input limits.

Metrics to check

Tracking RMSE, mean u², and number of safety terminations. Evaluate separately for each target.

HVAC and thermal process control

A design case for control that reduces both temperature deviation and energy use.

Observation·State

Temperature, target, outside-air conditions, and recent inputs. Include history when delays are substantial.

Action

Bounded continuous heating and cooling output.

Reward·Constraint

Temperature error and energy cost. Record safety limits as metrics separate from reward as well.

Comparison baseline

Thermostat and PI control.

Metrics to check

Error, energy, and time spent violating limits. Evaluate separately by outside-air condition.

Starting conditions

Use the baseline motor model and P control. Compare reference angular velocities of 0.5, 1, and 1.5.

Procedure

1. Record the model's input limits, time interval, and observation units, and run the baseline controller.
2. Compare RMSE, input cost, and return for each target.

Results to retain

• Evaluation table by target and representative tracking curves.

In the synthetic model, action $a\in[-1,1]$ corresponds to model input $u=2a\in[-2,2]$. Physical units for $u$ are unspecified.

Starting conditions

Apply gain .8→.6 or input delay 0→0.2 s, one at a time. Compare P control and briefly trained SAC under the same conditions.

Procedure

1. Identify and mark only the terms that the provided StressMotor changes from the nominal model.
2. Evaluate nominal, reduced-gain, and input-delay conditions, and compare the metrics.
3. Find one condition under which each controller is vulnerable and connect it to quantities to measure on a real device.

Results to retain

• Policy-by-condition table and tracking plot for a vulnerable condition.
• Record location: In the existing while loop in stress_test, immediately after each step.
• Values to preserve: env.t*0.1, env.omega, env.reference. Distinguish by policy, condition, and seed.
• Plot: For the vulnerable condition selected in the table, show the tracking responses of both policies and the target line using the existing seed=100.
• Axes: Time [s]–Angular velocity [rad/s]. Use only existing evaluation records.

Starting conditions

Choose the one of the three preceding conditions closest to your research.

Procedure

1. Choose one priority among physical parameters, sensor noise, and delays to identify.
2. Create a table of nominal values, measurement ranges, training ranges, and separate evaluation conditions.
3. If measured data are available, describe the estimation method; otherwise, state the minimum observations needed.

Results to retain

• Experiment plan table for domain randomization or system identification.

• Isaac Lab · Robot learning tutorial