SAC and continuous control

Consider both exploration and input cost in a continuous-input control problem, and compare with a simple controller.

• Required packages: gymnasium, stable-baselines3, and PyTorch.
• SB3 policy: MlpPolicy.
• Gymnasium step: 5 return values.
• Public environment visualization: Add gymnasium[classic-control].

If the required package is missing, run %pip install control==0.10.2 gymnasium==1.3.0 stable-baselines3==2.9.2 in a Colab or Jupyter code cell.

Drive-system tracking performance and input cost

A synthetic 1st-order motor model that tracks a constant reference angular velocity in each experiment. Separate training at target 1 from evaluation at 0.5, 1, and 1.5.

Observation·State

Angular velocity $\omega/5$, target $r/2$.

Action

Continuous $a\in[-1,1]$, actual input $u=2a$.

Reward·Constraint

$-((r-\omega)^2+0.05u^2)$. Termination at $|\omega|>5$; time limit at 100 steps.

Comparison baseline

P control with the same input limits.

Metrics to check

Tracking RMSE, mean u², and number of safety terminations. Evaluate separately for each target.

HVAC and thermal process control

A design case for control that reduces both temperature deviation and energy use.

Observation·State

Temperature, target, outside-air conditions, and recent inputs. Include history when delays are substantial.

Action

Bounded continuous heating and cooling output.

Reward·Constraint

Temperature error and energy cost. Record safety limits as metrics separate from reward as well.

Comparison baseline

Thermostat and PI control.

Metrics to check

Error, energy, and time spent violating limits. Evaluate separately by outside-air condition.

First check whether the discrete or continuous action choice matches the algorithm. In your domain, compare rule-based control using the same information and input limits.

Starting conditions

Synthetic motor, dt=0.1 s, input $u\in[-2,2]$, and training reference angular velocity 1. Evaluate targets 0.5, 1, and 1.5.

Procedure

1. Record baseline policy performance first. Check the provided wrapper's observation and action ranges and reset/step return values.
2. Run short training for 2048 transitions and evaluate deterministically under the same conditions.
3. Separate metrics that improved over the baseline from those that worsened, and interpret the limitations of short training.

Results to retain

• Evaluation table for baseline and learned policies. Tracking plots by reference angular velocity.
• Record location: In the existing evaluation loop in motor_evaluate, immediately after each step.
• Values to preserve: env.t*0.1, env.omega, env.reference.
• Plot: Tracking responses and target lines for each reference in the existing seed=100 evaluation.
• Axes: Time [s]–Angular velocity [rad/s]. Use only existing evaluation records.

In the synthetic model, action $a\in[-1,1]$ corresponds to model input $u=2a\in[-2,2]$. Physical units for $u$ are unspecified.

Starting conditions

Pendulum-v1 with SAC for 2048 transitions. Unlike the motor's angular velocity observation, observations are cosθ, sinθ, and angular velocity.

Procedure

1. Find observations, torque range, rewards, and time limits in the official environment description, and map them to Motor.
2. Compare zero torque, random actions, and briefly trained SAC on the same seed=100…102.
3. Interpret the return and behavior, and state which observations, rewards, and input limits must change when transferring to a real research environment.

Results to retain

• Environment mapping table and policy evaluation table. Optionally visualize one episode.

Starting conditions

Based on the preceding experiments, choose Queue, Motor, or one of your research problems and design an experiment for real application.

Procedure

1. Define state, observation, action, reward, termination, and time limit in one table. Record each variable's units and the information you can actually obtain.
2. Write 3 expected transitions numerically and check whether rewards for good and bad actions match your intent.
3. Choose one rule-based comparison baseline, one evaluation condition, and one failure case of greatest concern. Also list the required measurements or simulator.

Results to retain

• MDP design table, 3 transition examples, and comparative experiment plan.
• If real research data are unavailable, design using synthetic conditions from an application case.

You may continue with the research problem and design table selected earlier. Revisit existing entries and transition examples using the concepts learned in this activity, and update changed entries or reasons for retaining them.

• Pendulum · Environment definition
• SB3 · SAC