SAC and continuous control

Learning objectives

Consider both exploration and input cost in a continuous-input control problem, and compare with a simple controller.

Execution conditions

• Required packages: gymnasium, stable-baselines3, and PyTorch.
• SB3 policy: MlpPolicy.
• Gymnasium step: 5 return values.
• Public environment visualization: Add gymnasium[classic-control].

If the required package is missing, run %pip install control==0.10.2 gymnasium==1.3.0 stable-baselines3==2.9.2 in a Colab or Jupyter code cell.

Applications

Drive-system tracking performance and input cost

A synthetic 1st-order motor model that tracks a constant reference angular velocity in each experiment. Separate training at target 1 from evaluation at 0.5, 1, and 1.5.

Observation and state: angular velocity $\omega/5$, target $r/2$.

Action: Continuous $a\in[-1,1]$, actual input $u=2a$.

Reward and constraints: $-((r-\omega)^2+0.05u^2)$. Termination at $|\omega|>5$; time limit at 100 steps.

Comparison baseline: P control with the same input limits.

Metrics to check: Tracking RMSE, mean u², and number of safety terminations. Evaluate separately for each target.

HVAC and thermal process control

A design case for control that reduces both temperature deviation and energy use.

Observation and state: temperature, target, ambient conditions, and recent inputs. Include history if delays are large.

Action: Bounded continuous heating/cooling output.

Reward and constraints: Temperature error and energy cost. Record safety limits as separate metrics as well as in the reward.

Comparison baselines: Thermostat and PI control.

Metrics to check: Error, energy, and time spent violating limits. Evaluate separately by outside-air condition.

First check whether the discrete or continuous action choice matches the algorithm. In your domain, compare rule-based control using the same information and input limits.

Rule-based control and SAC

Starting conditions

Synthetic motor, dt=0.1 s, input $u\in[-2,2]$, and training reference angular velocity 1. Evaluate targets 0.5, 1, and 1.5.

Procedure

1. Record baseline policy performance first. Check the provided wrapper's observation and action ranges and reset/step return values.
2. Run short training for 2048 transitions and evaluate deterministically under the same conditions.
3. Separate metrics that improved over the baseline from those that worsened, and interpret the limitations of short training.

Results to retain

• Evaluation table for baseline and learned policies. Tracking plots by reference angular velocity.
• Record location: In the existing evaluation loop in motor_evaluate, immediately after each step.
• Values to preserve: env.t*0.1, env.omega, env.reference.
• Plot: Tracking responses and target lines for each reference in the existing seed=100 evaluation.
• Axes: Time [s]–Angular velocity [rad/s]. Use only existing evaluation records.

In the synthetic model, action $a\in[-1,1]$ corresponds to model input $u=2a\in[-2,2]$. Physical units for $u$ are unspecified.

"""Synthetic motor; not a hardware interface. dt=0.1 s, scalar speed and bounded input."""
import numpy as np
class Motor:
    def reset(self, seed=0, reference=1., noise=.01):
        self.rng = np.random.default_rng(seed)
        self.omega, self.t = 0., 0
        self.reference, self.noise = reference, noise
        return np.array([self.omega / 5, reference / 2], dtype=np.float32)
    def step(self, action):
        u = 2 * float(np.clip(np.asarray(action).item(), -1, 1))
        self.omega += .1 * (-.4 * self.omega + .8 * u + self.rng.uniform(-self.noise, self.noise))
        self.t += 1; error = self.reference - self.omega
        reward = -(error**2 + .05 * u**2)
        return np.array([self.omega / 5, self.reference / 2], dtype=np.float32), reward, abs(self.omega) > 5, self.t >= 100, {'error': error, 'u': u}

def motor_evaluate(policy, references=(.5, 1., 1.5)):
    rows=[]
    for reference in references:
        env=Motor(); obs=env.reset(100, reference)
        errors=[]; inputs=[]; total=0.
        while True:
            obs,r,terminated,truncated,info=env.step(policy(obs))
            errors.append(info['error']);inputs.append(info['u']);total+=r
            if terminated or truncated:break
        rows.append([reference, np.sqrt(np.mean(np.square(errors))),np.mean(np.square(inputs)),int(terminated),total])
    return np.array(rows)

def proportional(obs):return np.clip((2 * obs[1] - 5 * obs[0]), -1, 1)
print('reference / RMSE / mean u² / safety termination / Return')
print(motor_evaluate(proportional))

# Run the Queue or Motor definition first. Install gymnasium and stable-baselines3 if absent.
import gymnasium as gym
from gymnasium import spaces
from stable_baselines3 import A2C, PPO, SAC
class QueueGym(gym.Env):
    def __init__(self):
        self.model=Queue();self.action_space=spaces.Discrete(3)
        self.observation_space=spaces.Box(0.,1.,(1,),dtype=np.float32)
    def reset(self,seed=None,options=None):
        super().reset(seed=seed)
        return np.array([self.model.reset(seed)/20],dtype=np.float32),{}
    def step(self,a):
        q,r,te,tr,info=self.model.step(a)
        return np.array([q/20],dtype=np.float32),r/20,te,tr,info
class MotorGym(gym.Env):
    def __init__(self):
        self.model=Motor();self.action_space=spaces.Box(-1.,1.,(1,),dtype=np.float32)
        self.observation_space=spaces.Box(-np.inf,np.inf,(2,),dtype=np.float32)
    def reset(self,seed=None,options=None):
        super().reset(seed=seed)
        return self.model.reset(seed),{}
    def step(self,a):return self.model.step(a)

def fit_sb3(algorithm='PPO',seed=0,steps=2048):
    cls={'A2C':A2C,'PPO':PPO,'SAC':SAC}[algorithm]
    env=MotorGym() if algorithm=='SAC' else QueueGym()
    options={'policy_kwargs':dict(net_arch=[32,32]),'seed':seed,'verbose':0,'device':'cpu'}
    if algorithm=='PPO':options.update(n_steps=128,batch_size=64,n_epochs=3)
    if algorithm=='A2C':options.update(n_steps=32)
    if algorithm=='SAC':options.update(learning_starts=100,buffer_size=5000,batch_size=64)
    model=cls('MlpPolicy',env,**options);model.learn(total_timesteps=steps)
    return model

model=fit_sb3('SAC')
print(motor_evaluate(lambda obs:model.predict(obs,deterministic=True)[0]))

Record results and self-check

Policy:

Target velocity:

Evaluation return:

Tracking RMSE:

Mean u²:

Safety terminations:

Points to check

• Did you distinguish training return from evaluation metrics in their original units?
• Did you first validate with a synthetic model instead of real equipment?

Transfer SAC to a public continuous control environment

Starting conditions

Pendulum-v1 with SAC for 2048 transitions. Unlike the motor's angular velocity observation, observations are cosθ, sinθ, and angular velocity.

Procedure

1. Find observations, torque range, rewards, and time limits in the official environment description, and map them to Motor.
2. Compare zero torque, random actions, and briefly trained SAC on the same seed=100…102.
3. Interpret the return and behavior, and state which observations, rewards, and input limits must change when transferring to a real research environment.

Results to retain

• Environment mapping table and policy evaluation table. Optionally visualize one episode.

"""Small public-environment benchmark; evaluation traces are ordinary numeric arrays."""
import json
from importlib.metadata import version
from pathlib import Path
import numpy as np
import gymnasium as gym

def collect(env_id='CartPole-v1', policy=None, seeds=range(100,103)):
    env=gym.make(env_id); transitions=[]; summary=[]
    for seed in seeds:
        obs,info=env.reset(seed=seed);env.action_space.seed(seed);total=0.;length=0
        while True:
            action=env.action_space.sample() if policy is None else policy(obs)
            nxt,reward,terminated,truncated,info=env.step(action)
            transitions.append((obs.copy(),np.asarray(action).copy(),reward,nxt.copy(),terminated,truncated,seed))
            obs=nxt;total+=reward;length+=1
            if terminated or truncated:break
        summary.append([seed,total,length,int(terminated),int(truncated)])
    env.close()
    columns=list(zip(*transitions))
    data={k:np.asarray(v) for k,v in zip(['observations','actions','rewards','next_observations','terminated','truncated','episode_seed'],columns)}
    return np.asarray(summary),data

def cartpole_rule(obs):return int(obs[2]+.25*obs[3]>0)
def zero_torque(obs):return np.zeros(1,dtype=np.float32)

def save_trace(path, env_id, data, policy_name, seeds):
    # Contains numeric arrays and JSON text only; never requires pickle.
    metadata=dict(env_id=env_id,policy=policy_name,seeds=list(seeds),gymnasium=version('gymnasium'))
    np.savez_compressed(path,**data,metadata_json=np.array(json.dumps(metadata)))
    with np.load(path,allow_pickle=False) as saved:
        assert len(saved['observations'])==len(saved['rewards'])
        print(json.loads(str(saved['metadata_json'])))

def public_model(algorithm='DQN',env_id='CartPole-v1',seed=7,steps=2048,**changes):
    from stable_baselines3 import DQN,PPO,SAC
    options=dict(policy_kwargs=dict(net_arch=[32,32]),seed=seed,verbose=0,device='cpu')
    if algorithm=='DQN':options.update(buffer_size=5000,learning_starts=100,target_update_interval=100,batch_size=32)
    if algorithm=='PPO':options.update(n_steps=128,batch_size=64,n_epochs=3)
    if algorithm=='SAC':options.update(buffer_size=5000,learning_starts=100,batch_size=64)
    options.update(changes)
    env=gym.make(env_id)
    model={'DQN':DQN,'PPO':PPO,'SAC':SAC}[algorithm]('MlpPolicy',env,**options)
    model.learn(total_timesteps=steps)
    env.close()
    return model

print('Gymnasium',version('gymnasium'))
env=gym.make('CartPole-v1')
print('Observation:',env.observation_space,'Action:',env.action_space,'Limit:',env.spec.max_episode_steps)
env.close()
# For visual inspection, create a separate env with render_mode='rgb_array'.
# Rendering requires gymnasium[classic-control]; do not render every training step.

model=public_model('SAC','Pendulum-v1')
for name,policy in [('zero',zero_torque),('random',None),('sac',lambda o:model.predict(o,deterministic=True)[0])]:
    rows,data=collect('Pendulum-v1',policy=policy);print(name,rows)
# TODO: Complete the table mapping Pendulum observations and rewards to the Motor model.

Record results and self-check

Policy:

Environment:

Training budget:

Return mean and standard deviation:

Termination type:

Points to check

• Did you avoid directly comparing Pendulum scores with motor scores?
• Did you check actual results instead of assuming short training achieves swing-up?

Apply this to your research problem

Starting conditions

Based on the preceding experiments, choose Queue, Motor, or one of your research problems and design an experiment for real application.

Procedure

1. Define state, observation, action, reward, termination, and time limit in one table. Record each variable's units and the information you can actually obtain.
2. Write 3 expected transitions numerically and check whether rewards for good and bad actions match your intent.
3. Choose one rule-based comparison baseline, one evaluation condition, and one failure case of greatest concern. Also list the required measurements or simulator.

Results to retain

• MDP design table, 3 transition examples, and comparative experiment plan.
• If real research data are unavailable, design using synthetic conditions from an application case.

You may continue with the research problem and design table selected earlier. Revisit existing entries and transition examples using the concepts learned in this activity, and update changed entries or reasons for retaining them.

Record results and self-check

Item:

Concrete definition and units:

Observability and verification method:

Points to check

• Did you exclude future information and evaluation labels from observations?
• Did you define domain-specific metrics rather than judging the research contribution solely by training return?

Official examples and data

• Pendulum · Environment definition
• SB3 · SAC