PPO and public benchmarks

Connect the method for limiting large policy changes in a single update to actual policy performance.

• Required packages: gymnasium, stable-baselines3, and PyTorch.
• SB3 policy: MlpPolicy.
• Gymnasium step: 5 return values.
• Public environment visualization: Add gymnasium[classic-control].

If the required package is missing, run %pip install control==0.10.2 gymnasium==1.3.0 stable-baselines3==2.9.2 in a Colab or Jupyter code cell.

Throughput and cost of equipment and communication resources

An average of 2 items arrive at the synthetic queue each time step. Increasing service capacity reduces queue length but increases energy cost.

Observation·State

Queue length $q=0$…20. Observation $q/20$.

Action

Choose a processing capacity of 0, 2, or 4.

Reward·Constraint

−(next queue length + 0.2×capacity² + 10×dropped items). Time limit at 100 steps.

Comparison baseline

Constant capacity of 2, and a threshold policy based on queue length.

Metrics to check

Evaluation return, mean queue length, mean capacity², and dropped-item count.

Design energy storage operation

A research design case distinguishing the roles of price, load, and stored energy before connecting measured data.

Observation·State

SOC, load, price, and time. Observations must exclude future information.

Action

Charging and discharging power. Decide first between discrete levels and a continuous range.

Reward·Constraint

Electricity and degradation costs. Specify rules for SOC and power limits.

Comparison baseline

Time-of-day rules and, if possible, optimization using the same information.

Metrics to check

Total cost, constraint violations, and maximum grid power draw. Split train/test data by date.

First check whether the discrete or continuous action choice matches the algorithm. In your domain, compare rule-based control using the same information and input limits.

Starting conditions

Discrete actions in Queue

Procedure

1. Pass the two preceding rule-based policies policies to evaluate and record results for the common evaluation seeds. Find where QueueGym transforms queue length to $q/20$ and reward to $1/20$ of its original value.
2. Run fit_sb3("PPO",seed=0,steps=2048). Select the learned policy with model.predict(...,deterministic=True) and evaluate it on the same seeds using evaluate in the notebook.
3. Record episode return, mean queue length, mean squared capacity, and total dropped items for the rule-based and learned policies in a table with the same columns. Identify improvements and regressions, and state that your judgment is based on this training budget and evaluation seed.

Results to retain

• Evaluation table for baseline and learned policies. Plot comparing queue length and cost.

Starting conditions

CartPole-v1 with clip_range=.1 and .2, each for 2048 transitions. Use the same seed and evaluation group.

Procedure

1. Connect clip_range, n_steps, batch_size, and n_epochs in the public implementation to their roles using the code and official explanations.
2. Run two models that differ only in clipping settings with the same training budget.
3. Compare returns and lengths on the common evaluation set. Identify one difference from the CartPole PPO configuration in RL Baselines3 Zoo and state a hypothesis about its effect.

Results to retain

• Evaluation table for both configurations and interpretation of their differences.

Starting conditions

Based on the preceding experiments, choose Queue, Motor, or one of your research problems and design an experiment for real application.

Procedure

1. Define state, observation, action, reward, termination, and time limit in one table. Record each variable's units and the information you can actually obtain.
2. Write 3 expected transitions numerically and check whether rewards for good and bad actions match your intent.
3. Choose one rule-based comparison baseline, one evaluation condition, and one failure case of greatest concern. Also list the required measurements or simulator.

Results to retain

• MDP design table, 3 transition examples, and comparative experiment plan.
• If real research data are unavailable, design using synthetic conditions from an application case.

You may continue with the research problem and design table selected earlier. Revisit existing entries and transition examples using the concepts learned in this activity, and update changed entries or reasons for retaining them.

• SB3 · PPO
• RL Baselines3 Zoo · PPO configuration