Policy Gradient and Actor–Critic

Examine direct updates to a stochastic policy, then investigate how a critic changes learning over long episodes.

• Required packages: gymnasium, stable-baselines3, and PyTorch.
• SB3 policy: MlpPolicy.
• Gymnasium step: 5 return values.

If the required package is missing, run %pip install gymnasium==1.3.0 stable-baselines3==2.9.2 in a Colab or Jupyter code cell.

Throughput and cost of equipment and communication resources

An average of 2 items arrive at the synthetic queue each time step. Increasing service capacity reduces queue length but increases energy cost.

Observation·State

Queue length $q=0$…20. Observation $q/20$.

Action

Choose a processing capacity of 0, 2, or 4.

Reward·Constraint

−(next queue length + 0.2×capacity² + 10×dropped items). Time limit at 100 steps.

Comparison baseline

Constant capacity of 2, and a threshold policy based on queue length.

Metrics to check

Evaluation return, mean queue length, mean capacity², and dropped-item count.

Mobile robot paths and safety

An application design case choosing either grid paths or continuous velocity control.

Observation·State

Position, target-relative position, and nearby obstacle observations.

Action

Grid direction selection or bounded linear and angular velocities.

Reward·Constraint

Separate arrival, collision, and movement costs. Distinguish collisions from timeouts.

Comparison baseline

Shortest path and rule-based avoidance.

Metrics to check

Success rate, collision rate, travel distance, and time. Evaluate on unseen maps.

Starting conditions

Queue, softmax policy 1→16→3, $\gamma=.95$, Adam .005, and 30 episodes. Use a simple baseline that subtracts the mean return for each episode.

Procedure

1. Calculate return-to-go by hand for the fixed rewards [-1,-2,-3], then compare with the loop output.
2. Run the provided REINFORCE implementation and record training returns separately from deterministic evaluation returns.
3. Explain the difference between stochastic actions during collection and argmax selection during evaluation.

Results to retain

• 3-step calculation table, learning curves, and an evaluation table comparing with the rule-based policy.

Starting conditions

Discrete actions in Queue

Procedure

1. Pass the two preceding rule-based policies policies to evaluate and record results for the common evaluation seeds. Find where QueueGym transforms queue length to $q/20$ and reward to $1/20$ of its original value.
2. Run fit_sb3("A2C",seed=0,steps=2048). Select the learned policy with model.predict(...,deterministic=True) and evaluate it on the same seeds using evaluate in the notebook.
3. Record episode return, mean queue length, mean squared capacity, and total dropped items for the rule-based and learned policies in a table with the same columns. Identify improvements and regressions, and state that your judgment is based on this training budget and evaluation seed.

Results to retain

• Evaluation table for baseline and learned policies. Plot comparing queue length and cost.

Starting conditions

Based on the preceding experiments, choose Queue, Motor, or one of your research problems and design an experiment for real application.

Procedure

1. Define state, observation, action, reward, termination, and time limit in one table. Record each variable's units and the information you can actually obtain.
2. Write 3 expected transitions numerically and check whether rewards for good and bad actions match your intent.
3. Choose one rule-based comparison baseline, one evaluation condition, and one failure case of greatest concern. Also list the required measurements or simulator.

Results to retain

• MDP design table, 3 transition examples, and comparative experiment plan.
• If real research data are unavailable, design using synthetic conditions from an application case.

You may continue with the research problem and design table selected earlier. Revisit existing entries and transition examples using the concepts learned in this activity, and update changed entries or reasons for retaining them.

• SB3 · A2C