Public environments and experiment records

One line of termination handling can change the training target. Start by checking environment boundaries and fair policy evaluation.

Run the baseline Queue example with NumPy. Compare policies on seed=100…104, which was not used for training.

Public environment activities use gymnasium and stable-baselines3. gymnasium[classic-control] is required only for visualization.

If the required package is missing, run %pip install control==0.10.2 gymnasium==1.3.0 stable-baselines3==2.9.2 in a Colab or Jupyter code cell.

Throughput and cost of equipment and communication resources

An average of 2 items arrive at the synthetic queue each time step. Increasing service capacity reduces queue length but increases energy cost.

Observation·State

Queue length $q=0$…20. Observation $q/20$.

Action

Choose a processing capacity of 0, 2, or 4.

Reward·Constraint

−(next queue length + 0.2×capacity² + 10×dropped items). Time limit at 100 steps.

Comparison baseline

Constant capacity of 2, and a threshold policy based on queue length.

Metrics to check

Evaluation return, mean queue length, mean capacity², and dropped-item count.

Choose equipment inspection times

Frame the research problem as balancing the cost of inspecting every time against the cost of missed failures.

Observation·State

Recent sensor readings, trends, and time since the last inspection. Distinguish the actual fault state from observations.

Action

Continue operating, inspect, or replace.

Reward·Constraint

Operating profit minus inspection, replacement, and failure costs.

Comparison baseline

Periodic inspection and threshold alarms.

Metrics to check

Availability, failure count, and total cost. Check for failure label leakage.

Starting conditions

Run the reset → step → termination flow using the Queue example. Use seed=7, action 1 (capacity 2), and 100 steps.

Procedure

1. Record (q,a,r,q_next,terminated,truncated) for the first 5 transitions.
2. Set the time limit to 5 steps and inspect the last transition.
3. Distinguish terminal-state termination from a data collection time limit, and write the mask used for the bootstrap target.

Results to retain

• A transition table and an explanation of the last step's mask.

Starting conditions

Use Gymnasium CartPole-v1. Compare a random policy and a simple angle/angular velocity rule at seed=100,101,102.

Procedure

1. Print observation_space, action_space, and the episode limit, and check the official documentation for the meanings of the 4 observation components and the two actions.
2. Run both policies with the same seed and record returns, lengths, and termination types.
3. Save transition records as NPZ and load them again. Locate the first transition and episode boundaries, and check the correspondence between observation and action counts.

Results to retain

• An evaluation table with 3 rows per policy and 1 transition example.
• 1 trace file containing environment ID, policy, seed, and library versions.

Starting conditions

Based on the preceding experiments, choose Queue, Motor, or one of your research problems and design an experiment for real application.

Procedure

1. Define state, observation, action, reward, termination, and time limit in one table. Record each variable's units and the information you can actually obtain.
2. Write 3 expected transitions numerically and check whether rewards for good and bad actions match your intent.
3. Choose one rule-based comparison baseline, one evaluation condition, and one failure case of greatest concern. Also list the required measurements or simulator.

Results to retain

• MDP design table, 3 transition examples, and comparative experiment plan.
• If real research data are unavailable, design using synthetic conditions from an application case.

You may continue with the research problem and design table selected earlier. Revisit existing entries and transition examples using the concepts learned in this activity, and update changed entries or reasons for retaining them.

• Gymnasium · Basic usage
• CartPole · environment definition