Introduction to Reinforcement Learning

Check whether a high-reward policy actually produces the desired operation by examining cost and queue length separately.

Run the baseline Queue example with NumPy. Compare policies on seed=100…104, which were not used for training.

Throughput and cost of equipment and communication resources

An average of 2 items arrive at the synthetic queue each time step. Increasing service capacity reduces queue length but increases energy cost.

Observation·State

Queue length $q=0$…20. Observation $q/20$.

Action

Choose a processing capacity of 0, 2, or 4.

Reward·Constraint

−(next queue length + 0.2×capacity² + 10×dropped items). Time limit at 100 steps.

Comparison baseline

Constant capacity of 2, and a threshold policy based on queue length.

Metrics to check

Evaluation return, mean queue length, mean capacity², and dropped-item count.

Design energy storage operation

A research design case distinguishing the roles of price, load, and stored energy before connecting measured data.

Observation·State

SOC, load, price, and time. Observations must exclude future information.

Action

Charging and discharging power. Decide first between discrete levels and a continuous range.

Reward·Constraint

Electricity and degradation costs. Specify rules for SOC and power limits.

Comparison baseline

Time-of-day rules and, if possible, optimization using the same information.

Metrics to check

Total cost, constraint violations, and maximum grid power draw. Split train/test data by date.

Starting conditions

Use the Queue policies constant and threshold. The reward combines queue length, squared capacity cost, and the number of dropped items; the baseline cost coefficient is $0.2$.

Procedure

1. Run evaluate with the same evaluation seed. Its four returned columns are episode return, mean queue length per step, mean squared capacity per step, and total dropped items in the episode. Record the means for each policy.
2. Apply each cost coefficient $c\in\{0.05,0.2,1.0\}$ to the same evaluation records. Multiply the mean queue length and mean squared capacity by the episode length of 100, apply coefficient 10 to total dropped items as in the code, and negate the sum of the three costs.
3. Tabulate returns by cost coefficient and policy, and plot each policy's queue length and squared capacity in a scatter plot. Explain whether policy rankings change with the coefficient, and the basis for choosing a cost coefficient in real applications.

Results to retain

• A cost–queue-length scatter plot for both policies and a return table by cost coefficient.

Starting conditions

After a constant-arrival-rate queue, design a situation where the arrival rate varies by time of day.

Procedure

1. Describe two situations with the same queue length $q=4$ but current arrival rates of 0.5 and 4.
2. Explain the difference in decisions between observing only q and observing both q and time of day.
3. Distinguish information actually available, such as load and price, from information known only in the future.

Results to retain

• 2 scenario-comparison rows and an observation design table.

Starting conditions

Based on the preceding experiments, choose Queue, Motor, or one of your research problems and design an experiment for real application.

Procedure

1. Define state, observation, action, reward, termination, and time limit in one table. Record each variable's units and the information you can actually obtain.
2. Write 3 expected transitions numerically and check whether rewards for good and bad actions match your intent.
3. Choose one rule-based comparison baseline, one evaluation condition, and one failure case of greatest concern. Also list the required measurements or simulator.

Results to retain

• MDP design table, 3 transition examples, and comparative experiment plan.
• If real research data are unavailable, design using synthetic conditions from an application case.

You may continue with the research problem and design table selected earlier. Revisit existing entries and transition examples using the concepts learned in this activity, and update changed entries or reasons for retaining them.