NiiL

Introduction to Reinforcement Learning

English translation

Learning objectives

Check whether a high-reward policy actually produces the desired operation by examining cost and queue length separately.

Execution conditions

Run the baseline Queue example with NumPy. Compare policies on seed=100…104, which were not used for training.

Applications

Throughput and cost of equipment and communication resources

An average of 2 items arrive at the synthetic queue each time step. Increasing service capacity reduces queue length but increases energy cost.

Observation and state: Queue length q=0q=0…20. Observation q/20q/20.

Action: choose processing capacity 0, 2, or 4.

Reward and constraints: −(next queue length + 0.2×capacity² + 10×dropped-item count). Time limit at 100 steps.

Comparison baselines: Constant capacity of 2, and a threshold policy based on queue length.

Metrics to check: Evaluation return, mean queue length, mean capacity², and dropped items.

Design energy storage operation

A research design case distinguishing the roles of price, load, and stored energy before connecting measured data.

Observation and state: SOC, load, price, and time. Observations must exclude future information.

Action: charging/discharging power. First choose discrete levels or a continuous range.

Reward and constraints: electricity and degradation costs. Specify rules for handling SOC and power limits.

Comparison baseline: time-of-day rules and, if possible, optimization using the same information.

Metrics to check: total cost, constraint violations, and maximum grid power draw. Split train/test data by date.

Is an action that earns high rewards good operation?

Starting conditions

Use the Queue policies constant and threshold. The reward combines queue length, squared capacity cost, and the number of dropped items; the baseline cost coefficient is 0.20.2.

Procedure

  1. Run evaluate with the same evaluation seed. Its four returned columns are episode return, mean queue length per step, mean squared capacity per step, and total dropped items in the episode. Record the means for each policy.
  2. Apply each cost coefficient c∈{0.05,0.2,1.0}c\in\{0.05,0.2,1.0\} to the same evaluation records. Multiply the mean queue length and mean squared capacity by the episode length of 100, apply coefficient 10 to total dropped items as in the code, and negate the sum of the three costs.
  3. Tabulate returns by cost coefficient and policy, and plot each policy's queue length and squared capacity in a scatter plot. Explain whether policy rankings change with the coefficient, and the basis for choosing a cost coefficient in real applications.

Results to retain

"""Synthetic service queue. An action chooses capacity, including its energy cost."""
import numpy as np

class Queue:
    def reset(self, seed=0):
        self.rng = np.random.default_rng(seed)
        self.q, self.t = 0, 0
        return self.q

    def step(self, action):
        capacity = (0, 2, 4)[int(action)]
        arriving = int(self.rng.poisson(2))
        raw = max(0, self.q - capacity) + arriving
        dropped = max(0, raw - 20)
        self.q = min(20, raw)
        self.t += 1
        reward = -(self.q + .2 * capacity**2 + 10 * dropped)
        return self.q, reward, False, self.t >= 100, {'queue': self.q, 'capacity': capacity, 'dropped': dropped}

def evaluate(policy, seeds=range(100, 105)):
    rows = []
    for seed in seeds:
        env = Queue(); q = env.reset(seed)
        total = waiting = energy = dropped = 0
        while True:
            q, r, terminated, truncated, info = env.step(policy(q))
            total += r; waiting += q; energy += info['capacity']**2; dropped += info['dropped']
            if terminated or truncated: break
        rows.append([total, waiting / env.t, energy / env.t, dropped])
    return np.array(rows)

policies = {'constant': lambda q: 1, 'threshold': lambda q: 0 if q == 0 else (1 if q < 5 else 2)}
for name, policy in policies.items():
    print(name, evaluate(policy).mean(axis=0))

Record results and self-check

Policy:

Cost coefficient:

Mean queue length:

Dropped items:

Recalculated return:

Points to check

When the state is insufficient

Starting conditions

After a constant-arrival-rate queue, design a situation where the arrival rate varies by time of day.

Procedure

  1. Describe two situations with the same queue length q=4q=4 but current arrival rates of 0.5 and 4.
  2. Explain the difference in decisions between observing only q and observing both q and time of day.
  3. Distinguish information actually available, such as load and price, from information known only in the future.

Results to retain

Record results and self-check

Scenario:

Current observation:

Missing information:

Predicted action and reason:

Points to check

Apply this to your research problem

Starting conditions

Based on the preceding experiments, choose Queue, Motor, or one of your research problems and design an experiment for real application.

Procedure

  1. Define state, observation, action, reward, termination, and time limit in one table. Record each variable's units and the information you can actually obtain.
  2. Write 3 expected transitions numerically and check whether rewards for good and bad actions match your intent.
  3. Choose one rule-based comparison baseline, one evaluation condition, and one failure case of greatest concern. Also list the required measurements or simulator.

Results to retain

You may continue with the research problem and design table selected earlier. Revisit existing entries and transition examples using the concepts learned in this activity, and update changed entries or reasons for retaining them.

Record results and self-check

Item:

Concrete definition and units:

Observability and verification method:

Points to check