Introduction to Reinforcement Learning · Study

English translation

Execution conditions

Run the baseline Queue example with NumPy. Compare policies on seed=100…104, which were not used for training.

Topics covered

Check whether a high-reward policy actually produces the desired operation by examining cost and queue length separately.

Reward and the actual objective

Return results from reward design. Examine queue length, cost, and losses separately in their original units to judge whether a high return matches the actual objective.

Observation and state

Use observations available at decision time, such as current queue length and arrival rate. The actual number of future arrivals is not yet known to the agent.

Examine whether the next-state probability distribution can be determined from the same observation and action.

Worked example

Examine the meaning of cost coefficient 0.2 using Queue's two policies as a baseline.

"""Synthetic service queue. An action chooses capacity, including its energy cost."""
import numpy as np

class Queue:
    def reset(self, seed=0):
        self.rng = np.random.default_rng(seed)
        self.q, self.t = 0, 0
        return self.q

    def step(self, action):
        capacity = (0, 2, 4)[int(action)]
        arriving = int(self.rng.poisson(2))
        raw = max(0, self.q - capacity) + arriving
        dropped = max(0, raw - 20)
        self.q = min(20, raw)
        self.t += 1
        reward = -(self.q + .2 * capacity**2 + 10 * dropped)
        return self.q, reward, False, self.t >= 100, {'queue': self.q, 'capacity': capacity, 'dropped': dropped}

def evaluate(policy, seeds=range(100, 105)):
    rows = []
    for seed in seeds:
        env = Queue(); q = env.reset(seed)
        total = waiting = energy = dropped = 0
        while True:
            q, r, terminated, truncated, info = env.step(policy(q))
            total += r; waiting += q; energy += info['capacity']**2; dropped += info['dropped']
            if terminated or truncated: break
        rows.append([total, waiting / env.t, energy / env.t, dropped])
    return np.array(rows)

policies = {'constant': lambda q: 1, 'threshold': lambda q: 0 if q == 0 else (1 if q < 5 else 2)}
for name, policy in policies.items():
    print(name, evaluate(policy).mean(axis=0))

constant [-921.2      8.052    4.       3.6  ]
threshold [-387.4     2.77    5.52    0.  ]

Reference example

View more examples

Introduction to Reinforcement Learning