Value-Based Reinforcement Learning · Study

English translation

Execution conditions

Run the baseline Queue example with NumPy. Compare policies on seed=100…104, which were not used for training.

Topics covered

Solving a small problem with a table provides a baseline for comparing complex function approximation models.

Q update

Q(s,a) ← Q(s,a)+α[r+γ max Q(s′,·)−Q(s,a)]. Exclude future value when the task itself terminates. Interpret time limits separately.

Collection and evaluation

Exploratory actions are needed during training, but evaluate policies separately under the same conditions. Also inspect Q-values for unvisited states.

Worked example

Learn a Q-table for Queue with $q=0$ …20 and 3 actions. Train for 50 episodes using $\gamma=.95$ , $\alpha=.1$ , and $\epsilon=.2$ , with 100 steps per episode.

"""Synthetic service queue. An action chooses capacity, including its energy cost."""
import numpy as np

class Queue:
    def reset(self, seed=0):
        self.rng = np.random.default_rng(seed)
        self.q, self.t = 0, 0
        return self.q

    def step(self, action):
        capacity = (0, 2, 4)[int(action)]
        arriving = int(self.rng.poisson(2))
        raw = max(0, self.q - capacity) + arriving
        dropped = max(0, raw - 20)
        self.q = min(20, raw)
        self.t += 1
        reward = -(self.q + .2 * capacity**2 + 10 * dropped)
        return self.q, reward, False, self.t >= 100, {'queue': self.q, 'capacity': capacity, 'dropped': dropped}

def evaluate(policy, seeds=range(100, 105)):
    rows = []
    for seed in seeds:
        env = Queue(); q = env.reset(seed)
        total = waiting = energy = dropped = 0
        while True:
            q, r, terminated, truncated, info = env.step(policy(q))
            total += r; waiting += q; energy += info['capacity']**2; dropped += info['dropped']
            if terminated or truncated: break
        rows.append([total, waiting / env.t, energy / env.t, dropped])
    return np.array(rows)

policies = {'constant': lambda q: 1, 'threshold': lambda q: 0 if q == 0 else (1 if q < 5 else 2)}
for name, policy in policies.items():
    print(name, evaluate(policy).mean(axis=0))

Q=np.zeros((21,3));visits=np.zeros((21,3),dtype=int);history=[]
rng=np.random.default_rng(7)
for episode in range(50):
    env=Queue();q=env.reset(episode);total=0
    for k in range(100):
        a=int(rng.integers(3)) if rng.random()<.2 else int(Q[q].argmax())
        nq,r,te,tr,_=env.step(a)
        target=r+.95*(not te)*Q[nq].max()
        Q[q,a]+=.1*(target-Q[q,a]);visits[q,a]+=1;q=nq;total+=r
        if te or tr:break
    history.append(total)
print(evaluate(lambda q:int(Q[q].argmax())).mean(0))

constant [-921.2      8.052    4.       3.6  ]
threshold [-387.4     2.77    5.52    0.  ]
[-416.6      2.758    7.04     0.   ]

Reference example

View more examples

Value-Based Reinforcement Learning