Value-Based Reinforcement Learning

English translation

Learning objectives

Solving a small problem with a table provides a baseline for comparing complex function approximation models.

Execution conditions

Run the baseline Queue example with NumPy. Compare policies on seed=100…104, which were not used for training.

Applications

Throughput and cost of equipment and communication resources

An average of 2 items arrive at the synthetic queue each time step. Increasing service capacity reduces queue length but increases energy cost.

Observation and state: Queue length $q=0$ …20. Observation $q/20$ .

Action: choose processing capacity 0, 2, or 4.

Reward and constraints: −(next queue length + 0.2×capacity² + 10×dropped-item count). Time limit at 100 steps.

Comparison baselines: Constant capacity of 2, and a threshold policy based on queue length.

Metrics to check: Evaluation return, mean queue length, mean capacity², and dropped items.

Mobile robot paths and safety

An application design case choosing either grid paths or continuous velocity control.

Observation and state: Position, target-relative position, and nearby obstacle observations.

Action: choose a grid direction or bounded linear and angular velocities.

Reward and constraints: Separate arrival, collision, and movement costs. Distinguish collisions from timeouts.

Comparison baseline: shortest path and rule-based avoidance.

Metrics to check: Success rate, collision rate, travel distance, and time. Evaluate on unseen maps.

Q-learning in a small state space

Starting conditions

Learn Queue's $q=0,\ldots,20$ and three actions using a Q-table. Use $\gamma=0.95$ , $\alpha=0.1$ , and $\epsilon=0.2$ , and run 50 episodes with at most 100 steps per episode.

Procedure

Run the provided training loop and tabulate the Q-values and selected actions for $q\in\{0,5,10,20\}$ .
For evaluation, set $\epsilon=0$ and compare with the rule-based policy using the same seed.
If low capacity is chosen for large queues, first check visit counts and the reward structure.

Results to retain

A portion of the Q-table by state, a policy comparison table, and a return plot by training episode.

"""Synthetic service queue. An action chooses capacity, including its energy cost."""
import numpy as np

class Queue:
    def reset(self, seed=0):
        self.rng = np.random.default_rng(seed)
        self.q, self.t = 0, 0
        return self.q

    def step(self, action):
        capacity = (0, 2, 4)[int(action)]
        arriving = int(self.rng.poisson(2))
        raw = max(0, self.q - capacity) + arriving
        dropped = max(0, raw - 20)
        self.q = min(20, raw)
        self.t += 1
        reward = -(self.q + .2 * capacity**2 + 10 * dropped)
        return self.q, reward, False, self.t >= 100, {'queue': self.q, 'capacity': capacity, 'dropped': dropped}

def evaluate(policy, seeds=range(100, 105)):
    rows = []
    for seed in seeds:
        env = Queue(); q = env.reset(seed)
        total = waiting = energy = dropped = 0
        while True:
            q, r, terminated, truncated, info = env.step(policy(q))
            total += r; waiting += q; energy += info['capacity']**2; dropped += info['dropped']
            if terminated or truncated: break
        rows.append([total, waiting / env.t, energy / env.t, dropped])
    return np.array(rows)

policies = {'constant': lambda q: 1, 'threshold': lambda q: 0 if q == 0 else (1 if q < 5 else 2)}
for name, policy in policies.items():
    print(name, evaluate(policy).mean(axis=0))

Q=np.zeros((21,3));visits=np.zeros((21,3),dtype=int);history=[]
rng=np.random.default_rng(7)
for episode in range(50):
    env=Queue();q=env.reset(episode);total=0
    for k in range(100):
        a=int(rng.integers(3)) if rng.random()<.2 else int(Q[q].argmax())
        nq,r,te,tr,_=env.step(a)
        target=r+.95*(not te)*Q[nq].max()
        Q[q,a]+=.1*(target-Q[q,a]);visits[q,a]+=1;q=nq;total+=r
        if te or tr:break
    history.append(total)
print(evaluate(lambda q:int(Q[q].argmax())).mean(0))

Record results and self-check

Q( $a=0$ ):

Q( $a=1$ ):

Q( $a=2$ ):

Selected action:

Visit count:

Points to check

Did you distinguish action selection from exploration during evaluation?
Did you retain bootstrapping at time limits and separate evaluation seeds from training seeds?

Discount factor and decision making

Starting conditions

Compare $\gamma=.5$ and .95. Keep other training conditions and the total number of transitions fixed.

Procedure

Wrap the Q-table training loop in a train_q(gamma) function, and replace constant .95 in the target formula with argument gamma. Run this function for both discount factors.
Compare queue length, capacity², and dropped items on the same evaluation set.
If the selected action changed at a particular q, explain it in terms of short-term cost and future queue length.

Results to retain

A comparison table for both configurations and a plot of the selected action for each q.

# TODO: Wrap the training loop in train_q(gamma) and compare.

Record results and self-check

γ:

Evaluation return:

Mean queue length:

Capacity²:

Dropped items:

q values where policies differ:

Points to check

Did you keep the training budget unchanged?
Did you compare policies on a common evaluation set rather than one chance episode?

Apply this to your research problem

Starting conditions

Based on the preceding experiments, choose Queue, Motor, or one of your research problems and design an experiment for real application.

Procedure

Define state, observation, action, reward, termination, and time limit in one table. Record each variable's units and the information you can actually obtain.
Write 3 expected transitions numerically and check whether rewards for good and bad actions match your intent.
Choose one rule-based comparison baseline, one evaluation condition, and one failure case of greatest concern. Also list the required measurements or simulator.

Results to retain

MDP design table, 3 transition examples, and comparative experiment plan.
If real research data are unavailable, design using synthetic conditions from an application case.

You may continue with the research problem and design table selected earlier. Revisit existing entries and transition examples using the concepts learned in this activity, and update changed entries or reasons for retaining them.

Record results and self-check

Item:

Concrete definition and units:

Observability and verification method:

Points to check

Did you exclude future information and evaluation labels from observations?
Did you define domain-specific metrics rather than judging the research contribution solely by training return?