Value-Based Reinforcement Learning

Solving a small problem with a table provides a baseline for comparing complex function approximation models.

Run the baseline Queue example with NumPy. Compare policies on seed=100…104, which were not used for training.

Throughput and cost of equipment and communication resources

An average of 2 items arrive at the synthetic queue each time step. Increasing service capacity reduces queue length but increases energy cost.

Observation·State

Queue length $q=0$…20. Observation $q/20$.

Action

Choose a processing capacity of 0, 2, or 4.

Reward·Constraint

−(next queue length + 0.2×capacity² + 10×dropped items). Time limit at 100 steps.

Comparison baseline

Constant capacity of 2, and a threshold policy based on queue length.

Metrics to check

Evaluation return, mean queue length, mean capacity², and dropped-item count.

Mobile robot paths and safety

An application design case choosing either grid paths or continuous velocity control.

Observation·State

Position, target-relative position, and nearby obstacle observations.

Action

Grid direction selection or bounded linear and angular velocities.

Reward·Constraint

Separate arrival, collision, and movement costs. Distinguish collisions from timeouts.

Comparison baseline

Shortest path and rule-based avoidance.

Metrics to check

Success rate, collision rate, travel distance, and time. Evaluate on unseen maps.

Starting conditions

Learn Queue's $q=0,\ldots,20$ and three actions using a Q-table. Use $\gamma=0.95$, $\alpha=0.1$, and $\epsilon=0.2$, and run 50 episodes with at most 100 steps per episode.

Procedure

1. Run the provided training loop and tabulate the Q-values and selected actions for $q\in\{0,5,10,20\}$.
2. For evaluation, set $\epsilon=0$ and compare with the rule-based policy using the same seed.
3. If low capacity is chosen for large queues, first check visit counts and the reward structure.

Results to retain

• A portion of the Q-table by state, a policy comparison table, and a return plot by training episode.

Starting conditions

Compare $\gamma=.5$ and .95. Keep other training conditions and the total number of transitions fixed.

Procedure

1. Wrap the Q-table training loop in a train_q(gamma) function, and replace constant .95 in the target formula with argument gamma. Run this function for both discount factors.
2. Compare queue length, capacity², and dropped items on the same evaluation set.
3. If the selected action changed at a particular q, explain it in terms of short-term cost and future queue length.

Results to retain

• A comparison table for both configurations and a plot of the selected action for each q.

Starting conditions

Based on the preceding experiments, choose Queue, Motor, or one of your research problems and design an experiment for real application.

Procedure

1. Define state, observation, action, reward, termination, and time limit in one table. Record each variable's units and the information you can actually obtain.
2. Write 3 expected transitions numerically and check whether rewards for good and bad actions match your intent.
3. Choose one rule-based comparison baseline, one evaluation condition, and one failure case of greatest concern. Also list the required measurements or simulator.

Results to retain

• MDP design table, 3 transition examples, and comparative experiment plan.
• If real research data are unavailable, design using synthetic conditions from an application case.

You may continue with the research problem and design table selected earlier. Revisit existing entries and transition examples using the concepts learned in this activity, and update changed entries or reasons for retaining them.