DQN Variants and Replay Buffer Sampling

Identify which problem each network modification addresses and validate one needed change, rather than applying every improvement.

Uses NumPy, PyTorch, and Gymnasium. In the final activity, read an external data card and analyze transitions you collected.

Throughput and cost of equipment and communication resources

An average of 2 items arrive at the synthetic queue each time step. Increasing service capacity reduces queue length but increases energy cost.

Observation·State

Queue length $q=0$…20. Observation $q/20$.

Action

Choose a processing capacity of 0, 2, or 4.

Reward·Constraint

−(next queue length + 0.2×capacity² + 10×dropped items). Time limit at 100 steps.

Comparison baseline

Constant capacity of 2, and a threshold policy based on queue length.

Metrics to check

Evaluation return, mean queue length, mean capacity², and dropped-item count.

Choose equipment inspection times

Frame the research problem as balancing the cost of inspecting every time against the cost of missed failures.

Observation·State

Recent sensor readings, trends, and time since the last inspection. Distinguish the actual fault state from observations.

Action

Continue operating, inspect, or replace.

Reward·Constraint

Operating profit minus inspection, replacement, and failure costs.

Comparison baseline

Periodic inspection and threshold alarms.

Metrics to check

Availability, failure count, and total cost. Check for failure label leakage.

Starting conditions

• $Q_{\mathrm{online}}=[2,4,3]$, $Q_{\mathrm{target}}=[5,1,2]$.
• $r=1$, $\gamma=.95$.
• Dueling: $V=2$, $A=[1,2,6]$.
• TD Error: $[0.1,1,5]$.

Procedure

1. Calculate DQN and Double DQN targets by hand and mark the parts of the equations that differ.
2. Use small arrays to inspect Dueling's mean correction and PER sampling probabilities for $\alpha=0$/.6.
3. In one line each, connect the three methods to the possible failures they target.

Results to retain

• Calculation table distinguishing targets, representations, and sampling probabilities.

Starting conditions

On Queue, compare dqn with just one of double, dueling, or per. Use 1800 transitions for each, the same seed=0, and the same evaluation set.

Procedure

1. Find where the selected method changes the code and first state why it may change the results.
2. Run dqn and the selected method with the same training budget and evaluation set.
3. Record which of return, queue length, and cost actually changed, and judge whether the method is needed based on these results.

Results to retain

• Selection rationale and an evaluation table for both configurations. If needed, leave rerunning seed=1 as an extension.

Starting conditions

Compare transitions collected by Random and Rule in the public CartPole environment. Replay cannot create experiences outside the collection policies' visited range.

Procedure

1. Compare angle, angular velocity, and action distributions and episode lengths for both policies with the same seed.
2. Reload transitions saved as NPZ, and explain how drawing one transition repeatedly differs from experiencing a new state.
3. From the Minari Fourrooms-Random data card, find the collection policy, observation format, environment ID, and license, and tabulate the fields needed for your research data. Leave data downloading and training as extensions.

Results to retain

• 1 plot of the visited-state distribution and a data card summary table.

• Minari · Fourrooms-Random dataset card
• Minari · data usage