DQN variants and replay buffer sampling · Learning

English translation

Execution conditions

Uses NumPy, PyTorch, and Gymnasium. In the final activity, read an external data card and analyze transitions you collected.

Topics covered

Identify which problem each network modification addresses and validate one needed change, rather than applying every improvement.

Where the three techniques differ

Double changes next-action selection and value evaluation; Dueling changes the Q representation; PER changes experience sampling. They do not all address the same problem.

Representation and distribution

Dueling can combine streams using $Q=V+A-\operatorname{mean}(A)$ . PER samples high-error experiences more often while accounting for sampling bias and weights. It does not create states outside the experience distribution.

Worked example

$Q_{\mathrm{online}}=[2,4,3]$ , $Q_{\mathrm{target}}=[5,1,2]$ .
$r=1$ , $\gamma=.95$ .
Dueling: $V=2$ , $A=[1,2,6]$ .
TD Error: $[0.1,1,5]$ .

import numpy as np
online=np.array([2.,4.,3.]);target=np.array([5.,1.,2.])
print('DQN',1+.95*target.max(),'Double',1+.95*target[online.argmax()])
advantage=np.array([1.,2.,6.]);print('Dueling Q',2+advantage-advantage.mean())
error=np.array([.1,1.,5.])
for alpha in [0,.6]:
    probability=(error+1e-5)**alpha;probability/=probability.sum();print(alpha,probability)

DQN 5.75 Double 1.95
Dueling Q [0. 1. 5.]
0 [0.33333333 0.33333333 0.33333333]
0.6 [0.06478095 0.25788367 0.67733538]

Official documentation and data

Reference example

View more examples