Week 4 Double DQN

[Key objectives]

• Understand Double DQN and learn how it reduces Q-value overestimation.
• Understand how to apply Double DQN to existing DQN code.

[Q-value overestimation]

• Because DQN directly uses the maximum Q-value, noise or instability early in training can lead it to select values higher than their true values.
• This can distort the policy.

1. Example of Q-value overestimation: A game

• Problem: A simple maze escape game
• Goal: Escape the maze and obtain a reward
• Agent: the character trained using DQN

2. How DQN overestimation arises

• Early in training, the agent obtains a high Q-value when it happens to reach a particular maze region
• This high Q-value may overestimate the actual reward.
  • A high Q-value does not necessarily mean reaching that area is valuable.
  • It may simply mean that other valuable areas remain unknown.
• The agent treats these overestimated Q-values as 'truly valuable' and regards that region as the optimal path

3. Policy distortion and problems

• That region may not actually be the optimal path, and a shorter or safer alternative may exist
• However, the overestimated Q-values keep leading the agent to choose that region incorrectly
• As a result, the agent fails to learn the optimal path and either fails to escape the maze or takes longer.

[Double DQN idea]

• Separate action selection from Q-value evaluation
• Standard DQN TD target: Use the target network both to select the next-state action and to evaluate its Q-value.
• Double DQN:
  • Select the optimal action $a^*$ using the main network.
  • Evaluate the Q-value of the selected action $a^*$ using the target network.

[Double DQN training update]

• Temporal Difference Target (TD Target):
  • Conventional DQN: $r + \gamma \underset{a'}{max\;} Q(s', a'; \theta^-)$
  • Double DQN: $r + \gamma Q(s', \underset{a}{arg \text{ } max \text{ }} Q(s', a; \theta); \theta^-)$
    • $r$: immediate reward
    • $\gamma$: discount factor
    • $s'$: next state
    • $\theta$: main network parameters
    • $\theta^-$: target network parameters
    • $\underset{a}{arg \text{ } max \text{ }} Q(s', a; \theta)$: select the action $a$ with the highest Q-value using the main network
    • $Q(s', \underset{a}{arg \text{ } max \text{ }} Q(s', a; \theta); \theta^-)$: evaluate the target network Q-value for the selected action $a$
• Loss Function (Loss Function): train to minimize TD error
  • TD error: the difference between the current network prediction and the actual target
  • $Error_{TD, DQN} = Q(s, a; \theta) - (r + \gamma \underset{a'}{max} \text{ } Q(s', a'; \theta^-))$
    • $\underset{a'}{max} \text{ } Q(s', a'; \theta^-)$: use the same network ($\theta^-$) to select the maximum Q-value and evaluate Q-values
    • Q-value overestimation occurs
  • $Error_{TD, DDQN} = Q(s, a; \theta) - (r + \gamma Q(s', \underset{a}{arg \text{ } max \text{ }} Q(s', a; \theta); \theta^-))$
    • $\underset{a}{arg \text{ } max \text{ }} Q(S', a'; \theta)$: use the main network ($\theta$) to select, at the next state ($S'$), the action ($a'$) with the highest Q-value
    • $Q(S', \underset{a}{arg \text{ } max \text{ }} Q(S', a'; \theta); \theta')$: evaluate the selected action's ($a'$) Q-value using the target network ($θ'$)
    • This difference plays an important role in reducing Q-value overestimation
• Update: adjust main network parameters to minimize the loss function

[Exercise: train CartPole using Double DQN]

1. Environment setup

• gym: Provides the CartPole environment
• torch: Deep learning models and computations
• numpy: Array and numerical operations
• collections.deque: implement an experience replay buffer

import gymnasium as gym
import random
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from collections import deque

env = gym.make('CartPole-v1', render_mode = 'rgb_array')
state_size = env.observation_space.shape[0]
action_size = env.action_space.n

2. Define the DQN model

• DQNAgent class: Define the neural network model.
• Use 3 fully connected layers (nn.Linear).
• Apply the ReLU activation function (torch.relu)
• Input: state; output: Q-values for each action

class DQNAgent(nn.Module):
    def __init__(self, state_size, action_size):
        super(DQNAgent, self).__init__()
        self.fc1 = nn.Linear(state_size, 128)
        self.fc2 = nn.Linear(128, 128)
        self.fc3 = nn.Linear(128, action_size)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        return self.fc3(x)

3. Set hyperparameters

• Set the learning rate (learning_rate), discount factor (gamma), exploration probability (epsilon), etc.
• Set the minibatch size (batch_size) and memory size (memory_size).
• Set the target network update interval (target_update_frequency)

learning_rate = 0.0003
gamma = 0.99  # Discount factor
epsilon = 1.0  # Initial exploration probability
epsilon_min = 0.01
epsilon_decay = 0.995
batch_size = 128
memory_size = 30000
target_update_frequency = 5  # Target network update interval

4. Create the DQN agent and target network

• Create DQNAgent model instances (agent, target_agent)
• Initialize the target network (copy the agent network's weights).
• Set the Adam optimizer and MSE loss function.

agent = DQNAgent(state_size, action_size)
target_agent = DQNAgent(state_size, action_size)
target_agent.load_state_dict(agent.state_dict())
optimizer = optim.Adam(agent.parameters(), lr=learning_rate)
criterion = nn.MSELoss()

5. Initialize the experience replay buffer

• Initialize the experience replay buffer (memory) using deque.

memory = deque(maxlen=memory_size)

6. Training function (train)

• Sample a minibatch from the experience replay buffer.
• Extract state, action, reward, next state, and termination status
• Double DQN in brief: Select the action with the highest next-state Q-value using the agent network, and compute the Q-value of that action using the target network.
• Compute the loss function by comparing predicted and target Q-values.
• Backpropagation and weight updates

def train(batch_size):
    minibatch = random.sample(memory, batch_size)
    states = torch.tensor(np.array([transition[0] for transition in minibatch]), dtype=torch.float32)
    actions = torch.tensor(np.array([transition[1] for transition in minibatch]), dtype=torch.long)
    rewards = torch.tensor(np.array([transition[2] for transition in minibatch]), dtype=torch.float32)
    next_states = torch.tensor(np.array([transition[3] for transition in minibatch]), dtype=torch.float32)
    dones = torch.tensor(np.array([transition[4] for transition in minibatch]), dtype=torch.float32)

    # Double DQN key step: Select the action with the highest next-state Q-value using the agent network
    next_actions = agent(next_states).argmax(1).unsqueeze(1)
    # Compute the Q-value of the selected action using the target network
    target_q_values = target_agent(next_states).gather(1, next_actions).squeeze()
    targets = rewards + gamma * target_q_values * (1 - dones)

    predicted_q_values = agent(states).gather(1, actions.unsqueeze(1)).squeeze()
    loss = criterion(predicted_q_values, targets)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

7. Training loop

• Repeat episodes episodes times.
• Reset the environment and initialize the state for each episode
• Choose an action with an ε-greedy policy.
• Execute an action in the environment and obtain the next state, reward, and termination status.
• Add experiences to the replay buffer
• Call the training function when the buffer size reaches batch_size.
• Update the target network and decrease exploration probability.
• Print episode results

# Training loop
episodes = 100
for e in range(episodes):
    state, _ = env.reset()
    state = torch.tensor(state, dtype=torch.float32)
    done = False
    total_reward = 0

    while not done:
        if random.random() < epsilon:
            action = env.action_space.sample()
        else:
            action = agent(state).argmax().item()
        next_state, reward, terminated, truncated, _ = env.step(action)
        next_state = torch.tensor(next_state, dtype=torch.float32)
        done = terminated or truncated

        total_reward += reward
        memory.append((state, action, reward, next_state, done))

        if len(memory) >= batch_size:
            train(batch_size)

        state = next_state
        if e % target_update_frequency == 0:
            target_agent.load_state_dict(agent.state_dict())

        if epsilon > epsilon_min:
            epsilon *= epsilon_decay

    if (e+1) % 20 == 0:
        print(f"Episode {e + 1}/{episodes}, Total Reward: {total_reward}, Epsilon: {epsilon:.3f}")

Episode 20/100, Total Reward: 302.0, Epsilon: 0.010
Episode 40/100, Total Reward: 133.0, Epsilon: 0.010
Episode 60/100, Total Reward: 132.0, Epsilon: 0.010
Episode 80/100, Total Reward: 124.0, Epsilon: 0.010
Episode 100/100, Total Reward: 138.0, Epsilon: 0.010

8. Evaluate the trained agent (rendering with GIF)

import imageio
import os

frames = []
state, _ = env.reset()
state = torch.tensor(state, dtype=torch.float32)
done = False
for i in range(3000): # 3000 steps simulation
  action = agent(state).argmax().item()
  next_state, reward, terminated, truncated, _ = env.step(action)
  next_state = torch.tensor(next_state, dtype=torch.float32)
  state = next_state
  done = terminated or truncated
  frames.append(env.render())

  if done:
    break

imageio.mimsave('cartpole_simulation_doubleDQN.gif', frames, duration=33) # save to gif file
print("GIF saved as cartpole_simulation.gif")

GIF saved as cartpole_simulation.gif