Week 1: Introduction to Reinforcement Learning

[What is Reinforcement Learning?]

1. Definition

• A field of machine learning in which an agent interacts with an environment to learn a policy that maximizes reward

2. Main components

• Agent
  • The entity that selects actions according to a policy
• Environment
  • The external world that receives the agent's action and returns the next state and reward
• State
  • Information describing the environment's current situation
    • Example: Cart position and pole angle in CartPole
• Action
  • A choice the agent can make in the environment
    • Example: Move the cart left or right.
• Reward
  • A real-valued signal received from the environment as a result of an action
  • Goal: Improve the policy to maximize this reward.
• Policy
  • Definition: A strategy for deciding which action to take in a given state
  • Stochastic policy: π(a|s) is the probability of choosing action a in state s.

3. The objective of reinforcement learning

• Definition: find a policy that maximizes expected cumulative reward (return)
• Training process
  • Initialize the environment and receive the state
  • Take an action
  • Obtain the next state and reward
  • Construct and repeat episodes
  • Gradually improve the policy

4. Policy

• Definition: A strategy for deciding which action to take in a given state
• Feature: takes a state and outputs a probability distribution over actions
• Stochastic policy: π(a|s) is the probability of choosing action a in state s

5. Development environment setup (Google Colab)

• Feature: Cloud-based Jupyter notebook environment
• Benefit: Use PyTorch and OpenAI Gym without additional setup.

[Activity 1: PyTorch tensor basics]

1. Create tensors

import torch

x = torch.tensor([1.0, 2.0, 3.0])
print(x)

tensor([1., 2., 3.])

2. Tensor operations

y = x + 2
print(y)

tensor([3., 4., 5.])

3. Check tensor size

print(x.shape)

torch.Size([3])

4. Check tensor data types

print(x.dtype)

torch.float32

5. Check the device used for tensor operations

print(x.device)

cpu

6. Check GPU availability and move tensors

if torch.cuda.is_available():
  device = torch.device('cuda')
  print('GPU 사용 가능')
else:
  device = torch.device('cpu')
  print('GPU 사용 불가능')

x = x.to(device)
print(x.device)

GPU 사용 불가능
cpu

7. Access and modify tensor elements

print(x[0])

x[0] = 10.0
print(x)

tensor(1.)
tensor([10.,  2.,  3.])

8. Change tensor dimensions

x = x.view(1, 3)  # Reshape to (1, 3)
print(x)
print(x.shape)

tensor([[10.,  2.,  3.]])
torch.Size([1, 3])

9. Tensor element sums and maxima

print(torch.sum(x))

print(torch.max(x))

tensor(15.)
tensor(10.)

10. Convert tensors to NumPy arrays and NumPy arrays to tensors

x_np = x.cpu().numpy()
print(x_np)

x_tensor = torch.from_numpy(x_np)
print(x_tensor)

[[10.  2.  3.]]
tensor([[10.,  2.,  3.]])

[Exercise 2: take random actions in CartPole-v1]

• The agent takes random actions in the OpenAI Gym CartPole-v1 environment.

1. Import libraries

import gymnasium as gym
import random

2. Create the environment

• Create a CartPole-v1 environment using the OpenAI Gym make() function.

env = gym.make('CartPole-v1')

3. Repeat episodes

for episode in range(10):  # Run 10 episodes
    observation, _ = env.reset()  # Reset the environment
    done = False  # Initialize the variable indicating whether the episode has ended
    total_reward = 0  # Initialize the total reward for the episode

    while not done:  # Repeat until the episode ends
        # Select a random action (0: left, 1: right)
        action = env.action_space.sample()

        # Apply the selected action to the environment and return the next state, reward, episode end status, and additional information
        next_observation, reward, terminated, truncated, info = env.step(action)
        done = terminated or truncated

        total_reward += reward  # Accumulate rewards over the episode
        observation = next_observation  # Update to the next state

        # Render the environment (comment out to disable rendering)
        # env.render()

    # Print the total reward when each episode ends
    print(f"Episode {episode + 1}: Total reward = {total_reward}")

Episode 1: Total reward = 30.0
Episode 2: Total reward = 19.0
Episode 3: Total reward = 38.0
Episode 4: Total reward = 19.0
Episode 5: Total reward = 14.0
Episode 6: Total reward = 18.0
Episode 7: Total reward = 11.0
Episode 8: Total reward = 33.0
Episode 9: Total reward = 51.0
Episode 10: Total reward = 33.0

4. Close the environment

• Close the environment after all episodes end

env.close()