> ## Documentation Index
> Fetch the complete documentation index at: https://ai.tharung.in/llms.txt
> Use this file to discover all available pages before exploring further.

# Reinforcement Learning (DQN on CartPole)

## 1. What is Reinforcement Learning?

Reinforcement Learning (RL) is a machine learning approach where an **agent learns by interacting with an environment**.

The agent:

1. Observes the current state.
2. Selects an action.
3. Receives a reward.
4. Learns from the result.
5. Repeats the process.

```text theme={null}
State
  ↓
Agent
  ↓
Action
  ↓
Environment
  ↓
Reward + New State
  ↓
Learning
```

***

## 2. CartPole Environment

CartPole is a simple reinforcement learning environment.

The goal is to keep a pole balanced on top of a moving cart.

```text theme={null}
       |
       |
       |
-------O-------
      Cart
```

The agent can perform two actions:

```text theme={null}
0 → Move Left
1 → Move Right
```

The longer the pole stays balanced, the higher the total reward.

***

## 3. OpenAI Gym / Gymnasium

Gym provides environments that can be used to test reinforcement learning algorithms.

The modern package is **Gymnasium**.

Install it:

```bash theme={null}
pip install gymnasium
```

Create the CartPole environment:

```python theme={null}
import gymnasium as gym

env = gym.make("CartPole-v1")
```

***

## 4. State

The state describes the current condition of the CartPole environment.

CartPole has **4 state values**:

```text theme={null}
Cart position
Cart velocity
Pole angle
Pole angular velocity
```

The neural network receives these 4 values as input.

```text theme={null}
4 state values
      ↓
Neural Network
```

***

## 5. Action

CartPole has two possible actions:

```text theme={null}
0 → Left
1 → Right
```

The neural network produces two Q-values:

```text theme={null}
Q(left)
Q(right)
```

The action with the highest Q-value is selected.

***

## 6. Reward

A reward is feedback from the environment.

For CartPole, the agent receives a reward for keeping the pole balanced.

Example:

```text theme={null}
Step 1 → Reward 1
Step 2 → Reward 1
Step 3 → Reward 1
Step 4 → Reward 1
```

Total reward:

```text theme={null}
4
```

A higher reward means the agent kept the pole balanced for longer.

***

# 7. What is DQN?

DQN stands for:

**Deep Q-Network**

DQN combines:

```text theme={null}
Q-Learning
+
Neural Network
=
Deep Q-Network
```

Instead of storing Q-values in a table, a neural network predicts the Q-values.

```text theme={null}
State
  ↓
DQN
  ↓
Q-values
  ↓
Best Action
```

***

# 8. Q-Values

A Q-value represents how useful an action is for the current state.

Example:

```text theme={null}
State
  ↓
DQN
  ↓
Q(left)  = 2.1
Q(right) = 4.5
```

Since:

```text theme={null}
4.5 > 2.1
```

the agent selects:

```text theme={null}
Right
```

***

# 9. DQN Architecture

The simple model used in `rl_demo.py` is:

```text theme={null}
Input: 4 values
      ↓
Linear Layer
4 → 32
      ↓
ReLU
      ↓
Linear Layer
32 → 2
      ↓
Q(left), Q(right)
```

The code:

```python theme={null}
model = nn.Sequential(
    nn.Linear(4, 32),
    nn.ReLU(),
    nn.Linear(32, 2)
)
```

The final `2` represents the two possible actions.

***

# 10. Choosing an Action

The model predicts Q-values:

```python theme={null}
q_values = model(state_tensor)
```

Example:

```text theme={null}
tensor([1.5, 3.2])
```

The highest value is selected:

```python theme={null}
action = q_values.argmax().item()
```

Therefore:

```text theme={null}
Q(left)  = 1.5
Q(right) = 3.2

Selected action = right
```

***

# 11. Learning Process

The simplified DQN follows this process:

```text theme={null}
Get State
   ↓
Predict Q-values
   ↓
Select Best Action
   ↓
Perform Action
   ↓
Receive Reward
   ↓
Get Next State
   ↓
Calculate Target
   ↓
Calculate Loss
   ↓
Update Neural Network
```

This process repeats for many episodes.

***

# 12. Code

### `rl_demo.py`

```python theme={null}
import gymnasium as gym
import torch
import torch.nn as nn
import torch.optim as optim


# Create CartPole environment
env = gym.make("CartPole-v1")


# Create a simple Q-network
model = nn.Sequential(
    nn.Linear(4, 32),
    nn.ReLU(),
    nn.Linear(32, 2)
)


# Create optimizer
optimizer = optim.Adam(model.parameters(), lr=0.001)


# Train for multiple episodes
for episode in range(100):

    state, info = env.reset()
    total_reward = 0

    for step in range(500):

        # Convert state to tensor
        state_tensor = torch.tensor(
            state,
            dtype=torch.float32
        )

        # Predict Q-values
        q_values = model(state_tensor)

        # Choose action with highest Q-value
        action = q_values.argmax().item()

        # Perform action
        next_state, reward, terminated, truncated, info = env.step(action)

        # Convert next state to tensor
        next_state_tensor = torch.tensor(
            next_state,
            dtype=torch.float32
        )

        # Calculate target Q-value
        with torch.no_grad():

            next_q = model(next_state_tensor).max()

            target = torch.tensor(
                reward
            ) + 0.99 * next_q * (
                1 - int(terminated or truncated)
            )

        # Get current Q-value
        current_q = q_values[action]

        # Calculate loss
        loss = (current_q - target) ** 2

        # Update model
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # Move to next state
        state = next_state

        total_reward += reward

        # Stop when episode ends
        if terminated or truncated:
            break

    # Print episode reward
    print(
        f"Episode {episode + 1}: "
        f"Reward = {total_reward}"
    )


# Close environment
env.close()
```

Output

```text theme={null}
Episode 1: Total Reward: 9.0
Episode 2: Total Reward: 9.0
Episode 3: Total Reward: 9.0
Episode 4: Total Reward: 9.0
Episode 5: Total Reward: 9.0
Episode 6: Total Reward: 9.0
Episode 7: Total Reward: 10.0
Episode 8: Total Reward: 10.0
Episode 9: Total Reward: 10.0
Episode 10: Total Reward: 9.0
Episode 11: Total Reward: 9.0
Episode 12: Total Reward: 9.0
.
.
.
Episode 93: Total Reward: 10.0
Episode 94: Total Reward: 9.0
Episode 95: Total Reward: 10.0
Episode 96: Total Reward: 8.0
Episode 97: Total Reward: 11.0
Episode 98: Total Reward: 10.0
Episode 99: Total Reward: 9.0
Episode 100: Total Reward: 11.0
```

***

# 13. Install Dependencies

```bash theme={null}
pip install torch gymnasium
```

Run:

```bash theme={null}
python rl_demo.py
```

Example output:

```text theme={null}
Episode 1: Reward = 12
Episode 2: Reward = 15
Episode 3: Reward = 18
Episode 4: Reward = 22
...
Episode 50: Reward = 45
...
```

The exact output can vary because reinforcement learning involves randomness.

***

# 14. Understanding the Important Code

### Create environment

```python theme={null}
env = gym.make("CartPole-v1")
```

Creates the CartPole environment.

### Get state

```python theme={null}
state, info = env.reset()
```

Starts a new episode and returns the initial state.

### Predict Q-values

```python theme={null}
q_values = model(state_tensor)
```

The neural network predicts the value of each action.

### Select action

```python theme={null}
action = q_values.argmax().item()
```

Selects the action with the highest Q-value.

### Perform action

```python theme={null}
next_state, reward, terminated, truncated, info = env.step(action)
```

The environment returns:

```text theme={null}
Next state
Reward
Episode status
```

### Calculate loss

```python theme={null}
loss = (current_q - target) ** 2
```

Measures the difference between the predicted Q-value and the target Q-value.

### Update model

```python theme={null}
optimizer.zero_grad()
loss.backward()
optimizer.step()
```

Updates the neural network so that future Q-value predictions become better.

***

# 15. Complete Learning Flow

```text theme={null}
                 CartPole
                    ↑
                    |
                  Action
                    ↑
              Q-Network
                    ↑
                  State
                    |
                    ↓
                 Reward
                    |
                    ↓
                 Learning
```

Or simply:

```text theme={null}
State
 ↓
Neural Network
 ↓
Q-values
 ↓
Best Action
 ↓
CartPole
 ↓
Reward
 ↓
Loss
 ↓
Update Network
```

***

# 16. Main Learning

The main concepts covered are:

```text theme={null}
Reinforcement Learning
        ↓
Environment
        ↓
State
        ↓
Action
        ↓
Reward
        ↓
Q-Learning
        ↓
Neural Network
        ↓
DQN
```

### Key takeaway

**DQN uses a neural network to estimate Q-values and choose actions that can maximize future rewards.**