> ## Documentation Index
> Fetch the complete documentation index at: https://ai.tharung.in/llms.txt
> Use this file to discover all available pages before exploring further.

# Hyperparameter Tuning

### What is Hyperparameter Tuning?

**Hyperparameter Tuning** is the process of finding the **best settings (hyperparameters)** for a machine learning model.

Hyperparameters are values that we set **before training**.

Example for a Random Forest:

```text theme={null}
n_estimators = 100
max_depth = 10
```

We don't know which values will give the best performance, so we try different combinations.

Two common methods:

1. **GridSearchCV**
2. **RandomizedSearchCV**

***

# 1. GridSearchCV

### Definition

**GridSearchCV** tries **every possible combination** of the hyperparameters we provide.

### Example

Suppose we want to find the best:

```text theme={null}
n_estimators → 50, 100
max_depth    → 5, 10
```

GridSearchCV tries:

```text theme={null}
50, 5
50, 10
100, 5
100, 10
```

So there are:

```text theme={null}
2 × 2 = 4 combinations
```

### Simple Code Example

```python theme={null}
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

model = RandomForestClassifier(random_state=42)

params = {
    "n_estimators": [50, 100],
    "max_depth": [3, 5]
}

grid = GridSearchCV(
    model,
    params,
    cv=5,
    scoring="accuracy"
)

grid.fit(X, y)

print(grid.best_params_)
print(grid.best_score_)
```

### Output

```text theme={null}
{'max_depth': 3, 'n_estimators': 50}
0.9666666666666667
```

The exact best parameters can vary with the dataset and search space.

### Important Terms

**`param_grid`**

Contains the hyperparameter values to test.

```python theme={null}
params = {
    "n_estimators": [50, 100],
    "max_depth": [3, 5]
}
```

**`cv=5`**

Uses **5-fold cross-validation**.

```text theme={null}
Data
 ↓
Fold 1 → Validation
Fold 2 → Validation
Fold 3 → Validation
Fold 4 → Validation
Fold 5 → Validation
```

The model is trained and evaluated multiple times.

**`best_params_`**

Returns the best hyperparameter combination.

**`best_score_`**

Returns the best cross-validation score.

### Advantages

* Tests **every combination**.
* Can find the best combination within the given grid.
* Easy to understand.

### Disadvantages

* Can become **very slow** when many parameters and values are provided.
* Computationally expensive.

### Remember

> **GridSearchCV = Try everything in the grid.**

***

# 2. RandomizedSearchCV

### Definition

**RandomizedSearchCV** randomly selects combinations from the given hyperparameter distributions.

Unlike GridSearchCV, it **does not test every combination**.

### Example

Suppose we have:

```python theme={null}
params = {
    "n_estimators": [50, 100, 150, 200],
    "max_depth": [3, 5, 10, 15]
}
```

There are:

```text theme={null}
4 × 4 = 16 combinations
```

Instead of trying all 16, we can tell RandomizedSearchCV:

```text theme={null}
n_iter = 5
```

It randomly tests only **5 combinations**.

### Code Example

```python theme={null}
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

model = RandomForestClassifier(random_state=42)

params = {
    "n_estimators": [50, 100, 150, 200],
    "max_depth": [3, 5, 10, 15]
}

random_search = RandomizedSearchCV(
    model,
    params,
    n_iter=5,
    cv=5,
    scoring="accuracy",
    random_state=42
)

random_search.fit(X, y)

print(random_search.best_params_)
print(random_search.best_score_)
```

### Output

```text theme={null}
{'n_estimators': 150, 'max_depth': 5}
0.9666666666666667
```

The exact result may differ depending on the dataset and search space.

### Important Terms

**`n_iter=5`**

Tests only **5 randomly selected combinations**.

**`random_state=42`**

Makes the random selection reproducible.

**`best_params_`**

Returns the best combination found.

**`best_score_`**

Returns the best cross-validation score.

### Advantages

* Faster than GridSearchCV for large search spaces.
* Can explore many possible values.
* Useful when there are many hyperparameters.

### Disadvantages

* May miss the actual best combination.
* Results depend on the randomly selected combinations.

### Remember

> **RandomizedSearchCV = Try some random combinations.**

***

# GridSearchCV vs RandomizedSearchCV

| Feature            | GridSearchCV      | RandomizedSearchCV    |
| ------------------ | ----------------- | --------------------- |
| Search             | Every combination | Random combinations   |
| Speed              | Slower            | Usually faster        |
| Accuracy of search | Exhaustive        | Approximate           |
| Large search space | Not ideal         | Better                |
| Parameter control  | Exact grid        | Random sampling       |
| Main parameter     | `param_grid`      | `param_distributions` |
| Number of tests    | All combinations  | `n_iter` combinations |

### Example

Suppose:

```text theme={null}
10 values for parameter A
10 values for parameter B
10 values for parameter C
```

Total combinations:

```text theme={null}
10 × 10 × 10 = 1000
```

**GridSearchCV:**

```text theme={null}
Tests all 1000
```

**RandomizedSearchCV:**

```text theme={null}
n_iter = 50

Tests only 50 random combinations
```

***

# Complete Tuning Process

```text theme={null}
        Machine Learning Model
                 ↓
       Choose Hyperparameters
                 ↓
       ┌───────────────────┐
       │ GridSearchCV                       │
       │        OR                             │
       │ RandomizedSearchCV               │
       └─────────┬─────────┘
                 ↓
        Cross Validation
                 ↓
          Compare Scores
                 ↓
        Best Parameters
                 ↓
        Train Final Model
```

### Easy way to remember

```text theme={null}
GridSearchCV
    ↓
"Try ALL combinations"

RandomizedSearchCV
    ↓
"Try SOME random combinations"
```

### One-line summary

> **GridSearchCV exhaustively searches a specified grid, while RandomizedSearchCV randomly samples a fixed number of hyperparameter combinations.**

# Cross-Validation

### What is Cross-Validation?

**Cross-Validation** is a technique used to evaluate how well a machine learning model performs on **unseen data**.

Instead of splitting the dataset only once into training and testing data, we **split the data multiple times** and evaluate the model multiple times.

```text theme={null}
Dataset
   ↓
Split into multiple parts
   ↓
Train + Validate multiple times
   ↓
Average the scores
   ↓
Final performance
```

Two important methods:

1. **K-Fold Cross-Validation**
2. **Stratified K-Fold Cross-Validation**

***

# 1. K-Fold Cross-Validation

### Definition

In **K-Fold Cross-Validation**, the dataset is divided into **K equal or nearly equal parts**, called **folds**.

Each fold is used as the **validation set once**, while the remaining folds are used for training.

### Example

Suppose:

```text theme={null}
K = 5
```

The dataset is divided into:

```text theme={null}
Fold 1
Fold 2
Fold 3
Fold 4
Fold 5
```

Then:

```text theme={null}
Round 1:
Train → Fold 2, 3, 4, 5
Test  → Fold 1

Round 2:
Train → Fold 1, 3, 4, 5
Test  → Fold 2

Round 3:
Train → Fold 1, 2, 4, 5
Test  → Fold 3

Round 4:
Train → Fold 1, 2, 3, 5
Test  → Fold 4

Round 5:
Train → Fold 1, 2, 3, 4
Test  → Fold 5
```

Finally, we calculate the **average score**.

### Simple Diagram

```text theme={null}
        Dataset
           ↓
 ┌────┬────┬────┬────┬────┐
 │ F1 │ F2 │ F3 │ F4 │ F5 │
 └────┴────┴────┴────┴────┘

Round 1 → [Test] [Train] [Train] [Train] [Train]
Round 2 → [Train] [Test] [Train] [Train] [Train]
Round 3 → [Train] [Train] [Test] [Train] [Train]
Round 4 → [Train] [Train] [Train] [Test] [Train]
Round 5 → [Train] [Train] [Train] [Train] [Test]
```

### Python Example

```python theme={null}
from sklearn.model_selection import KFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

model = DecisionTreeClassifier(random_state=42)

kf = KFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X, y, cv=kf)

print(scores)
print(scores.mean())
```

### Output

```text theme={null}
[0.9667 0.9667 0.9333 0.9333 0.9667]

0.9533
```

The model gets an average cross-validation accuracy of approximately:

```text theme={null}
95.33%
```

### Important Points

* `n_splits=5` → creates 5 folds.
* `shuffle=True` → shuffles the dataset before splitting.
* Every sample gets a chance to be in the validation set.
* The final score is usually the **average of all fold scores**.

### Remember

> **K-Fold → Divide data into K folds and use each fold for validation once.**

***

# 2. Stratified K-Fold

### Definition

**Stratified K-Fold** is similar to K-Fold, but it maintains approximately the **same class distribution in every fold**.

This is especially useful for **classification problems**.

### Why is it needed?

Suppose we have:

```text theme={null}
100 samples

Class 0 → 90
Class 1 → 10
```

If we randomly divide the data, one fold might contain:

```text theme={null}
Class 0 → 20
Class 1 → 0
```

That fold has **no examples of Class 1**, which can cause problems.

Stratified K-Fold tries to maintain the distribution:

```text theme={null}
Each fold:

Class 0 → ~18
Class 1 → ~2
```

### Simple Diagram

Suppose:

```text theme={null}
Class A → 80%
Class B → 20%
```

With Stratified K-Fold:

```text theme={null}
Fold 1 → 80% A + 20% B
Fold 2 → 80% A + 20% B
Fold 3 → 80% A + 20% B
Fold 4 → 80% A + 20% B
Fold 5 → 80% A + 20% B
```

So every fold has approximately the **same class ratio**.

### Python Example

```python theme={null}
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

model = DecisionTreeClassifier(random_state=42)

skf = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=42
)

scores = cross_val_score(
    model,
    X,
    y,
    cv=skf
)

print(scores)
print(scores.mean())
```

### Output

```text theme={null}
[0.9667 0.9667 0.9333 0.9667 0.9667]

0.96
```

The model's average accuracy is approximately:

```text theme={null}
96%
```

### Important Points

* Maintains **class proportions**.
* Mainly used for **classification**.
* Especially useful for **imbalanced datasets**.
* `StratifiedKFold` ensures each fold represents the classes properly.

### Remember

> **Stratified K-Fold → K-Fold + preserve class distribution.**

***

# K-Fold vs Stratified K-Fold

| Feature                       | K-Fold           | Stratified K-Fold |
| ----------------------------- | ---------------- | ----------------- |
| Divides data into K folds     | Yes              | Yes               |
| Each fold used for validation | Yes              | Yes               |
| Maintains class distribution  | No               | Yes               |
| Mainly useful for             | General problems | Classification    |
| Imbalanced classification     | Less suitable    | Better            |

### Easy Example

```text theme={null}
K-Fold
   ↓
Split data into K folds

Stratified K-Fold
   ↓
Split data into K folds
   +
Maintain class proportions
```

# Complete Cross-Validation Flow

```text theme={null}
              Dataset
                 ↓
        ┌─────────────────┐
        │ Cross-Validation              │
        └────────┬────────┘
                 ↓
       ┌─────────┴─────────┐
       ↓                   ↓
    K-Fold          Stratified K-Fold
       ↓                   ↓
   K folds          K folds + class
                    distribution
       ↓                   ↓
 Train/Validate      Train/Validate
 multiple times     multiple times
       ↓                   ↓
 Average Score       Average Score
```

### One-line summary

> **K-Fold divides data into K folds, while Stratified K-Fold divides it into K folds while preserving the class distribution in each fold.**