What is Hyperparameter Tuning?

Hyperparameter Tuning is the process of finding the best settings (hyperparameters) for a machine learning model. Hyperparameters are values that we set before training. Example for a Random Forest:

n_estimators = 100
max_depth = 10

We don’t know which values will give the best performance, so we try different combinations. Two common methods:

GridSearchCV
RandomizedSearchCV

1. GridSearchCV

Definition

GridSearchCV tries every possible combination of the hyperparameters we provide.

Example

Suppose we want to find the best:

n_estimators → 50, 100
max_depth    → 5, 10

GridSearchCV tries:

So there are:

2 × 2 = 4 combinations

Simple Code Example

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

model = RandomForestClassifier(random_state=42)

params = {
    "n_estimators": [50, 100],
    "max_depth": [3, 5]
}

grid = GridSearchCV(
    model,
    params,
    cv=5,
    scoring="accuracy"
)

grid.fit(X, y)

print(grid.best_params_)
print(grid.best_score_)

Output

{'max_depth': 3, 'n_estimators': 50}
0.9666666666666667

The exact best parameters can vary with the dataset and search space.

Important Terms

param_grid Contains the hyperparameter values to test.

params = {
    "n_estimators": [50, 100],
    "max_depth": [3, 5]
}

cv=5 Uses 5-fold cross-validation.

Data
 ↓
Fold 1 → Validation
Fold 2 → Validation
Fold 3 → Validation
Fold 4 → Validation
Fold 5 → Validation

The model is trained and evaluated multiple times. best_params_ Returns the best hyperparameter combination. best_score_ Returns the best cross-validation score.

Advantages

Tests every combination.
Can find the best combination within the given grid.
Easy to understand.

Disadvantages

Can become very slow when many parameters and values are provided.
Computationally expensive.

Remember

GridSearchCV = Try everything in the grid.

2. RandomizedSearchCV

Definition

RandomizedSearchCV randomly selects combinations from the given hyperparameter distributions. Unlike GridSearchCV, it does not test every combination.

Example

Suppose we have:

params = {
    "n_estimators": [50, 100, 150, 200],
    "max_depth": [3, 5, 10, 15]
}

There are:

4 × 4 = 16 combinations

Instead of trying all 16, we can tell RandomizedSearchCV:

n_iter = 5

It randomly tests only 5 combinations.

Code Example

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

model = RandomForestClassifier(random_state=42)

params = {
    "n_estimators": [50, 100, 150, 200],
    "max_depth": [3, 5, 10, 15]
}

random_search = RandomizedSearchCV(
    model,
    params,
    n_iter=5,
    cv=5,
    scoring="accuracy",
    random_state=42
)

random_search.fit(X, y)

print(random_search.best_params_)
print(random_search.best_score_)

Output

{'n_estimators': 150, 'max_depth': 5}
0.9666666666666667

The exact result may differ depending on the dataset and search space.

Important Terms

n_iter=5 Tests only 5 randomly selected combinations. random_state=42 Makes the random selection reproducible. best_params_ Returns the best combination found. best_score_ Returns the best cross-validation score.

Advantages

Faster than GridSearchCV for large search spaces.
Can explore many possible values.
Useful when there are many hyperparameters.

Disadvantages

May miss the actual best combination.
Results depend on the randomly selected combinations.

Remember

RandomizedSearchCV = Try some random combinations.

GridSearchCV vs RandomizedSearchCV

Feature	GridSearchCV	RandomizedSearchCV
Search	Every combination	Random combinations
Speed	Slower	Usually faster
Accuracy of search	Exhaustive	Approximate
Large search space	Not ideal	Better
Parameter control	Exact grid	Random sampling
Main parameter	`param_grid`	`param_distributions`
Number of tests	All combinations	`n_iter` combinations

Example

Suppose:

values for parameter A
values for parameter B
values for parameter C

Total combinations:

10 × 10 × 10 = 1000

GridSearchCV:

Tests all 1000

RandomizedSearchCV:

n_iter = 50

Tests only 50 random combinations

Complete Tuning Process

        Machine Learning Model
                 ↓
       Choose Hyperparameters
                 ↓
       ┌───────────────────┐
       │ GridSearchCV                       │
       │        OR                             │
       │ RandomizedSearchCV               │
       └─────────┬─────────┘
                 ↓
        Cross Validation
                 ↓
          Compare Scores
                 ↓
        Best Parameters
                 ↓
        Train Final Model

Easy way to remember

GridSearchCV
    ↓
"Try ALL combinations"

RandomizedSearchCV
    ↓
"Try SOME random combinations"

One-line summary

GridSearchCV exhaustively searches a specified grid, while RandomizedSearchCV randomly samples a fixed number of hyperparameter combinations.

Cross-Validation

What is Cross-Validation?

Cross-Validation is a technique used to evaluate how well a machine learning model performs on unseen data. Instead of splitting the dataset only once into training and testing data, we split the data multiple times and evaluate the model multiple times.

Dataset
   ↓
Split into multiple parts
   ↓
Train + Validate multiple times
   ↓
Average the scores
   ↓
Final performance

Two important methods:

K-Fold Cross-Validation
Stratified K-Fold Cross-Validation

1. K-Fold Cross-Validation

Definition

In K-Fold Cross-Validation, the dataset is divided into K equal or nearly equal parts, called folds. Each fold is used as the validation set once, while the remaining folds are used for training.

Example

Suppose:

K = 5

The dataset is divided into:

Fold 1
Fold 2
Fold 3
Fold 4
Fold 5

Then:

Round 1:
Train → Fold 2, 3, 4, 5
Test  → Fold 1

Round 2:
Train → Fold 1, 3, 4, 5
Test  → Fold 2

Round 3:
Train → Fold 1, 2, 4, 5
Test  → Fold 3

Round 4:
Train → Fold 1, 2, 3, 5
Test  → Fold 4

Round 5:
Train → Fold 1, 2, 3, 4
Test  → Fold 5

Finally, we calculate the average score.

Simple Diagram

        Dataset
           ↓
 ┌────┬────┬────┬────┬────┐
 │ F1 │ F2 │ F3 │ F4 │ F5 │
 └────┴────┴────┴────┴────┘

Round 1 → [Test] [Train] [Train] [Train] [Train]
Round 2 → [Train] [Test] [Train] [Train] [Train]
Round 3 → [Train] [Train] [Test] [Train] [Train]
Round 4 → [Train] [Train] [Train] [Test] [Train]
Round 5 → [Train] [Train] [Train] [Train] [Test]

Python Example

from sklearn.model_selection import KFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

model = DecisionTreeClassifier(random_state=42)

kf = KFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X, y, cv=kf)

print(scores)
print(scores.mean())

Output

[0.9667 0.9667 0.9333 0.9333 0.9667]

0.9533

The model gets an average cross-validation accuracy of approximately:

95.33%

Important Points

n_splits=5 → creates 5 folds.
shuffle=True → shuffles the dataset before splitting.
Every sample gets a chance to be in the validation set.
The final score is usually the average of all fold scores.

Remember

K-Fold → Divide data into K folds and use each fold for validation once.

2. Stratified K-Fold

Definition

Stratified K-Fold is similar to K-Fold, but it maintains approximately the same class distribution in every fold. This is especially useful for classification problems.

Why is it needed?

Suppose we have:

100 samples

Class 0 → 90
Class 1 → 10

If we randomly divide the data, one fold might contain:

Class 0 → 20
Class 1 → 0

That fold has no examples of Class 1, which can cause problems. Stratified K-Fold tries to maintain the distribution:

Each fold:

Class 0 → ~18
Class 1 → ~2

Simple Diagram

Suppose:

Class A → 80%
Class B → 20%

With Stratified K-Fold:

Fold 1 → 80% A + 20% B
Fold 2 → 80% A + 20% B
Fold 3 → 80% A + 20% B
Fold 4 → 80% A + 20% B
Fold 5 → 80% A + 20% B

So every fold has approximately the same class ratio.

Python Example

from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)

model = DecisionTreeClassifier(random_state=42)

skf = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=42
)

scores = cross_val_score(
    model,
    X,
    y,
    cv=skf
)

print(scores)
print(scores.mean())

Output

[0.9667 0.9667 0.9333 0.9667 0.9667]

0.96

The model’s average accuracy is approximately:

96%

Important Points

Maintains class proportions.
Mainly used for classification.
Especially useful for imbalanced datasets.
StratifiedKFold ensures each fold represents the classes properly.

Remember

Stratified K-Fold → K-Fold + preserve class distribution.

K-Fold vs Stratified K-Fold

Feature	K-Fold	Stratified K-Fold
Divides data into K folds	Yes	Yes
Each fold used for validation	Yes	Yes
Maintains class distribution	No	Yes
Mainly useful for	General problems	Classification
Imbalanced classification	Less suitable	Better

Easy Example

K-Fold
   ↓
Split data into K folds

Stratified K-Fold
   ↓
Split data into K folds
   +
Maintain class proportions

Complete Cross-Validation Flow

              Dataset
                 ↓
        ┌─────────────────┐
        │ Cross-Validation              │
        └────────┬────────┘
                 ↓
       ┌─────────┴─────────┐
       ↓                   ↓
    K-Fold          Stratified K-Fold
       ↓                   ↓
   K folds          K folds + class
                    distribution
       ↓                   ↓
 Train/Validate      Train/Validate
 multiple times     multiple times
       ↓                   ↓
 Average Score       Average Score

One-line summary

K-Fold divides data into K folds, while Stratified K-Fold divides it into K folds while preserving the class distribution in each fold.

​What is Hyperparameter Tuning?

​1. GridSearchCV

​Definition

​Example

​Simple Code Example

​Output

​Important Terms

​Advantages

​Disadvantages

​Remember

​2. RandomizedSearchCV

​Definition

​Example

​Code Example

​Output

​Important Terms

​Advantages

​Disadvantages

​Remember

​GridSearchCV vs RandomizedSearchCV

​Example

​Complete Tuning Process

​Easy way to remember

​One-line summary

​Cross-Validation

​What is Cross-Validation?

​1. K-Fold Cross-Validation

​Definition

​Example

​Simple Diagram

​Python Example

​Output

​Important Points

​Remember

​2. Stratified K-Fold

​Definition

​Why is it needed?

​Simple Diagram

​Python Example

​Output

​Important Points

​Remember

​K-Fold vs Stratified K-Fold

​Easy Example

​Complete Cross-Validation Flow

​One-line summary

What is Hyperparameter Tuning?

1. GridSearchCV

Definition

Example

Simple Code Example

Output

Important Terms

Advantages

Disadvantages

Remember

2. RandomizedSearchCV

Definition

Example

Code Example

Output

Important Terms

Advantages

Disadvantages

Remember

GridSearchCV vs RandomizedSearchCV

Example

Complete Tuning Process

Easy way to remember

One-line summary

Cross-Validation

What is Cross-Validation?

1. K-Fold Cross-Validation

Definition

Example

Simple Diagram

Python Example

Output

Important Points

Remember

2. Stratified K-Fold

Definition

Why is it needed?

Simple Diagram

Python Example

Output

Important Points

Remember

K-Fold vs Stratified K-Fold

Easy Example

Complete Cross-Validation Flow

One-line summary