1. What is PyTorch?

PyTorch is an open-source machine-learning and deep-learning framework. It provides tools for creating, training, evaluating, and deploying neural networks. PyTorch is especially popular because it provides:

Easy-to-use tensor operations
Automatic differentiation
GPU acceleration
Neural-network building blocks
Flexible training loops
Support for CNNs, RNNs, Transformers, and generative models
Strong research and production ecosystem

Main PyTorch components

PyTorch
│
├── Tensors
├── Autograd
├── torch.nn
├── Loss Functions
├── Optimizers
├── Dataset / DataLoader
├── CUDA / GPU
├── CNN
├── RNN / LSTM / GRU
├── Attention / Transformers
├── Model Saving
└── Deployment

Installation

pip install torch torchvision torchaudio

Check the installation:

import torch

print(torch.__version__)

Why learn PyTorch?

If you want to work in:

Deep Learning
Computer Vision
NLP
Generative AI
LLMs
Research
AI engineering

then PyTorch is one of the most important frameworks to understand.

2. Importing PyTorch

The basic import is:

import torch

For neural networks:

import torch.nn as nn

For optimizers:

import torch.optim as optim

For datasets and batching:

from torch.utils.data import Dataset, DataLoader

For computer vision:

import torchvision
import torchvision.transforms as transforms

What does `nn` mean?

torch.nn contains the building blocks used to construct neural networks. For example:

nn.Linear()
nn.ReLU()
nn.Conv2d()
nn.Dropout()
nn.BatchNorm1d()

3. What is a Tensor?

A tensor is the fundamental data structure in PyTorch. You can think of tensors as generalized arrays.

Scalar → 0-dimensional tensor
Vector → 1-dimensional tensor
Matrix → 2-dimensional tensor
3D Tensor → e.g. image
4D Tensor → e.g. batch of images

Scalar

x = torch.tensor(10)

print(x)
print(x.shape)

A scalar contains one value. Its shape is:

torch.Size([])

Vector

x = torch.tensor([1, 2, 3, 4])

print(x)
print(x.shape)

Shape:

[4]

Matrix

x = torch.tensor([
    [1, 2, 3],
    [4, 5, 6]
])

print(x)
print(x.shape)

Shape:

[2, 3]

That means:

2 rows
3 columns

Why are tensors important?

Almost everything in a neural network is represented using tensors:

Input data → Tensor
Weights → Tensor
Biases → Tensor
Prediction → Tensor
Loss → Tensor
Gradients → Tensor

Understanding tensor shape, dtype, and device is one of the most important PyTorch skills.

4. Creating Tensors

`torch.zeros()`

Creates a tensor filled with zeros.

x = torch.zeros(3, 4)

print(x)

This creates:

3 rows × 4 columns

Useful when initializing values.

`torch.ones()`

x = torch.ones(2, 3)

print(x)

Creates a tensor containing only ones.

`torch.rand()`

Creates random numbers between 0 and 1.

x = torch.rand(3, 3)

print(x)

`torch.randn()`

Generates random values approximately following a standard normal distribution.

x = torch.randn(3, 3)

print(x)

This is frequently used for testing neural networks and creating synthetic data.

`torch.randint()`

Creates random integers.

x = torch.randint(
    0,
    10,
    (3, 3)
)

print(x)

This generates values from 0 through 9.

`torch.arange()`

Creates a sequence.

x = torch.arange(0, 10)

print(x)

Output:

tensor([0, 1, 2, ..., 9])

5. Tensor Data Types

Every tensor has a data type.

x = torch.tensor([1, 2, 3])

print(x.dtype)

Common data types include:

torch.float32
torch.float64
torch.int32
torch.int64
torch.bool

For example:

x = torch.tensor(
    [1, 2, 3],
    dtype=torch.float32
)

print(x.dtype)

Why does dtype matter?

Neural networks normally perform calculations using floating-point numbers. For example:

x = torch.tensor([1, 2, 3])

x = x.float()

Now:

torch.int64 → torch.float32

For classification labels, however, torch.int64 is commonly required by CrossEntropyLoss. So dtype depends on what the tensor represents.

6. Tensor Shape

Shape tells us the dimensions of a tensor.

x = torch.randn(2, 3, 4)

print(x.shape)
print(x.ndim)

Output:

torch.Size([2, 3, 4])
3

This means:

dimension 1 = 2
dimension 2 = 3
dimension 3 = 4

Number of elements

print(x.numel())

For a [2, 3, 4] tensor:

2 × 3 × 4 = 24

So numel() returns 24.

7. Tensor Indexing

PyTorch indexing works similarly to NumPy.

x = torch.tensor([
    [10, 20, 30],
    [40, 50, 60]
])

First row:

print(x[0])

Output:

tensor([10, 20, 30])

Specific element:

print(x[0, 1])

Output:

tensor(20)

Slicing

print(x[:, 1])

The : means “all rows”. So this means:

all rows, column 1

Result:

tensor([20, 50])

8. Tensor Arithmetic

Given:

a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])

Addition:

print(a + b)

Subtraction:

print(a - b)

Element-wise multiplication:

print(a * b)

Division:

print(a / b)

Power:

print(a ** 2)

Important distinction:

a * b

means element-wise multiplication. It does NOT mean matrix multiplication.

9. Matrix Multiplication

Consider:

a = torch.tensor([
    [1, 2],
    [3, 4]
])

b = torch.tensor([
    [5, 6],
    [7, 8]
])

Element-wise multiplication:

a * b

gives:

[[5, 12],
 [21, 32]]

Matrix multiplication:

torch.matmul(a, b)

or:

a @ b

gives:

[[19, 22],
 [43, 50]]

Important interview question

What is the difference between:

a * b

and:

a @ b

Answer:

*  → element-wise multiplication
@  → matrix multiplication

10. Useful Tensor Functions

PyTorch provides many mathematical operations.

x = torch.tensor([1., 2., 3., 4.])

print(torch.sum(x))
print(torch.mean(x))
print(torch.max(x))
print(torch.min(x))

Other useful operations:

torch.abs(x)
torch.sqrt(x)
torch.exp(x)
torch.log(x)
torch.argmax(x)
torch.argmin(x)

`argmax()`

argmax() returns the index of the largest value.

x = torch.tensor([10., 30., 20.])

index = torch.argmax(x)

print(index)
print(x[index])

Result:

index = 1
value = 30

This is frequently used for classification predictions.

11. Reshaping Tensors

Neural networks often require data in a particular shape. Suppose:

x = torch.arange(12)

Current shape:

[12]

We can reshape it:

x = x.reshape(3, 4)

print(x)

Now:

[3, 4]

The number of elements must remain the same:

12 = 3 × 4

`view()`

Another way:

x = torch.arange(12)

x = x.view(3, 4)

view() has stricter memory-layout requirements than reshape(), so reshape() is often the more convenient choice.

`flatten()`

x = torch.randn(2, 3, 4)

x = x.flatten()

print(x.shape)

This converts:

[2, 3, 4]

into:

[24]

Flattening is particularly important when connecting CNN features to fully connected layers.

12. `unsqueeze()` and `squeeze()`

These operations add and remove dimensions. Suppose:

x = torch.tensor([1, 2, 3])

print(x.shape)

Shape:

[3]

Add a dimension:

x = x.unsqueeze(0)

print(x.shape)

Now:

[1, 3]

This is useful when a model expects a batch dimension. For example:

Single sample:
[3]

Batch containing one sample:
[1, 3]

Remove a dimension:

x = x.squeeze()

print(x.shape)

13. Concatenation and Stacking

`torch.cat()`

Concatenates tensors along an existing dimension.

a = torch.tensor([1, 2])
b = torch.tensor([3, 4])

x = torch.cat([a, b])

print(x)

Result:

tensor([1, 2, 3, 4])

`torch.stack()`

Creates a new dimension.

x = torch.stack([a, b])

print(x)
print(x.shape)

Result shape:

[2, 2]

Difference

cat
 ↓
joins existing dimensions

stack
 ↓
creates a new dimension

14. NumPy and PyTorch

PyTorch works closely with NumPy. NumPy → PyTorch:

import numpy as np
import torch

arr = np.array([1, 2, 3])

x = torch.from_numpy(arr)

print(x)

PyTorch → NumPy:

arr = x.numpy()

print(arr)

When possible, these objects can share the same underlying memory. Therefore, changing one can affect the other.

15. CPU and GPU

Deep-learning training can be much faster on GPUs. Check whether CUDA is available:

print(torch.cuda.is_available())

Select a device:

device = torch.device(
    "cuda" if torch.cuda.is_available()
    else "cpu"
)

print(device)

Move a tensor:

x = torch.randn(3, 3)

x = x.to(device)

Move the model:

model = model.to(device)

During training, both inputs and model parameters must generally be on compatible devices:

X = X.to(device)
y = y.to(device)

Important rule

You cannot normally do:

Model → GPU
Input → CPU

and expect the operation to work. Both need to be on compatible devices.

16. Autograd

One of PyTorch’s most important features is automatic differentiation. Suppose:

y = x²

Mathematically:

dy/dx = 2x

If:

x = 2

then:

dy/dx = 4

PyTorch can calculate this automatically.

x = torch.tensor(
    2.0,
    requires_grad=True
)

y = x ** 2

y.backward()

print(x.grad)

Output:

tensor(4.)

What happened?

x
↓
x²
↓
y
↓
backward()
↓
gradient stored in x.grad

This mechanism is the foundation of neural-network training.

17. Gradient Descent

Suppose our model is:

y = wx + b

The model has parameters:

w
b

During training:

Input
 ↓
Model
 ↓
Prediction
 ↓
Loss
 ↓
Gradients
 ↓
Parameter update
 ↓
Repeat

Example:

x = torch.tensor(2.0)

w = torch.tensor(
    1.0,
    requires_grad=True
)

y = w * x

loss = (y - 10) ** 2

loss.backward()

print(w.grad)

The gradient tells us how changing w would affect the loss. The optimizer uses this information to update w.

18. Why `zero_grad()`?

PyTorch gradients accumulate by default. For example:

x = torch.tensor(
    2.0,
    requires_grad=True
)

y = x ** 2
y.backward()

print(x.grad)

y = x ** 2
y.backward()

print(x.grad)

The second backward pass adds to the existing gradient. Therefore, during normal training we use:

optimizer.zero_grad()

The standard sequence is:

optimizer.zero_grad()

loss.backward()

optimizer.step()

Meaning:

zero_grad()
    ↓
remove old gradients

backward()
    ↓
calculate new gradients

step()
    ↓
update parameters

19. `nn.Module`

Neural networks normally inherit from:

nn.Module

Example:

import torch
import torch.nn as nn

class SimpleModel(nn.Module):

    def __init__(self):
        super().__init__()

        self.linear = nn.Linear(2, 1)

    def forward(self, x):
        return self.linear(x)

Create the model:

model = SimpleModel()

print(model)

Two important methods

__init__() defines the layers.

def __init__(self):

forward() defines how data flows through the model.

def forward(self, x):

When you write:

output = model(x)

PyTorch internally calls the model’s forward logic.

20. `nn.Linear`

A linear layer performs:

y = xWᵀ + b

Example:

layer = nn.Linear(
    in_features=3,
    out_features=2
)

x = torch.randn(5, 3)

y = layer(x)

print(y.shape)

Output:

[5, 2]

Why?

= batch size
= input features
= output features

So:

[5, 3]
    ↓
Linear(3, 2)
    ↓
[5, 2]

21. Activation Functions

A neural network containing only linear operations is still effectively a linear transformation. Activation functions introduce non-linearity.

ReLU

relu = nn.ReLU()

x = torch.tensor([
    -2., -1., 0., 1., 2.
])

print(relu(x))

Mathematically:

ReLU(x) = max(0, x)

Therefore:

-2 → 0
-1 → 0
 0 → 0
 1 → 1
 2 → 2

ReLU is one of the most commonly used hidden-layer activations.

Sigmoid

sigmoid = nn.Sigmoid()

x = torch.tensor([
    -2., 0., 2.
])

print(sigmoid(x))

Output values lie between:

0 and 1

It is commonly useful for binary probabilities, although modern PyTorch training often uses BCEWithLogitsLoss directly on logits rather than explicitly applying sigmoid before the loss.

Tanh

tanh = nn.Tanh()

print(tanh(x))

Output range:

-1 to 1

Softmax

Softmax converts logits into a probability distribution across classes.

softmax = nn.Softmax(dim=1)

x = torch.tensor([
    [1., 2., 3.]
])

print(softmax(x))

The probabilities approximately sum to:

However, when using:

nn.CrossEntropyLoss()

you normally pass the raw logits directly rather than applying Softmax first.

22. Building a Neural Network

class NeuralNetwork(nn.Module):

    def __init__(self):
        super().__init__()

        self.network = nn.Sequential(
            nn.Linear(4, 16),
            nn.ReLU(),

            nn.Linear(16, 8),
            nn.ReLU(),

            nn.Linear(8, 2)
        )

    def forward(self, x):
        return self.network(x)

The architecture is:

4 input features
      ↓
16 neurons
      ↓
ReLU
      ↓
8 neurons
      ↓
ReLU
      ↓
2 outputs

The final 2 might represent two output classes.

23. `nn.Sequential`

nn.Sequential allows you to define layers in order. Instead of:

x = self.layer1(x)
x = self.relu(x)
x = self.layer2(x)

you can write:

self.layers = nn.Sequential(
    nn.Linear(10, 32),
    nn.ReLU(),
    nn.Linear(32, 16),
    nn.ReLU(),
    nn.Linear(16, 3)
)

Then:

def forward(self, x):
    return self.layers(x)

It is excellent for simple feed-forward architectures. For complicated architectures with branches or multiple inputs, explicit forward() logic is usually preferable.

24. Loss Functions

A loss function measures how different the model prediction is from the target. Conceptually:

Prediction
     ↓
Loss Function
     ↓
How wrong?

The optimizer then uses the gradient of this loss to improve the model.

MSE Loss

Mean Squared Error is commonly used for regression.

loss_fn = nn.MSELoss()

prediction = torch.tensor([
    2., 4., 6.
])

target = torch.tensor([
    1., 5., 6.
])

loss = loss_fn(
    prediction,
    target
)

print(loss)

Conceptually:

MSE = average((prediction - target)²)

Cross Entropy

Used very commonly for multi-class classification.

loss_fn = nn.CrossEntropyLoss()

logits = torch.tensor([
    [2.0, 1.0, 0.1],
    [0.2, 0.3, 2.0]
])

targets = torch.tensor([
    0,
    2
])

loss = loss_fn(
    logits,
    targets
)

Important:

CrossEntropyLoss expects raw logits.

Do not normally do:

softmax(logits)

before passing them to CrossEntropyLoss.

BCEWithLogitsLoss

For binary classification:

loss_fn = nn.BCEWithLogitsLoss()

This combines sigmoid behavior with binary cross entropy in a numerically stable way. Therefore it is generally preferred over:

Sigmoid()
+
BCELoss()

as a training setup.

25. Optimizers

The optimizer changes the model parameters using gradients. Common optimizers:

SGD
Adam
AdamW
RMSprop

SGD

optimizer = torch.optim.SGD(
    model.parameters(),
    lr=0.01
)

Adam

optimizer = torch.optim.Adam(
    model.parameters(),
    lr=0.001
)

AdamW

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=0.001,
    weight_decay=0.01
)

What is learning rate?

Learning rate controls how large each parameter update is. Very large:

Updates may become unstable.

Very small:

Training may become extremely slow.

The learning rate is one of the most important hyperparameters.

26. Complete Training Loop

The training loop is arguably the most important PyTorch pattern to understand.

for epoch in range(10):

    model.train()

    for X, y in train_loader:

        X = X.to(device)
        y = y.to(device)

        predictions = model(X)

        loss = loss_fn(
            predictions,
            y
        )

        optimizer.zero_grad()

        loss.backward()

        optimizer.step()

Understand every line:

1. `model.train()`

model.train()

Tells PyTorch that the model is in training mode. This matters for layers such as:

Dropout
BatchNorm

2. Move data

X = X.to(device)
y = y.to(device)

Moves data to CPU or GPU.

3. Forward pass

predictions = model(X)

The model generates predictions.

4. Calculate loss

loss = loss_fn(predictions, y)

Measures prediction error.

5. Clear old gradients

optimizer.zero_grad()

6. Backpropagation

loss.backward()

Calculates gradients.

7. Update parameters

optimizer.step()

Updates model weights. The entire process is:

Data
 ↓
Model
 ↓
Prediction
 ↓
Loss
 ↓
zero_grad()
 ↓
backward()
 ↓
step()
 ↓
Updated model

27. Dataset

A Dataset defines how your data is accessed.

from torch.utils.data import Dataset

class MyDataset(Dataset):

    def __init__(self):

        self.X = torch.tensor([
            [1., 2.],
            [3., 4.],
            [5., 6.]
        ])

        self.y = torch.tensor([
            0,
            1,
            0
        ])

    def __len__(self):
        return len(self.X)

    def __getitem__(self, index):
        return self.X[index], self.y[index]

There are three important parts.

`init()`

Stores or prepares the data.

`len()`

Returns the number of samples.

`getitem()`

Returns one sample. For example:

dataset = MyDataset()

print(len(dataset))
print(dataset[0])

28. DataLoader

A Dataset gives individual samples. A DataLoader creates batches.

from torch.utils.data import DataLoader

loader = DataLoader(
    dataset,
    batch_size=2,
    shuffle=True
)

Then:

for X, y in loader:

    print(X)
    print(y)

Why use DataLoader?

Instead of processing:

sample
sample
sample
...

we process:

batch
batch
batch
...

This makes training more efficient. Important parameters include:

DataLoader(
    dataset,
    batch_size=32,
    shuffle=True,
    num_workers=2
)

29. Training, Validation and Test Sets

A common structure is:

Dataset
│
├── Training
├── Validation
└── Test

Training set

Used to update model parameters.

Validation set

Used during development to make decisions such as:

Which model is better?
Which hyperparameters should we use?
When should training stop?

Test set

Used for final evaluation. Example:

from torch.utils.data import random_split

train_size = int(
    0.8 * len(dataset)
)

test_size = (
    len(dataset) - train_size
)

train_dataset, test_dataset = random_split(
    dataset,
    [train_size, test_size]
)

For real projects, validation should usually be kept separate from the final test set.

30. Evaluation

During evaluation:

model.eval()

Then disable gradient calculation:

with torch.no_grad():

    predictions = model(X)

Why? During inference we don’t need gradients. This saves:

Memory
Computation
Time

31. Classification Accuracy

Example:

model.eval()

correct = 0
total = 0

with torch.no_grad():

    for X, y in test_loader:

        X = X.to(device)
        y = y.to(device)

        outputs = model(X)

        predictions = outputs.argmax(
            dim=1
        )

        correct += (
            predictions == y
        ).sum().item()

        total += y.size(0)

accuracy = correct / total

print("Accuracy:", accuracy)

Suppose:

95 predictions correct
100 total predictions

Then:

accuracy = 95 / 100
         = 0.95
         = 95%

32. `model.train()` vs `model.eval()`

This distinction is extremely important. Training:

model.train()

Evaluation:

model.eval()

Why does this matter? Because some layers behave differently during training and evaluation. Most importantly:

Dropout
Batch Normalization

For example, Dropout randomly removes activations during training but should not randomly remove them during evaluation.

33. Dropout

Dropout is a regularization technique. It randomly disables some activations during training.

nn.Dropout(0.5)

Approximately 50% of eligible activations are dropped during training. Example:

class Model(nn.Module):

    def __init__(self):
        super().__init__()

        self.network = nn.Sequential(
            nn.Linear(10, 64),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(64, 2)
        )

    def forward(self, x):
        return self.network(x)

Why use Dropout?

It can reduce overfitting. Without regularization:

Training performance → very high
Validation performance → poor

This suggests the model may be memorizing training data.

34. Batch Normalization

Batch normalization normalizes activations using batch statistics during training. Example:

self.network = nn.Sequential(
    nn.Linear(10, 64),
    nn.BatchNorm1d(64),
    nn.ReLU(),
    nn.Linear(64, 2)
)

It can help stabilize optimization and sometimes speed up training. During evaluation, BatchNorm uses stored running statistics, which is another reason model.eval() matters.

35. Weight Initialization

Neural-network layers receive default parameter initialization from PyTorch. You can customize it. Xavier initialization:

layer = nn.Linear(10, 5)

nn.init.xavier_uniform_(
    layer.weight
)

nn.init.zeros_(
    layer.bias
)

Kaiming initialization:

nn.init.kaiming_normal_(
    layer.weight,
    nonlinearity="relu"
)

Good initialization can help optimization, especially in deeper networks.

36. CNN — Convolutional Neural Network

CNNs are designed particularly for spatial data such as images. Typical architecture:

Image
 ↓
Convolution
 ↓
ReLU
 ↓
Pooling
 ↓
Convolution
 ↓
ReLU
 ↓
Pooling
 ↓
Flatten
 ↓
Linear
 ↓
Prediction

A CNN learns features progressively. Early layers may learn:

Edges

Middle layers may learn:

Textures / shapes

Deeper layers may learn:

Objects / semantic patterns

37. `Conv2d`

Example:

conv = nn.Conv2d(
    in_channels=3,
    out_channels=32,
    kernel_size=3,
    stride=1,
    padding=1
)

For an RGB image:

in_channels = 3

The 32 output channels mean the layer learns 32 filters. Input:

x = torch.randn(
    8, 3, 32, 32
)

Meaning:

→ batch size
→ channels
→ height
→ width

38. Pooling

Max pooling reduces spatial dimensions.

pool = nn.MaxPool2d(
    kernel_size=2
)

x = torch.randn(
    8, 32, 32, 32
)

y = pool(x)

print(y.shape)

Spatial dimensions change:

32 × 32
    ↓
16 × 16

Pooling reduces spatial resolution and can make representations more compact.

39. Complete CNN

class CNN(nn.Module):

    def __init__(self):
        super().__init__()

        self.features = nn.Sequential(

            nn.Conv2d(
                3, 32,
                kernel_size=3,
                padding=1
            ),

            nn.ReLU(),

            nn.MaxPool2d(2),

            nn.Conv2d(
                32, 64,
                kernel_size=3,
                padding=1
            ),

            nn.ReLU(),

            nn.MaxPool2d(2)
        )

        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(
                64 * 8 * 8,
                10
            )
        )

    def forward(self, x):

        x = self.features(x)

        return self.classifier(x)

If the input is:

3 × 32 × 32

after two 2×2 pooling operations:

So the feature map becomes:

64 × 8 × 8

That explains:

nn.Linear(64 * 8 * 8, 10)

40. Image Dataset with Torchvision

PyTorch provides datasets through torchvision. Example using MNIST:

from torchvision import datasets
from torchvision import transforms

transform = transforms.ToTensor()

train_dataset = datasets.MNIST(
    root="./data",
    train=True,
    download=True,
    transform=transform
)

Create a DataLoader:

train_loader = DataLoader(
    train_dataset,
    batch_size=64,
    shuffle=True
)

41. Image Transformations

Transforms preprocess and augment images. Convert image to tensor:

transforms.ToTensor()

Resize:

transforms.Resize(
    (224, 224)
)

Normalize:

transforms.Normalize(
    mean=[0.485, 0.456, 0.406],
    std=[0.229, 0.224, 0.225]
)

Data augmentation:

transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(10),
    transforms.ToTensor()
])

Why augmentation?

Instead of showing the model exactly the same images repeatedly, we create slightly modified versions. Examples:

Original image
      ↓
Flip
Rotate
Crop
Resize
Color changes

This can improve generalization.

42. Transfer Learning

Training a large image model from scratch requires a lot of data and computation. Transfer learning starts with a model that has already learned useful visual features. Example:

from torchvision.models import (
    resnet18,
    ResNet18_Weights
)

model = resnet18(
    weights=ResNet18_Weights.DEFAULT
)

Replace the final layer:

model.fc = nn.Linear(
    model.fc.in_features,
    10
)

If your dataset has 10 classes, the final layer needs 10 outputs.

Freezing the pretrained layers

for param in model.parameters():
    param.requires_grad = False

Then enable training for the classifier:

for param in model.fc.parameters():
    param.requires_grad = True

This is called feature extraction. Later, you can optionally unfreeze some pretrained layers and fine-tune them.

43. RNN

RNN stands for Recurrent Neural Network. RNNs process sequential information. Examples:

Text
Speech
Time series
Sensor data

Example:

rnn = nn.RNN(
    input_size=10,
    hidden_size=20,
    batch_first=True
)

x = torch.randn(
    32,
    5,
    10
)

output, hidden = rnn(x)

print(output.shape)

Shape:

[batch, sequence, features]

Here:

→ batch
→ sequence length
→ input features

44. LSTM

LSTM means Long Short-Term Memory. It was designed to better preserve useful information over longer sequences than a basic RNN.

lstm = nn.LSTM(
    input_size=10,
    hidden_size=32,
    num_layers=2,
    batch_first=True
)

x = torch.randn(
    16,
    20,
    10
)

output, (hidden, cell) = lstm(x)

print(output.shape)

LSTM maintains:

Hidden state
+
Cell state

These allow information to persist through the sequence.

45. GRU

GRU means Gated Recurrent Unit. It is another recurrent architecture.

gru = nn.GRU(
    input_size=10,
    hidden_size=32,
    batch_first=True
)

x = torch.randn(
    16,
    20,
    10
)

output, hidden = gru(x)

print(output.shape)

Compared with LSTM, GRU has a simpler gating structure and does not maintain a separate cell state.

46. Transformer

Transformers became fundamental to modern NLP and generative AI. A Transformer uses attention to determine which parts of a sequence are important to each other. Example:

encoder_layer = nn.TransformerEncoderLayer(
    d_model=512,
    nhead=8,
    batch_first=True
)

transformer = nn.TransformerEncoder(
    encoder_layer,
    num_layers=6
)

x = torch.randn(
    32,
    100,
    512
)

output = transformer(x)

print(output.shape)

Conceptually:

Input
 ↓
Embedding
 ↓
Positional Information
 ↓
Self-Attention
 ↓
Feed Forward Network
 ↓
Normalization
 ↓
Output

Transformers are the foundation of many modern language models.

47. Attention

Attention answers a basic question:

“Which other pieces of information should this token pay attention to?”

The mathematical form is:

Attention(Q,K,V)
=
softmax(QKᵀ / √d)V

Where:

Q = Query
K = Key
V = Value

PyTorch implementation:

attention = nn.MultiheadAttention(
    embed_dim=512,
    num_heads=8,
    batch_first=True
)

x = torch.randn(
    32,
    100,
    512
)

output, weights = attention(
    x, x, x
)

print(output.shape)

When:

Q = K = V

this is self-attention.

48. Embeddings

Neural networks cannot directly understand words such as:

cat
dog
computer

They are represented as token IDs. For example:

cat → 17
dog → 42

An embedding converts these IDs into dense vectors.

embedding = nn.Embedding(
    num_embeddings=10000,
    embedding_dim=128
)

tokens = torch.tensor([
    [1, 20, 300],
    [4, 50, 600]
])

output = embedding(tokens)

print(output.shape)

Shape:

[2, 3, 128]

Meaning:

2 sentences
3 tokens per sentence
128-dimensional embedding

49. Simple NLP Model

class TextClassifier(nn.Module):

    def __init__(
        self,
        vocab_size,
        embed_dim,
        num_classes
    ):
        super().__init__()

        self.embedding = nn.Embedding(
            vocab_size,
            embed_dim
        )

        self.fc = nn.Linear(
            embed_dim,
            num_classes
        )

    def forward(self, x):

        x = self.embedding(x)

        x = x.mean(dim=1)

        return self.fc(x)

The flow is:

Token IDs
 ↓
Embedding
 ↓
Average token vectors
 ↓
Linear layer
 ↓
Class prediction

This is a simple educational model, not a modern LLM architecture.

50. Model Parameters

Every neural network contains parameters such as weights and biases. View them:

for name, param in model.named_parameters():

    print(name)
    print(param.shape)

Count all parameters:

total = sum(
    p.numel()
    for p in model.parameters()
)

print(total)

Count only trainable parameters:

trainable = sum(
    p.numel()
    for p in model.parameters()
    if p.requires_grad
)

print(trainable)

This is particularly useful when comparing model sizes.

51. Saving a Model

The recommended approach is usually to save the model’s state_dict.

torch.save(
    model.state_dict(),
    "model.pth"
)

Later:

model = NeuralNetwork()

model.load_state_dict(
    torch.load(
        "model.pth",
        weights_only=True
    )
)

model.eval()

What is `state_dict()`?

It is essentially a dictionary containing the model’s learned parameters and buffers.

52. Saving a Checkpoint

If training takes hours or days, save checkpoints.

checkpoint = {
    "epoch": epoch,
    "model_state": model.state_dict(),
    "optimizer_state": optimizer.state_dict(),
    "loss": loss.item()
}

torch.save(
    checkpoint,
    "checkpoint.pth"
)

Load:

checkpoint = torch.load(
    "checkpoint.pth",
    weights_only=False
)

model.load_state_dict(
    checkpoint["model_state"]
)

optimizer.load_state_dict(
    checkpoint["optimizer_state"]
)

epoch = checkpoint["epoch"]

A checkpoint allows you to continue training instead of starting from zero.

53. Learning Rate Scheduler

A learning-rate scheduler changes the learning rate during training. Example:

optimizer = torch.optim.Adam(
    model.parameters(),
    lr=0.001
)

scheduler = torch.optim.lr_scheduler.StepLR(
    optimizer,
    step_size=10,
    gamma=0.1
)

Then:

for epoch in range(30):

    # training code

    scheduler.step()

Conceptually:

Initial LR
   ↓
Train
   ↓
Reduce LR
   ↓
Train
   ↓
Reduce LR

Other schedulers include:

CosineAnnealingLR
ReduceLROnPlateau
OneCycleLR
LinearLR

54. Early Stopping

Sometimes validation performance stops improving. Instead of continuing forever, we can stop training. Concept:

Validation loss improves
        ↓
Continue training

Validation loss does not improve
        ↓
Increase patience counter

Counter reaches patience
        ↓
Stop training

Example:

best_loss = float("inf")

patience = 3
counter = 0

for epoch in range(100):

    val_loss = validate()

    if val_loss < best_loss:

        best_loss = val_loss
        counter = 0

        torch.save(
            model.state_dict(),
            "best_model.pth"
        )

    else:

        counter += 1

        if counter >= patience:
            print("Early stopping")
            break

The important idea is to save the best model, not necessarily the model from the final epoch.

55. Mixed Precision

Modern GPUs can perform some operations efficiently using lower-precision numerical formats. Mixed precision can provide:

Faster training
Lower GPU memory usage
Better hardware utilization

Example:

from torch.amp import autocast, GradScaler

scaler = GradScaler("cuda")

for X, y in train_loader:

    X = X.to(device)
    y = y.to(device)

    optimizer.zero_grad()

    with autocast("cuda"):

        output = model(X)

        loss = loss_fn(
            output,
            y
        )

    scaler.scale(
        loss
    ).backward()

    scaler.step(
        optimizer
    )

    scaler.update()

This is especially useful for large neural networks and GPU training.

56. Gradient Clipping

Sometimes gradients become extremely large. This is called exploding gradients. Gradient clipping limits their magnitude.

loss.backward()

torch.nn.utils.clip_grad_norm_(
    model.parameters(),
    max_norm=1.0
)

optimizer.step()

It is particularly useful in some recurrent-network training scenarios.

57. Freezing Parameters

Sometimes you don’t want to train every layer.

for param in model.parameters():
    param.requires_grad = False

Now those parameters will not receive normal gradient-based updates. To train a specific layer:

for param in model.fc.parameters():
    param.requires_grad = True

This is heavily used in transfer learning and fine-tuning.

58. `torch.no_grad()`

For inference:

model.eval()

with torch.no_grad():

    output = model(X)

no_grad() tells autograd that gradients are not needed. Benefits:

Less memory
Less computation
Faster inference

Remember:

model.eval()

and:

with torch.no_grad():

serve different purposes. eval() changes model behavior. no_grad() disables gradient tracking. You commonly use both during inference.

59. `detach()`

Suppose:

x = torch.tensor(
    2.0,
    requires_grad=True
)

y = x ** 2

z = y.detach()

print(z.requires_grad)

Output:

False

detach() creates a tensor that is disconnected from the current computation graph. For example:

output = model(X)

output = (
    output
    .detach()
    .cpu()
    .numpy()
)

This is useful when you want to take model outputs outside PyTorch’s gradient computation.

60. Random Seeds

For reproducibility:

torch.manual_seed(42)

For CUDA:

torch.cuda.manual_seed(42)

However, a seed does not automatically guarantee perfect reproducibility in every environment. Results can also depend on:

GPU hardware
CUDA version
Backend algorithms
Parallelism
Other libraries

61. Classification Example

import torch
import torch.nn as nn
import torch.optim as optim

class Classifier(nn.Module):

    def __init__(self):
        super().__init__()

        self.model = nn.Sequential(
            nn.Linear(4, 16),
            nn.ReLU(),
            nn.Linear(16, 8),
            nn.ReLU(),
            nn.Linear(8, 3)
        )

    def forward(self, x):
        return self.model(x)


model = Classifier()

loss_fn = nn.CrossEntropyLoss()

optimizer = optim.Adam(
    model.parameters(),
    lr=0.001
)

Training:

for epoch in range(100):

    X = torch.randn(32, 4)

    y = torch.randint(
        0,
        3,
        (32,)
    )

    output = model(X)

    loss = loss_fn(
        output,
        y
    )

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()

    if epoch % 10 == 0:

        print(
            epoch,
            loss.item()
        )

The model receives:

32 samples
4 features each

and predicts one of:

3 classes

62. Regression Example

Regression predicts continuous values. Suppose:

y = 3x + 2

We can create synthetic training data:

X = torch.randn(100, 1)

y = 3 * X + 2

Model:

model = nn.Linear(1, 1)

Loss:

loss_fn = nn.MSELoss()

Optimizer:

optimizer = torch.optim.SGD(
    model.parameters(),
    lr=0.01
)

Training:

for epoch in range(1000):

    prediction = model(X)

    loss = loss_fn(
        prediction,
        y
    )

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()

    if epoch % 100 == 0:

        print(
            epoch,
            loss.item()
        )

The model should learn parameters close to:

weight ≈ 3
bias ≈ 2

63. Binary Classification

For binary classification, the model can produce one logit per sample.

class BinaryClassifier(nn.Module):

    def __init__(self):
        super().__init__()

        self.network = nn.Sequential(
            nn.Linear(4, 16),
            nn.ReLU(),
            nn.Linear(16, 1)
        )

    def forward(self, x):
        return self.network(x)

Loss:

loss_fn = nn.BCEWithLogitsLoss()

Training:

logits = model(X)

loss = loss_fn(
    logits,
    y.float().unsqueeze(1)
)

Convert logits to probabilities:

probability = torch.sigmoid(logits)

Convert probabilities to binary predictions:

prediction = (
    probability > 0.5
).float()

Important distinction:

Logit
 ↓ sigmoid
Probability
 ↓ threshold
Class prediction

64. Multi-Class Classification

Suppose the classes are:

Cat
Dog
Horse

Then the output layer needs three outputs:

nn.Linear(
    hidden_size,
    3
)

The model returns logits:

output = model(X)

Choose the class with the highest logit:

prediction = output.argmax(
    dim=1
)

Loss:

loss_fn = nn.CrossEntropyLoss()

Targets should normally be class indices:

tensor([
    0,
    2,
    1,
    0
])

rather than one-hot vectors.

65. Autoencoder

An autoencoder learns to reconstruct its input. Architecture:

Input
 ↓
Encoder
 ↓
Latent Representation
 ↓
Decoder
 ↓
Reconstructed Input

Example:

class Autoencoder(nn.Module):

    def __init__(self):
        super().__init__()

        self.encoder = nn.Sequential(
            nn.Linear(784, 128),
            nn.ReLU(),
            nn.Linear(128, 32)
        )

        self.decoder = nn.Sequential(
            nn.Linear(32, 128),
            nn.ReLU(),
            nn.Linear(128, 784),
            nn.Sigmoid()
        )

    def forward(self, x):

        latent = self.encoder(x)

        reconstruction = (
            self.decoder(latent)
        )

        return reconstruction

Loss:

loss_fn = nn.MSELoss()

The model attempts to make:

reconstruction ≈ original input

66. GAN Basics

GAN means Generative Adversarial Network. It contains two networks:

Generator
    ↓
Fake Data
    ↓
Discriminator
    ↓
Real or Fake?

The Generator tries to create convincing fake samples. The Discriminator tries to distinguish real samples from generated samples. Generator:

class Generator(nn.Module):

    def __init__(self):
        super().__init__()

        self.model = nn.Sequential(
            nn.Linear(100, 256),
            nn.ReLU(),
            nn.Linear(256, 784),
            nn.Tanh()
        )

    def forward(self, z):
        return self.model(z)

Discriminator:

class Discriminator(nn.Module):

    def __init__(self):
        super().__init__()

        self.model = nn.Sequential(
            nn.Linear(784, 256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, 1)
        )

    def forward(self, x):
        return self.model(x)

GAN training is more complicated than ordinary supervised training because two networks are optimized against each other.

67. Custom Loss Function

You can create your own loss.

class MyLoss(nn.Module):

    def __init__(self):
        super().__init__()

    def forward(
        self,
        prediction,
        target
    ):

        return torch.mean(
            (prediction - target) ** 2
        )

Use it:

loss_fn = MyLoss()

loss = loss_fn(
    prediction,
    target
)

This is useful when the problem requires a specialized objective.

68. Custom Layer

PyTorch also allows custom neural-network layers.

class MyLayer(nn.Module):

    def __init__(
        self,
        input_size,
        output_size
    ):
        super().__init__()

        self.weight = nn.Parameter(
            torch.randn(
                input_size,
                output_size
            )
        )

        self.bias = nn.Parameter(
            torch.zeros(output_size)
        )

    def forward(self, x):

        return (
            x @ self.weight
            + self.bias
        )

Why `nn.Parameter`?

nn.Parameter tells PyTorch:

“This tensor is a learnable model parameter.”

Therefore it appears in:

model.parameters()

and can be updated by the optimizer.

69. Hooks

Hooks allow you to inspect intermediate values. Example:

def hook_fn(
    module,
    input,
    output
):

    print(
        "Output shape:",
        output.shape
    )

handle = model.fc.register_forward_hook(
    hook_fn
)

Run the model:

output = model(X)

The hook executes automatically. Remove it:

handle.remove()

Hooks are useful for:

Debugging
Inspecting activations
Visualizing intermediate layers
Model analysis

70. Profiling

When a model is slow, profiling helps identify bottlenecks. Example:

with torch.profiler.profile(
    activities=[
        torch.profiler.ProfilerActivity.CPU
    ]
) as prof:

    output = model(X)

print(
    prof.key_averages()
)

Profiling can help answer:

Which operation is slow?
Which layer uses the most time?
Where is the performance bottleneck?

71. Important Tensor Shapes

Understanding tensor shapes is critical.

Tabular data

Usually:

[batch, features]

Example:

[32, 10]

Meaning:

32 samples
10 features

Images

Usually:

[batch, channels, height, width]

Example:

[32, 3, 224, 224]

Meaning:

images
channels
height
width

Sequences

Often:

[batch, sequence_length, features]

Example:

[32, 100, 512]

Meaning:

sequences
tokens/time steps
features

Interview tip

When debugging a PyTorch model, always print:

print(x.shape)

Shape errors are among the most common problems in deep learning.

72. Common PyTorch Errors

Shape mismatch

Example:

mat1 and mat2 shapes cannot be multiplied

Check:

print(x.shape)
print(layer.weight.shape)

Make sure the input feature dimension matches the layer’s expected in_features.

CPU/GPU mismatch

Incorrect:

model = model.to("cuda")

output = model(X)

if X is still on CPU. Correct:

model = model.to(device)

X = X.to(device)

output = model(X)

Incorrect target dtype

For CrossEntropyLoss, targets should generally be integer class indices with dtype:

torch.int64

You can use:

target = target.long()

73. Debugging Checklist

When your PyTorch model fails, check:

print(X.shape)
print(X.dtype)
print(X.device)

print(y.shape)
print(y.dtype)

print(
    next(model.parameters()).device
)

Check for invalid values:

print(
    torch.isnan(X).any()
)

print(
    torch.isinf(X).any()
)

Check gradients:

for name, param in model.named_parameters():

    if param.grad is not None:

        print(
            name,
            param.grad.abs().mean()
        )

This can help identify:

NaN gradients
Zero gradients
Exploding gradients
Wrong devices
Wrong shapes
Wrong dtypes

74. Overfitting

Overfitting means the model performs very well on training data but poorly on unseen data. Example:

Training accuracy → 99%
Validation accuracy → 70%

Possible solutions:

More training data
Data augmentation
Dropout
Weight decay
Early stopping
Smaller model
Transfer learning

Example:

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=0.001,
    weight_decay=0.01
)

75. Underfitting

Underfitting means the model cannot even perform well on the training data. Example:

Training accuracy → 60%
Validation accuracy → 58%

Possible solutions:

Increase model capacity
Train longer
Improve features
Adjust learning rate
Reduce excessive regularization
Use a better architecture

76. Learning Rate

The learning rate determines the size of parameter updates. Conceptually:

New parameter
=
Old parameter
-
Learning rate × Gradient

Too high:

Training may oscillate
or diverge

Too low:

Training can be extremely slow

Typical starting points can be:

Adam  → 0.001
SGD   → 0.01 or 0.1

These are only starting points. The appropriate value depends on the model, data, optimizer, and training setup.

77. Batch Size

Suppose:

Dataset = 10,000 samples
Batch size = 100

Then:

10,000 / 100
= 100 batches per epoch

Large batch sizes can provide:

Better hardware utilization
More stable gradient estimates

but require:

More memory

Small batch sizes require less memory but can result in noisier gradients and potentially slower hardware utilization.

78. Epoch

An epoch means:

One complete pass through the training dataset.

Example:

for epoch in range(10):

    for X, y in train_loader:

        ...

This means the model processes the training dataset ten times.

79. Iteration

An iteration usually refers to one batch/optimizer update. For example:

10,000 samples
Batch size = 100

gives:

100 iterations per epoch

For:

10 epochs

you get approximately:

100 × 10
= 1,000 iterations

80. `state_dict`

PyTorch stores model parameters in a state_dict.

print(
    model.state_dict()
)

You can inspect a particular parameter:

print(
    model.state_dict()["fc.weight"]
)

Save:

torch.save(
    model.state_dict(),
    "weights.pth"
)

Load:

model.load_state_dict(
    torch.load(
        "weights.pth",
        weights_only=True
    )
)

81. Inference

Inference means using a trained model to make predictions. Standard pattern:

model.eval()

with torch.no_grad():

    output = model(X)

    prediction = output.argmax(
        dim=1
    )

print(prediction)

Remember:

Training:
model.train()

Inference:
model.eval()
+
torch.no_grad()

82. Clean PyTorch Project Structure

A real project may be organized like this:

my_project/
│
├── data/
│
├── models/
│   └── model.py
│
├── dataset/
│   └── dataset.py
│
├── train.py
│
├── evaluate.py
│
├── inference.py
│
├── config.py
│
├── checkpoints/
│
└── requirements.txt

This makes the project easier to maintain than putting everything into one huge Python file.

83. Complete PyTorch Workflow

A typical machine-learning project follows:

Collect Data
      ↓
Clean / Preprocess
      ↓
Create Dataset
      ↓
Create DataLoader
      ↓
Define Model
      ↓
Select Device
      ↓
Define Loss
      ↓
Define Optimizer
      ↓
Train
      ↓
Validate
      ↓
Save Best Model
      ↓
Test
      ↓
Inference
      ↓
Deploy

The most important thing is to understand what happens at every stage.

84. Complete Training Template

Here is a practical template:

import torch
import torch.nn as nn


# -----------------------
# Device
# -----------------------

device = torch.device(
    "cuda"
    if torch.cuda.is_available()
    else "cpu"
)


# -----------------------
# Model
# -----------------------

class Model(nn.Module):

    def __init__(self):
        super().__init__()

        self.network = nn.Sequential(
            nn.Linear(10, 64),
            nn.ReLU(),

            nn.Linear(64, 32),
            nn.ReLU(),

            nn.Linear(32, 3)
        )

    def forward(self, x):

        return self.network(x)


model = Model().to(device)


# -----------------------
# Loss
# -----------------------

loss_fn = nn.CrossEntropyLoss()


# -----------------------
# Optimizer
# -----------------------

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=0.001,
    weight_decay=0.01
)


# -----------------------
# Training
# -----------------------

for epoch in range(10):

    model.train()

    for X, y in train_loader:

        X = X.to(device)
        y = y.to(device)

        # Forward pass

        output = model(X)

        # Calculate loss

        loss = loss_fn(
            output,
            y
        )

        # Clear old gradients

        optimizer.zero_grad()

        # Backpropagation

        loss.backward()

        # Update parameters

        optimizer.step()

    print(
        f"Epoch {epoch + 1}: "
        f"Loss = {loss.item():.4f}"
    )


# -----------------------
# Evaluation
# -----------------------

model.eval()

correct = 0
total = 0

with torch.no_grad():

    for X, y in test_loader:

        X = X.to(device)
        y = y.to(device)

        output = model(X)

        prediction = output.argmax(
            dim=1
        )

        correct += (
            prediction == y
        ).sum().item()

        total += y.size(0)


accuracy = correct / total

print(
    "Accuracy:",
    accuracy
)


# -----------------------
# Save model
# -----------------------

torch.save(
    model.state_dict(),
    "model.pth"
)

The most important sequence is:

Data
 ↓
model(X)
 ↓
prediction
 ↓
loss_fn(prediction, target)
 ↓
optimizer.zero_grad()
 ↓
loss.backward()
 ↓
optimizer.step()

85. What You Should Memorize for Interviews

Don’t try to memorize every PyTorch function. Understand these concepts deeply.

Level 1 — Tensors

Know:

torch.tensor()
torch.zeros()
torch.ones()
torch.randn()

x.shape
x.dtype
x.device

reshape()
view()
flatten()

unsqueeze()
squeeze()

You should be able to explain:

What is a tensor and why are tensor shapes important?

Level 2 — Autograd

Know:

requires_grad=True
loss.backward()
tensor.grad
detach()
torch.no_grad()

Understand:

Forward pass
 ↓
Loss
 ↓
Backward pass
 ↓
Gradients

Level 3 — Neural Networks

Know:

nn.Module
nn.Linear
nn.Sequential
nn.ReLU
nn.Sigmoid
nn.Softmax

You should understand what happens inside:

model(X)

Level 4 — Training

Know:

loss_fn
optimizer
optimizer.zero_grad()
loss.backward()
optimizer.step()

This is the core of PyTorch training.

Level 5 — Data

Know:

Dataset
DataLoader
batch_size
shuffle
transforms

Understand why batching is needed.

Level 6 — Computer Vision

Know:

Conv2d
MaxPool2d
BatchNorm
Dropout
CNN
Transfer Learning

Understand image tensor shape:

[batch, channels, height, width]

Level 7 — Advanced Deep Learning

Know the concepts:

RNN
LSTM
GRU
Attention
Transformer
Mixed Precision
Gradient Clipping
Learning-rate Scheduling
Transfer Learning
Distributed Training
Model Optimization
Deployment

86. PyTorch Cheat Sheet

# -----------------------
# Import
# -----------------------

import torch
import torch.nn as nn


# -----------------------
# Tensor
# -----------------------

x = torch.tensor([
    1,
    2,
    3
])


# -----------------------
# Random Tensor
# -----------------------

x = torch.randn(
    3,
    4
)


# -----------------------
# Shape
# -----------------------

print(x.shape)


# -----------------------
# Reshape
# -----------------------

x = x.reshape(
    2,
    6
)


# -----------------------
# Device
# -----------------------

device = torch.device(
    "cuda"
    if torch.cuda.is_available()
    else "cpu"
)

x = x.to(device)


# -----------------------
# Model
# -----------------------

model = nn.Linear(
    10,
    2
)


# -----------------------
# Loss
# -----------------------

loss_fn = nn.CrossEntropyLoss()


# -----------------------
# Optimizer
# -----------------------

optimizer = torch.optim.Adam(
    model.parameters(),
    lr=0.001
)


# -----------------------
# Training
# -----------------------

model.train()

output = model(x)

loss = loss_fn(
    output,
    target
)

optimizer.zero_grad()

loss.backward()

optimizer.step()


# -----------------------
# Evaluation
# -----------------------

model.eval()

with torch.no_grad():

    output = model(x)


# -----------------------
# Save
# -----------------------

torch.save(
    model.state_dict(),
    "model.pth"
)


# -----------------------
# Load
# -----------------------

model.load_state_dict(
    torch.load(
        "model.pth",
        weights_only=True
    )
)

87. Recommended Learning Order

Don’t try to learn everything simultaneously. Follow this progression:

Python
  ↓
NumPy
  ↓
PyTorch tensors
  ↓
Tensor operations
  ↓
Tensor shapes
  ↓
Autograd
  ↓
nn.Module
  ↓
Linear Regression
  ↓
Classification
  ↓
Loss Functions
  ↓
Optimizers
  ↓
Dataset
  ↓
DataLoader
  ↓
Training Loop
  ↓
Validation / Testing
  ↓
CNN
  ↓
Transfer Learning
  ↓
RNN
  ↓
LSTM / GRU
  ↓
Attention
  ↓
Transformers
  ↓
Mixed Precision
  ↓
Distributed Training
  ↓
Model Optimization
  ↓
Deployment

88. The Core PyTorch Mental Model

If you remember only one thing, remember this:

                 DATA
                   ↓
              DataLoader
                   ↓
              Batch X, y
                   ↓
                MODEL
                   ↓
             Prediction
                   ↓
             LOSS FUNCTION
                   ↓
                 Loss
                   ↓
          optimizer.zero_grad()
                   ↓
             loss.backward()
                   ↓
           Gradients calculated
                   ↓
           optimizer.step()
                   ↓
          Parameters updated
                   ↓
               Repeat

The model is essentially learning:

Input
  ↓
Weights
  ↓
Prediction
  ↓
Error
  ↓
Gradient
  ↓
Better weights

After thousands or millions of updates, the model ideally learns parameters that produce useful predictions.

89. The Most Important PyTorch Concepts

For practical work and interviews, prioritize these concepts:

1. Tensor

Understand:

Shape
Dtype
Device
Operations

2. Autograd

Understand:

Gradient
Backward pass
Computation graph

3. `nn.Module`

Understand how neural networks are constructed.

4. Forward pass

prediction = model(X)

5. Loss

loss = loss_fn(
    prediction,
    target
)

6. Backpropagation

loss.backward()

7. Optimizer

optimizer.step()

8. Dataset/DataLoader

Understand how data reaches the model.

9. Training vs evaluation

model.train()
model.eval()

10. GPU

model.to(device)
X.to(device)

11. CNN

Understand image processing.

12. Attention/Transformer

Understand the architecture behind many modern NLP and generative-AI systems.

90. Final PyTorch Pattern to Memorize

The most important PyTorch code pattern is:

model.train()

for X, y in train_loader:

    X = X.to(device)
    y = y.to(device)

    optimizer.zero_grad()

    prediction = model(X)

    loss = loss_fn(
        prediction,
        y
    )

    loss.backward()

    optimizer.step()

Understand what each line means:

model.train()
→ Put model into training mode.

X, y
→ Get a batch of input data and labels.

X.to(device)
→ Move input to CPU/GPU.

optimizer.zero_grad()
→ Remove gradients from the previous update.

model(X)
→ Forward pass.

loss_fn(...)
→ Measure prediction error.

loss.backward()
→ Calculate gradients using backpropagation.

optimizer.step()
→ Update model parameters.

Then evaluation:

model.eval()

with torch.no_grad():

    prediction = model(X)

So the complete mental model is:

TENSORS
   ↓
AUTOGRAD
   ↓
nn.Module
   ↓
FORWARD PASS
   ↓
LOSS
   ↓
BACKPROPAGATION
   ↓
OPTIMIZER
   ↓
UPDATED PARAMETERS
   ↓
DATASET + DATALOADER
   ↓
TRAINING LOOP
   ↓
CNN / RNN / TRANSFORMER
   ↓
TRAINED MODEL
   ↓
INFERENCE
   ↓
DEPLOYMENT

​1. What is PyTorch?

​Main PyTorch components

​Installation

​Why learn PyTorch?

​2. Importing PyTorch

​What does nn mean?

​3. What is a Tensor?

​Scalar

​Vector

​Matrix

​Why are tensors important?

​4. Creating Tensors

​torch.zeros()

​torch.ones()

​torch.rand()

​torch.randn()

​torch.randint()

​torch.arange()

​5. Tensor Data Types

​Why does dtype matter?

​6. Tensor Shape

​Number of elements

​7. Tensor Indexing

​Slicing

​8. Tensor Arithmetic

​9. Matrix Multiplication

​Important interview question

​10. Useful Tensor Functions

​argmax()

​11. Reshaping Tensors

​view()

​flatten()

​12. unsqueeze() and squeeze()

​13. Concatenation and Stacking

​torch.cat()

​torch.stack()

​Difference

​14. NumPy and PyTorch

​15. CPU and GPU

​Important rule

​16. Autograd

​What happened?

​17. Gradient Descent

​18. Why zero_grad()?

​19. nn.Module

​Two important methods

​20. nn.Linear

​21. Activation Functions

​ReLU

​Sigmoid

​Tanh

​Softmax

​22. Building a Neural Network

​23. nn.Sequential

​24. Loss Functions

​MSE Loss

​Cross Entropy

​BCEWithLogitsLoss

​25. Optimizers

​SGD

​Adam

​AdamW

​What is learning rate?

​26. Complete Training Loop

​1. model.train()

​2. Move data

​3. Forward pass

​4. Calculate loss

​5. Clear old gradients

​6. Backpropagation

​7. Update parameters

​27. Dataset

​__init__()

​__len__()

​__getitem__()

​28. DataLoader

​Why use DataLoader?

​29. Training, Validation and Test Sets

​Training set

​Validation set

1. What is PyTorch?

Main PyTorch components

Installation

Why learn PyTorch?

2. Importing PyTorch

What does `nn` mean?

3. What is a Tensor?

Scalar

Vector

Matrix

Why are tensors important?

4. Creating Tensors

`torch.zeros()`

`torch.ones()`

`torch.rand()`

`torch.randn()`

`torch.randint()`

`torch.arange()`

5. Tensor Data Types

Why does dtype matter?

6. Tensor Shape

Number of elements

7. Tensor Indexing

Slicing

8. Tensor Arithmetic

9. Matrix Multiplication

Important interview question

10. Useful Tensor Functions

`argmax()`

11. Reshaping Tensors

`view()`

`flatten()`

12. `unsqueeze()` and `squeeze()`

13. Concatenation and Stacking

`torch.cat()`

`torch.stack()`

Difference

14. NumPy and PyTorch

15. CPU and GPU

Important rule

16. Autograd

What happened?

17. Gradient Descent

18. Why `zero_grad()`?

19. `nn.Module`

Two important methods

20. `nn.Linear`

21. Activation Functions

ReLU

Sigmoid

Tanh

Softmax

22. Building a Neural Network

23. `nn.Sequential`

24. Loss Functions

MSE Loss

Cross Entropy

BCEWithLogitsLoss

25. Optimizers

SGD

Adam

AdamW

What is learning rate?

26. Complete Training Loop

1. `model.train()`

2. Move data

3. Forward pass

4. Calculate loss

5. Clear old gradients

6. Backpropagation

7. Update parameters

27. Dataset

`init()`

`len()`

`getitem()`

28. DataLoader

Why use DataLoader?

29. Training, Validation and Test Sets

Training set

Validation set