Chapter 32

08. PyTorch Paper Replicating Exercises

NotebookPython 324 cells

08. PyTorch Paper Replicating Exercises

Welcome to the 08. PyTorch Paper Replicating exercises.

Your objective is to write code to satisify each of the exercises below.

Some starter code has been provided to make sure you have all the resources you need.

Note: There may be more than one solution to each of the exercises.

Resources

These exercises/solutions are based on section 08. PyTorch Paper Replicating of the Learn PyTorch for Deep Learning course by Zero to Mastery.
See a live walkthrough of the solutions (errors and all) on YouTube (but try the exercises yourself first!).
See all solutions on the course GitHub.

Note: The first section of this notebook is dedicated to getting various helper functions and datasets used for the exercises. The exercises start at the heading "Exercise 1: ...".

Get various imports and helper functions

The code in the following cells prepares imports and data for the exercises below. They are taken from 08. PyTorch Paper Replicating.

In [2]python · cell 4

python

# For this notebook to run with updated APIs, we need torch 1.12+ and torchvision 0.13+
try:
    import torch
    import torchvision
    assert int(torch.__version__.split(".")[1]) >= 12, "torch version should be 1.12+"
    assert int(torchvision.__version__.split(".")[1]) >= 13, "torchvision version should be 0.13+"
    print(f"torch version: {torch.__version__}")
    print(f"torchvision version: {torchvision.__version__}")
except:
    print(f"[INFO] torch/torchvision versions not as required, installing nightly versions.")
    !pip3 install -U --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cu113
    import torch
    import torchvision
    print(f"torch version: {torch.__version__}")
    print(f"torchvision version: {torchvision.__version__}")

Output

torch version: 1.12.0+cu113
torchvision version: 0.13.0+cu113

In [3]python · cell 5

python

# Continue with regular imports
import matplotlib.pyplot as plt
import torch
import torchvision

from torch import nn
from torchvision import transforms

# Try to get torchinfo, install it if it doesn't work
try:
    from torchinfo import summary
except:
    print("[INFO] Couldn't find torchinfo... installing it.")
    !pip install -q torchinfo
    from torchinfo import summary

# Try to import the going_modular directory, download it from GitHub if it doesn't work
try:
    from going_modular.going_modular import data_setup, engine
    from helper_functions import download_data, set_seeds, plot_loss_curves
except:
    # Get the going_modular scripts
    print("[INFO] Couldn't find going_modular or helper_functions scripts... downloading them from GitHub.")
    !git clone https://github.com/mrdbourke/pytorch-deep-learning
    !mv pytorch-deep-learning/going_modular .
    !mv pytorch-deep-learning/helper_functions.py . # get the helper_functions.py script
    !rm -rf pytorch-deep-learning
    from going_modular.going_modular import data_setup, engine
    from helper_functions import download_data, set_seeds, plot_loss_curves

Output

[INFO] Couldn't find torchinfo... installing it.
[INFO] Couldn't find going_modular or helper_functions scripts... downloading them from GitHub.
Cloning into 'pytorch-deep-learning'...
remote: Enumerating objects: 2589, done.[K
remote: Counting objects: 100% (28/28), done.[K
remote: Compressing objects: 100% (17/17), done.[K
remote: Total 2589 (delta 11), reused 28 (delta 11), pack-reused 2561[K
Receiving objects: 100% (2589/2589), 446.45 MiB | 41.00 MiB/s, done.
Resolving deltas: 100% (1455/1455), done.
Checking out files: 100% (184/184), done.

In [4]python · cell 6

python

device = "cuda" if torch.cuda.is_available() else "cpu"
device

Output

'cuda'

Get data

Want to download the data we've been using in PyTorch Paper Replicating: https://www.learnpytorch.io/08_pytorch_paper_replicating/#1-get-data

In [5]python · cell 8

python

# Download pizza, steak, sushi images from GitHub
image_path = download_data(source="https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi.zip",
                           destination="pizza_steak_sushi")
image_path

Output

[INFO] Did not find data/pizza_steak_sushi directory, creating one...
[INFO] Downloading pizza_steak_sushi.zip from https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi.zip...
[INFO] Unzipping pizza_steak_sushi.zip data...

PosixPath('data/pizza_steak_sushi')

In [6]python · cell 9

python

# Setup directory paths to train and test images
train_dir = image_path / "train"
test_dir = image_path / "test"

Preprocess data

Turn images into tensors using same code as PyTorch Paper Replicating section 2.1 and 2.2: https://www.learnpytorch.io/08_pytorch_paper_replicating/#21-prepare-transforms-for-images

In [7]python · cell 11

python

# Create image size (from Table 3 in the ViT paper) 
IMG_SIZE = 224

# Create transform pipeline manually
manual_transforms = transforms.Compose([
    transforms.Resize((IMG_SIZE, IMG_SIZE)),
    transforms.ToTensor(),
])           
print(f"Manually created transforms: {manual_transforms}")

Output

Manually created transforms: Compose(
    Resize(size=(224, 224), interpolation=bilinear, max_size=None, antialias=None)
    ToTensor()
)

In [8]python · cell 12

python

# Set the batch size
BATCH_SIZE = 32 # this is lower than the ViT paper but it's because we're starting small

# Create data loaders
train_dataloader, test_dataloader, class_names = data_setup.create_dataloaders(
    train_dir=train_dir,
    test_dir=test_dir,
    transform=manual_transforms, # use manually created transforms
    batch_size=BATCH_SIZE
)

train_dataloader, test_dataloader, class_names

Output

(<torch.utils.data.dataloader.DataLoader at 0x7f7e08f76c90>,
 <torch.utils.data.dataloader.DataLoader at 0x7f7e0889f590>,
 ['pizza', 'steak', 'sushi'])

In [9]python · cell 13

python

# Get a batch of images
image_batch, label_batch = next(iter(train_dataloader))

# Get a single image from the batch
image, label = image_batch[0], label_batch[0]

# View the batch shapes
image.shape, label

Output

(torch.Size([3, 224, 224]), tensor(0))

In [10]python · cell 14

python

# Plot image with matplotlib
plt.imshow(image.permute(1, 2, 0)) # rearrange image dimensions to suit matplotlib [color_channels, height, width] -> [height, width, color_channels]
plt.title(class_names[label])
plt.axis(False);

Output

<Figure size 432x288 with 1 Axes>

[省略较大 image/png 输出]

1. Replicate the ViT architecture we created with in-built PyTorch transformer layers.

You'll want to look into replacing our TransformerEncoderBlock() class with torch.nn.TransformerEncoderLayer() (these contain the same layers as our custom blocks).
You can stack torch.nn.TransformerEncoderLayer()'s on top of each other with torch.nn.TransformerEncoder().

In [11]python · cell 16

python

# TODO: your code

2. Turn the custom ViT architecture we created into a Python script, for example, `vit.py`.

You should be able to import an entire ViT model using something likefrom vit import ViT.
We covered the art of turning code cells into Python scrips in 05. PyTorch Going Modular.

In [12]python · cell 18

python

# TODO: your code

3. Train a pretrained ViT feature extractor model (like the one we made in 08. PyTorch Paper Replicating section 10) on 20% of the pizza, steak and sushi data like the dataset we used in 07. PyTorch Experiment Tracking section 7.3

See how it performs compared to the EffNetB2 model we compared it to in 08. PyTorch Paper Replicating section 10.6.

In [13]python · cell 20

python

# TODO: your code

4. Try repeating the steps from excercise 3 but this time use the "`ViT_B_16_Weights.IMAGENET1K_SWAG_E2E_V1`" pretrained weights from `torchvision.models.vit_b_16()`.

Note: ViT pretrained with SWAG weights has a minimum input image size of (384, 384), though this is accessible in the weights .transforms() method.

In [14]python · cell 22

python

# TODO: your code

5. Our custom ViT model architecture closely mimics that of the ViT paper, however, our training recipe misses a few things.

Research some of the following topics from Table 3 in the ViT paper that we miss and write a sentence about each and how it might help with training:
- ImageNet-21k pretraining
- Learning rate warmup
- Learning rate decay
- Gradient clipping

In [15]python · cell 24

python

# TODO: your explanations of the above terms

Resources

Note: The first section of this notebook is dedicated to getting various helper functions and datasets used for the exercises. The exercises start at the heading "Exercise 1: ...".

# For this notebook to run with updated APIs, we need torch 1.12+ and torchvision 0.13+ try: import torch import torchvision assert int(torch.__version__.split(".")[1]) >= 12, "torch version should be 1.12+" assert int(torchvision.__version__.split(".")[1]) >= 13, "torchvision version should be 0.13+" print(f"torch version: {torch.__version__}") print(f"torchvision version: {torchvision.__version__}") except: print(f"[INFO] torch/torchvision versions not as required, installing nightly versions.") !pip3 install -U --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cu113 import torch import torchvision print(f"torch version: {torch.__version__}") print(f"torchvision version: {torchvision.__version__}")

# Continue with regular imports import matplotlib.pyplot as plt import torch import torchvision from torch import nn from torchvision import transforms # Try to get torchinfo, install it if it doesn't work try: from torchinfo import summary except: print("[INFO] Couldn't find torchinfo... installing it.") !pip install -q torchinfo from torchinfo import summary # Try to import the going_modular directory, download it from GitHub if it doesn't work try: from going_modular.going_modular import data_setup, engine from helper_functions import download_data, set_seeds, plot_loss_curves except: # Get the going_modular scripts print("[INFO] Couldn't find going_modular or helper_functions scripts... downloading them from GitHub.") !git clone https://github.com/mrdbourke/pytorch-deep-learning !mv pytorch-deep-learning/going_modular . !mv pytorch-deep-learning/helper_functions.py . # get the helper_functions.py script !rm -rf pytorch-deep-learning from going_modular.going_modular import data_setup, engine from helper_functions import download_data, set_seeds, plot_loss_curves

[INFO] Couldn't find torchinfo... installing it. [INFO] Couldn't find going_modular or helper_functions scripts... downloading them from GitHub. Cloning into 'pytorch-deep-learning'... remote: Enumerating objects: 2589, done.[K remote: Counting objects: 100% (28/28), done.[K remote: Compressing objects: 100% (17/17), done.[K remote: Total 2589 (delta 11), reused 28 (delta 11), pack-reused 2561[K Receiving objects: 100% (2589/2589), 446.45 MiB | 41.00 MiB/s, done. Resolving deltas: 100% (1455/1455), done. Checking out files: 100% (184/184), done.

# Download pizza, steak, sushi images from GitHub image_path = download_data(source="https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi.zip", destination="pizza_steak_sushi") image_path

[INFO] Did not find data/pizza_steak_sushi directory, creating one... [INFO] Downloading pizza_steak_sushi.zip from https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi.zip... [INFO] Unzipping pizza_steak_sushi.zip data...

# Create image size (from Table 3 in the ViT paper) IMG_SIZE = 224 # Create transform pipeline manually manual_transforms = transforms.Compose([ transforms.Resize((IMG_SIZE, IMG_SIZE)), transforms.ToTensor(), ]) print(f"Manually created transforms: {manual_transforms}")

# Set the batch size BATCH_SIZE = 32 # this is lower than the ViT paper but it's because we're starting small # Create data loaders train_dataloader, test_dataloader, class_names = data_setup.create_dataloaders( train_dir=train_dir, test_dir=test_dir, transform=manual_transforms, # use manually created transforms batch_size=BATCH_SIZE ) train_dataloader, test_dataloader, class_names

# Get a batch of images image_batch, label_batch = next(iter(train_dataloader)) # Get a single image from the batch image, label = image_batch[0], label_batch[0] # View the batch shapes image.shape, label

# Plot image with matplotlib plt.imshow(image.permute(1, 2, 0)) # rearrange image dimensions to suit matplotlib [color_channels, height, width] -> [height, width, color_channels] plt.title(class_names[label]) plt.axis(False);

08. PyTorch Paper Replicating Exercises

Resources

Get various imports and helper functions

Get data

Preprocess data

1. Replicate the ViT architecture we created with in-built PyTorch transformer layers.

2. Turn the custom ViT architecture we created into a Python script, for example, vit.py.

3. Train a pretrained ViT feature extractor model (like the one we made in 08. PyTorch Paper Replicating section 10) on 20% of the pizza, steak and sushi data like the dataset we used in 07. PyTorch Experiment Tracking section 7.3

4. Try repeating the steps from excercise 3 but this time use the "ViT_B_16_Weights.IMAGENET1K_SWAG_E2E_V1" pretrained weights from torchvision.models.vit_b_16().

5. Our custom ViT model architecture closely mimics that of the ViT paper, however, our training recipe misses a few things.

08. PyTorch Paper Replicating Exercises

Resources

Get various imports and helper functions

Get data

Preprocess data

1. Replicate the ViT architecture we created with in-built PyTorch transformer layers.

2. Turn the custom ViT architecture we created into a Python script, for example, vit.py.

3. Train a pretrained ViT feature extractor model (like the one we made in 08. PyTorch Paper Replicating section 10) on 20% of the pizza, steak and sushi data like the dataset we used in 07. PyTorch Experiment Tracking section 7.3

4. Try repeating the steps from excercise 3 but this time use the "ViT_B_16_Weights.IMAGENET1K_SWAG_E2E_V1" pretrained weights from torchvision.models.vit_b_16().

5. Our custom ViT model architecture closely mimics that of the ViT paper, however, our training recipe misses a few things.

2. Turn the custom ViT architecture we created into a Python script, for example, `vit.py`.

4. Try repeating the steps from excercise 3 but this time use the "`ViT_B_16_Weights.IMAGENET1K_SWAG_E2E_V1`" pretrained weights from `torchvision.models.vit_b_16()`.

2. Turn the custom ViT architecture we created into a Python script, for example, `vit.py`.

4. Try repeating the steps from excercise 3 but this time use the "`ViT_B_16_Weights.IMAGENET1K_SWAG_E2E_V1`" pretrained weights from `torchvision.models.vit_b_16()`.