Chapter 29
Training a State-of-the-Art Model
NotebookPython 330 cells
In [ ]python · cell 1
python
#hide
! [ -e /content ] && pip install -Uqq fastbook
import fastbook
fastbook.setup_book()In [ ]python · cell 2
python
#hide
from fastbook import *Training a State-of-the-Art Model
Imagenette
In [ ]python · cell 5
python
from fastai.vision.all import *
path = untar_data(URLs.IMAGENETTE)In [ ]python · cell 6
python
dblock = DataBlock(blocks=(ImageBlock(), CategoryBlock()),
get_items=get_image_files,
get_y=parent_label,
item_tfms=Resize(460),
batch_tfms=aug_transforms(size=224, min_scale=0.75))
dls = dblock.dataloaders(path, bs=64)In [ ]python · cell 7
python
model = xresnet50(n_out=dls.c)
learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), metrics=accuracy)
learn.fit_one_cycle(5, 3e-3)Normalization
In [ ]python · cell 9
python
x,y = dls.one_batch()
x.mean(dim=[0,2,3]),x.std(dim=[0,2,3])In [ ]python · cell 10
python
def get_dls(bs, size):
dblock = DataBlock(blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
get_y=parent_label,
item_tfms=Resize(460),
batch_tfms=[*aug_transforms(size=size, min_scale=0.75),
Normalize.from_stats(*imagenet_stats)])
return dblock.dataloaders(path, bs=bs)In [ ]python · cell 11
python
dls = get_dls(64, 224)In [ ]python · cell 12
python
x,y = dls.one_batch()
x.mean(dim=[0,2,3]),x.std(dim=[0,2,3])In [ ]python · cell 13
python
model = xresnet50(n_out=dls.c)
learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), metrics=accuracy)
learn.fit_one_cycle(5, 3e-3)Progressive Resizing
In [ ]python · cell 15
python
dls = get_dls(128, 128)
learn = Learner(dls, xresnet50(n_out=dls.c), loss_func=CrossEntropyLossFlat(),
metrics=accuracy)
learn.fit_one_cycle(4, 3e-3)In [ ]python · cell 16
python
learn.dls = get_dls(64, 224)
learn.fine_tune(5, 1e-3)Test Time Augmentation
In [ ]python · cell 18
python
preds,targs = learn.tta()
accuracy(preds, targs).item()Mixup
Sidebar: Papers and Math
End sidebar
In [ ]python · cell 22
python
church = PILImage.create(get_image_files_sorted(path/'train'/'n03028079')[0])
gas = PILImage.create(get_image_files_sorted(path/'train'/'n03425413')[0])
church = church.resize((256,256))
gas = gas.resize((256,256))
tchurch = tensor(church).float() / 255.
tgas = tensor(gas).float() / 255.
_,axs = plt.subplots(1, 3, figsize=(12,4))
show_image(tchurch, ax=axs[0]);
show_image(tgas, ax=axs[1]);
show_image((0.3*tchurch + 0.7*tgas), ax=axs[2]);Label Smoothing
Sidebar: Label Smoothing, the Paper
End sidebar
Conclusion
Questionnaire
- What is the difference between ImageNet and Imagenette? When is it better to experiment on one versus the other?
- What is normalization?
- Why didn't we have to care about normalization when using a pretrained model?
- What is progressive resizing?
- Implement progressive resizing in your own project. Did it help?
- What is test time augmentation? How do you use it in fastai?
- Is using TTA at inference slower or faster than regular inference? Why?
- What is Mixup? How do you use it in fastai?
- Why does Mixup prevent the model from being too confident?
- Why does training with Mixup for five epochs end up worse than training without Mixup?
- What is the idea behind label smoothing?
- What problems in your data can label smoothing help with?
- When using label smoothing with five categories, what is the target associated with the index 1?
- What is the first step to take when you want to prototype quick experiments on a new dataset?
Further Research
- Use the fastai documentation to build a function that crops an image to a square in each of the four corners, then implement a TTA method that averages the predictions on a center crop and those four crops. Did it help? Is it better than the TTA method of fastai?
- Find the Mixup paper on arXiv and read it. Pick one or two more recent articles introducing variants of Mixup and read them, then try to implement them on your problem.
- Find the script training Imagenette using Mixup and use it as an example to build a script for a long training on your own project. Execute it and see if it helps.
- Read the sidebar "Label Smoothing, the Paper", look at the relevant section of the original paper and see if you can follow it. Don't be afraid to ask for help!
In [ ]python · cell 30
python