Chapter 26
Under the Hood: Training a Digit Classifier
NotebookPython 3 (ipykernel)168 cells
In [ ]python · cell 1
python
#hide
! [ -e /content ] && pip install -Uqq fastbook
import fastbook
fastbook.setup_book()In [ ]python · cell 2
python
#hide
from fastai.vision.all import *
from fastbook import *
matplotlib.rc('image', cmap='Greys')Under the Hood: Training a Digit Classifier
Pixels: The Foundations of Computer Vision
Sidebar: Tenacity and Deep Learning
End sidebar
In [ ]python · cell 7
python
path = untar_data(URLs.MNIST_SAMPLE)In [ ]python · cell 8
python
#hide
Path.BASE_PATH = pathIn [ ]python · cell 9
python
path.ls()In [ ]python · cell 10
python
(path/'train').ls()In [ ]python · cell 11
python
threes = (path/'train'/'3').ls().sorted()
sevens = (path/'train'/'7').ls().sorted()
threesIn [ ]python · cell 12
python
im3_path = threes[1]
im3 = Image.open(im3_path)
im3In [ ]python · cell 13
python
array(im3)[4:10,4:10]In [ ]python · cell 14
python
tensor(im3)[4:10,4:10]In [ ]python · cell 15
python
im3_t = tensor(im3)
df = pd.DataFrame(im3_t[4:15,4:22])
df.style.set_properties(**{'font-size':'6pt'}).background_gradient('Greys')First Try: Pixel Similarity
In [ ]python · cell 17
python
seven_tensors = [tensor(Image.open(o)) for o in sevens]
three_tensors = [tensor(Image.open(o)) for o in threes]
len(three_tensors),len(seven_tensors)In [ ]python · cell 18
python
show_image(three_tensors[1]);In [ ]python · cell 19
python
stacked_sevens = torch.stack(seven_tensors).float()/255
stacked_threes = torch.stack(three_tensors).float()/255
stacked_threes.shapeIn [ ]python · cell 20
python
len(stacked_threes.shape)In [ ]python · cell 21
python
stacked_threes.ndimIn [ ]python · cell 22
python
mean3 = stacked_threes.mean(0)
show_image(mean3);In [ ]python · cell 23
python
mean7 = stacked_sevens.mean(0)
show_image(mean7);In [ ]python · cell 24
python
a_3 = stacked_threes[1]
show_image(a_3);In [ ]python · cell 25
python
dist_3_abs = (a_3 - mean3).abs().mean()
dist_3_sqr = ((a_3 - mean3)**2).mean().sqrt()
dist_3_abs,dist_3_sqrIn [ ]python · cell 26
python
dist_7_abs = (a_3 - mean7).abs().mean()
dist_7_sqr = ((a_3 - mean7)**2).mean().sqrt()
dist_7_abs,dist_7_sqrIn [ ]python · cell 27
python
F.l1_loss(a_3.float(),mean7), F.mse_loss(a_3,mean7).sqrt()NumPy Arrays and PyTorch Tensors
In [ ]python · cell 29
python
data = [[1,2,3],[4,5,6]]
arr = array (data)
tns = tensor(data)In [ ]python · cell 30
python
arr # numpyIn [ ]python · cell 31
python
tns # pytorchIn [ ]python · cell 32
python
tns[1]In [ ]python · cell 33
python
tns[:,1]In [ ]python · cell 34
python
tns[1,1:3]In [ ]python · cell 35
python
tns+1In [ ]python · cell 36
python
tns.type()In [ ]python · cell 37
python
tns*1.5Computing Metrics Using Broadcasting
In [ ]python · cell 39
python
valid_3_tens = torch.stack([tensor(Image.open(o))
for o in (path/'valid'/'3').ls()])
valid_3_tens = valid_3_tens.float()/255
valid_7_tens = torch.stack([tensor(Image.open(o))
for o in (path/'valid'/'7').ls()])
valid_7_tens = valid_7_tens.float()/255
valid_3_tens.shape,valid_7_tens.shapeIn [ ]python · cell 40
python
def mnist_distance(a,b): return (a-b).abs().mean((-1,-2))
mnist_distance(a_3, mean3)In [ ]python · cell 41
python
valid_3_dist = mnist_distance(valid_3_tens, mean3)
valid_3_dist, valid_3_dist.shapeIn [ ]python · cell 42
python
tensor([1,2,3]) + tensor(1)In [ ]python · cell 43
python
(valid_3_tens-mean3).shapeIn [ ]python · cell 44
python
def is_3(x): return mnist_distance(x,mean3) < mnist_distance(x,mean7)In [ ]python · cell 45
python
is_3(a_3), is_3(a_3).float()In [ ]python · cell 46
python
is_3(valid_3_tens)In [ ]python · cell 47
python
accuracy_3s = is_3(valid_3_tens).float() .mean()
accuracy_7s = (1 - is_3(valid_7_tens).float()).mean()
accuracy_3s,accuracy_7s,(accuracy_3s+accuracy_7s)/2Stochastic Gradient Descent (SGD)
In [ ]python · cell 49
python
gv('''
init->predict->loss->gradient->step->stop
step->predict[label=repeat]
''')In [ ]python · cell 50
python
def f(x): return x**2In [ ]python · cell 51
python
plot_function(f, 'x', 'x**2')In [ ]python · cell 52
python
plot_function(f, 'x', 'x**2')
plt.scatter(-1.5, f(-1.5), color='red');Calculating Gradients
In [ ]python · cell 54
python
xt = tensor(3.).requires_grad_()In [ ]python · cell 55
python
yt = f(xt)
ytIn [ ]python · cell 56
python
yt.backward()In [ ]python · cell 57
python
xt.gradIn [ ]python · cell 58
python
xt = tensor([3.,4.,10.]).requires_grad_()
xtIn [ ]python · cell 59
python
def f(x): return (x**2).sum()
yt = f(xt)
ytIn [ ]python · cell 60
python
yt.backward()
xt.gradStepping With a Learning Rate
An End-to-End SGD Example
In [ ]python · cell 63
python
time = torch.arange(0,20).float(); timeIn [ ]python · cell 64
python
speed = torch.randn(20)*3 + 0.75*(time-9.5)**2 + 1
plt.scatter(time,speed);In [ ]python · cell 65
python
def f(t, params):
a,b,c = params
return a*(t**2) + (b*t) + cIn [ ]python · cell 66
python
def mse(preds, targets): return ((preds-targets)**2).mean()Step 1: Initialize the parameters
In [ ]python · cell 68
python
params = torch.randn(3).requires_grad_()In [ ]python · cell 69
python
#hide
orig_params = params.clone()Step 2: Calculate the predictions
In [ ]python · cell 71
python
preds = f(time, params)In [ ]python · cell 72
python
def show_preds(preds, ax=None):
if ax is None: ax=plt.subplots()[1]
ax.scatter(time, speed)
ax.scatter(time, to_np(preds), color='red')
ax.set_ylim(-300,100)In [ ]python · cell 73
python
show_preds(preds)Step 3: Calculate the loss
In [ ]python · cell 75
python
loss = mse(preds, speed)
lossStep 4: Calculate the gradients
In [ ]python · cell 77
python
loss.backward()
params.gradIn [ ]python · cell 78
python
params.grad * 1e-5In [ ]python · cell 79
python
paramsStep 5: Step the weights.
In [ ]python · cell 81
python
lr = 1e-5
params.data -= lr * params.grad.data
params.grad = NoneIn [ ]python · cell 82
python
preds = f(time,params)
mse(preds, speed)In [ ]python · cell 83
python
show_preds(preds)In [ ]python · cell 84
python
def apply_step(params, prn=True):
preds = f(time, params)
loss = mse(preds, speed)
loss.backward()
params.data -= lr * params.grad.data
params.grad = None
if prn: print(loss.item())
return predsStep 6: Repeat the process
In [ ]python · cell 86
python
for i in range(10): apply_step(params)In [ ]python · cell 87
python
#hide
params = orig_params.detach().requires_grad_()In [ ]python · cell 88
python
_,axs = plt.subplots(1,4,figsize=(12,3))
for ax in axs: show_preds(apply_step(params, False), ax)
plt.tight_layout()Step 7: stop
Summarizing Gradient Descent
In [ ]python · cell 91
python
gv('''
init->predict->loss->gradient->step->stop
step->predict[label=repeat]
''')The MNIST Loss Function
In [ ]python · cell 93
python
train_x = torch.cat([stacked_threes, stacked_sevens]).view(-1, 28*28)In [ ]python · cell 94
python
train_y = tensor([1]*len(threes) + [0]*len(sevens)).unsqueeze(1)
train_x.shape,train_y.shapeIn [ ]python · cell 95
python
dset = list(zip(train_x,train_y))
x,y = dset[0]
x.shape,yIn [ ]python · cell 96
python
valid_x = torch.cat([valid_3_tens, valid_7_tens]).view(-1, 28*28)
valid_y = tensor([1]*len(valid_3_tens) + [0]*len(valid_7_tens)).unsqueeze(1)
valid_dset = list(zip(valid_x,valid_y))In [ ]python · cell 97
python
def init_params(size, std=1.0): return (torch.randn(size)*std).requires_grad_()In [ ]python · cell 98
python
weights = init_params((28*28,1))In [ ]python · cell 99
python
bias = init_params(1)In [ ]python · cell 100
python
(train_x[0]*weights.T).sum() + biasIn [ ]python · cell 101
python
def linear1(xb): return xb@weights + bias
preds = linear1(train_x)
predsIn [ ]python · cell 102
python
corrects = (preds>0.0).float() == train_y
correctsIn [ ]python · cell 103
python
corrects.float().mean().item()In [ ]python · cell 104
python
with torch.no_grad(): weights[0] *= 1.0001In [ ]python · cell 105
python
preds = linear1(train_x)
((preds>0.0).float() == train_y).float().mean().item()In [ ]python · cell 106
python
trgts = tensor([1,0,1])
prds = tensor([0.9, 0.4, 0.2])In [ ]python · cell 107
python
def mnist_loss(predictions, targets):
return torch.where(targets==1, 1-predictions, predictions).mean()In [ ]python · cell 108
python
torch.where(trgts==1, 1-prds, prds)In [ ]python · cell 109
python
mnist_loss(prds,trgts)In [ ]python · cell 110
python
mnist_loss(tensor([0.9, 0.4, 0.8]),trgts)Sigmoid
In [ ]python · cell 112
python
def sigmoid(x): return 1/(1+torch.exp(-x))In [ ]python · cell 113
python
plot_function(torch.sigmoid, title='Sigmoid', min=-4, max=4)In [ ]python · cell 114
python
def mnist_loss(predictions, targets):
predictions = predictions.sigmoid()
return torch.where(targets==1, 1-predictions, predictions).mean()SGD and Mini-Batches
In [ ]python · cell 116
python
coll = range(15)
dl = DataLoader(coll, batch_size=5, shuffle=True)
list(dl)In [ ]python · cell 117
python
ds = L(enumerate(string.ascii_lowercase))
dsIn [ ]python · cell 118
python
dl = DataLoader(ds, batch_size=6, shuffle=True)
list(dl)Putting It All Together
In [ ]python · cell 120
python
weights = init_params((28*28,1))
bias = init_params(1)In [ ]python · cell 121
python
dl = DataLoader(dset, batch_size=256)
xb,yb = first(dl)
xb.shape,yb.shapeIn [ ]python · cell 122
python
valid_dl = DataLoader(valid_dset, batch_size=256)In [ ]python · cell 123
python
batch = train_x[:4]
batch.shapeIn [ ]python · cell 124
python
preds = linear1(batch)
predsIn [ ]python · cell 125
python
loss = mnist_loss(preds, train_y[:4])
lossIn [ ]python · cell 126
python
loss.backward()
weights.grad.shape,weights.grad.mean(),bias.gradIn [ ]python · cell 127
python
def calc_grad(xb, yb, model):
preds = model(xb)
loss = mnist_loss(preds, yb)
loss.backward()In [ ]python · cell 128
python
calc_grad(batch, train_y[:4], linear1)
weights.grad.mean(),bias.gradIn [ ]python · cell 129
python
calc_grad(batch, train_y[:4], linear1)
weights.grad.mean(),bias.gradIn [ ]python · cell 130
python
weights.grad.zero_()
bias.grad.zero_();In [ ]python · cell 131
python
def train_epoch(model, lr, params):
for xb,yb in dl:
calc_grad(xb, yb, model)
for p in params:
p.data -= p.grad*lr
p.grad.zero_()In [ ]python · cell 132
python
(preds>0.0).float() == train_y[:4]In [ ]python · cell 133
python
def batch_accuracy(xb, yb):
preds = xb.sigmoid()
correct = (preds>0.5) == yb
return correct.float().mean()In [ ]python · cell 134
python
batch_accuracy(linear1(batch), train_y[:4])In [ ]python · cell 135
python
def validate_epoch(model):
accs = [batch_accuracy(model(xb), yb) for xb,yb in valid_dl]
return round(torch.stack(accs).mean().item(), 4)In [ ]python · cell 136
python
validate_epoch(linear1)In [ ]python · cell 137
python
lr = 1.
params = weights,bias
train_epoch(linear1, lr, params)
validate_epoch(linear1)In [ ]python · cell 138
python
for i in range(20):
train_epoch(linear1, lr, params)
print(validate_epoch(linear1), end=' ')Creating an Optimizer
In [ ]python · cell 140
python
linear_model = nn.Linear(28*28,1)In [ ]python · cell 141
python
w,b = linear_model.parameters()
w.shape,b.shapeIn [ ]python · cell 142
python
class BasicOptim:
def __init__(self,params,lr): self.params,self.lr = list(params),lr
def step(self, *args, **kwargs):
for p in self.params: p.data -= p.grad.data * self.lr
def zero_grad(self, *args, **kwargs):
for p in self.params: p.grad = NoneIn [ ]python · cell 143
python
opt = BasicOptim(linear_model.parameters(), lr)In [ ]python · cell 144
python
def train_epoch(model):
for xb,yb in dl:
calc_grad(xb, yb, model)
opt.step()
opt.zero_grad()In [ ]python · cell 145
python
validate_epoch(linear_model)In [ ]python · cell 146
python
def train_model(model, epochs):
for i in range(epochs):
train_epoch(model)
print(validate_epoch(model), end=' ')In [ ]python · cell 147
python
train_model(linear_model, 20)In [ ]python · cell 148
python
linear_model = nn.Linear(28*28,1)
opt = SGD(linear_model.parameters(), lr)
train_model(linear_model, 20)In [ ]python · cell 149
python
dls = DataLoaders(dl, valid_dl)In [ ]python · cell 150
python
learn = Learner(dls, nn.Linear(28*28,1), opt_func=SGD,
loss_func=mnist_loss, metrics=batch_accuracy)In [ ]python · cell 151
python
learn.fit(10, lr=lr)Adding a Nonlinearity
In [ ]python · cell 153
python
def simple_net(xb):
res = xb@w1 + b1
res = res.max(tensor(0.0))
res = res@w2 + b2
return resIn [ ]python · cell 154
python
w1 = init_params((28*28,30))
b1 = init_params(30)
w2 = init_params((30,1))
b2 = init_params(1)In [ ]python · cell 155
python
plot_function(F.relu)In [ ]python · cell 156
python
simple_net = nn.Sequential(
nn.Linear(28*28,30),
nn.ReLU(),
nn.Linear(30,1)
)In [ ]python · cell 157
python
learn = Learner(dls, simple_net, opt_func=SGD,
loss_func=mnist_loss, metrics=batch_accuracy)In [ ]python · cell 158
python
learn.fit(40, 0.1)In [ ]python · cell 159
python
plt.plot(L(learn.recorder.values).itemgot(2));In [ ]python · cell 160
python
learn.recorder.values[-1][2]Going Deeper
In [ ]python · cell 162
python
dls = ImageDataLoaders.from_folder(path)
learn = vision_learner(dls, resnet18, pretrained=False,
loss_func=F.cross_entropy, metrics=accuracy)
learn.fit_one_cycle(1, 0.1)Jargon Recap
Questionnaire
- How is a grayscale image represented on a computer? How about a color image?
- How are the files and folders in the
MNIST_SAMPLEdataset structured? Why? - Explain how the "pixel similarity" approach to classifying digits works.
- What is a list comprehension? Create one now that selects odd numbers from a list and doubles them.
- What is a "rank-3 tensor"?
- What is the difference between tensor rank and shape? How do you get the rank from the shape?
- What are RMSE and L1 norm?
- How can you apply a calculation on thousands of numbers at once, many thousands of times faster than a Python loop?
- Create a 3×3 tensor or array containing the numbers from 1 to 9. Double it. Select the bottom-right four numbers.
- What is broadcasting?
- Are metrics generally calculated using the training set, or the validation set? Why?
- What is SGD?
- Why does SGD use mini-batches?
- What are the seven steps in SGD for machine learning?
- How do we initialize the weights in a model?
- What is "loss"?
- Why can't we always use a high learning rate?
- What is a "gradient"?
- Do you need to know how to calculate gradients yourself?
- Why can't we use accuracy as a loss function?
- Draw the sigmoid function. What is special about its shape?
- What is the difference between a loss function and a metric?
- What is the function to calculate new weights using a learning rate?
- What does the
DataLoaderclass do? - Write pseudocode showing the basic steps taken in each epoch for SGD.
- Create a function that, if passed two arguments
[1,2,3,4]and'abcd', returns[(1, 'a'), (2, 'b'), (3, 'c'), (4, 'd')]. What is special about that output data structure? - What does
viewdo in PyTorch? - What are the "bias" parameters in a neural network? Why do we need them?
- What does the
@operator do in Python? - What does the
backwardmethod do? - Why do we have to zero the gradients?
- What information do we have to pass to
Learner? - Show Python or pseudocode for the basic steps of a training loop.
- What is "ReLU"? Draw a plot of it for values from
-2to+2. - What is an "activation function"?
- What's the difference between
F.reluandnn.ReLU? - The universal approximation theorem shows that any function can be approximated as closely as needed using just one nonlinearity. So why do we normally use more?
Further Research
- Create your own implementation of
Learnerfrom scratch, based on the training loop shown in this chapter. - Complete all the steps in this chapter using the full MNIST datasets (that is, for all digits, not just 3s and 7s). This is a significant project and will take you quite a bit of time to complete! You'll need to do some of your own research to figure out how to overcome some obstacles you'll meet on the way.
In [ ]python · cell 168
python