Chapter 39
A Neural Net from the Foundations
NotebookPython 3102 cells
In [ ]python · cell 1
python
#hide
! [ -e /content ] && pip install -Uqq fastbook
import fastbook
fastbook.setup_book()A Neural Net from the Foundations
Building a Neural Net Layer from Scratch
Modeling a Neuron
Matrix Multiplication from Scratch
In [ ]python · cell 6
python
import torch
from torch import tensorIn [ ]python · cell 7
python
def matmul(a,b):
ar,ac = a.shape # n_rows * n_cols
br,bc = b.shape
assert ac==br
c = torch.zeros(ar, bc)
for i in range(ar):
for j in range(bc):
for k in range(ac): c[i,j] += a[i,k] * b[k,j]
return cIn [ ]python · cell 8
python
m1 = torch.randn(5,28*28)
m2 = torch.randn(784,10)In [ ]python · cell 9
python
%time t1=matmul(m1, m2)In [ ]python · cell 10
python
%timeit -n 20 t2=m1@m2Elementwise Arithmetic
In [ ]python · cell 12
python
a = tensor([10., 6, -4])
b = tensor([2., 8, 7])
a + bIn [ ]python · cell 13
python
a < bIn [ ]python · cell 14
python
(a < b).all(), (a==b).all()In [ ]python · cell 15
python
(a + b).mean().item()In [ ]python · cell 16
python
m = tensor([[1., 2, 3], [4,5,6], [7,8,9]])
m*mIn [ ]python · cell 17
python
n = tensor([[1., 2, 3], [4,5,6]])
m*nIn [ ]python · cell 18
python
def matmul(a,b):
ar,ac = a.shape
br,bc = b.shape
assert ac==br
c = torch.zeros(ar, bc)
for i in range(ar):
for j in range(bc): c[i,j] = (a[i] * b[:,j]).sum()
return cIn [ ]python · cell 19
python
%timeit -n 20 t3 = matmul(m1,m2)Broadcasting
Broadcasting with a scalar
In [ ]python · cell 22
python
a = tensor([10., 6, -4])
a > 0In [ ]python · cell 23
python
m = tensor([[1., 2, 3], [4,5,6], [7,8,9]])
(m - 5) / 2.73Broadcasting a vector to a matrix
In [ ]python · cell 25
python
c = tensor([10.,20,30])
m = tensor([[1., 2, 3], [4,5,6], [7,8,9]])
m.shape,c.shapeIn [ ]python · cell 26
python
m + cIn [ ]python · cell 27
python
c.expand_as(m)In [ ]python · cell 28
python
t = c.expand_as(m)
t.storage()In [ ]python · cell 29
python
t.stride(), t.shapeIn [ ]python · cell 30
python
c + mIn [ ]python · cell 31
python
c = tensor([10.,20,30])
m = tensor([[1., 2, 3], [4,5,6]])
c+mIn [ ]python · cell 32
python
c = tensor([10.,20])
m = tensor([[1., 2, 3], [4,5,6]])
c+mIn [ ]python · cell 33
python
c = tensor([10.,20,30])
m = tensor([[1., 2, 3], [4,5,6], [7,8,9]])
c = c.unsqueeze(1)
m.shape,c.shapeIn [ ]python · cell 34
python
c+mIn [ ]python · cell 35
python
t = c.expand_as(m)
t.storage()In [ ]python · cell 36
python
t.stride(), t.shapeIn [ ]python · cell 37
python
c = tensor([10.,20,30])
c.shape, c.unsqueeze(0).shape,c.unsqueeze(1).shapeIn [ ]python · cell 38
python
c.shape, c[None,:].shape,c[:,None].shapeIn [ ]python · cell 39
python
c[None].shape,c[...,None].shapeIn [ ]python · cell 40
python
def matmul(a,b):
ar,ac = a.shape
br,bc = b.shape
assert ac==br
c = torch.zeros(ar, bc)
for i in range(ar):
# c[i,j] = (a[i,:] * b[:,j]).sum() # previous
c[i] = (a[i ].unsqueeze(-1) * b).sum(dim=0)
return cIn [ ]python · cell 41
python
%timeit -n 20 t4 = matmul(m1,m2)Broadcasting rules
Einstein Summation
In [ ]python · cell 44
python
def matmul(a,b): return torch.einsum('ik,kj->ij', a, b)In [ ]python · cell 45
python
%timeit -n 20 t5 = matmul(m1,m2)The Forward and Backward Passes
Defining and Initializing a Layer
In [ ]python · cell 48
python
def lin(x, w, b): return x @ w + bIn [ ]python · cell 49
python
x = torch.randn(200, 100)
y = torch.randn(200)In [ ]python · cell 50
python
w1 = torch.randn(100,50)
b1 = torch.zeros(50)
w2 = torch.randn(50,1)
b2 = torch.zeros(1)In [ ]python · cell 51
python
l1 = lin(x, w1, b1)
l1.shapeIn [ ]python · cell 52
python
l1.mean(), l1.std()In [ ]python · cell 53
python
x = torch.randn(200, 100)
for i in range(50): x = x @ torch.randn(100,100)
x[0:5,0:5]In [ ]python · cell 54
python
x = torch.randn(200, 100)
for i in range(50): x = x @ (torch.randn(100,100) * 0.01)
x[0:5,0:5]In [ ]python · cell 55
python
x = torch.randn(200, 100)
for i in range(50): x = x @ (torch.randn(100,100) * 0.1)
x[0:5,0:5]In [ ]python · cell 56
python
x.std()In [ ]python · cell 57
python
x = torch.randn(200, 100)
y = torch.randn(200)In [ ]python · cell 58
python
from math import sqrt
w1 = torch.randn(100,50) / sqrt(100)
b1 = torch.zeros(50)
w2 = torch.randn(50,1) / sqrt(50)
b2 = torch.zeros(1)In [ ]python · cell 59
python
l1 = lin(x, w1, b1)
l1.mean(),l1.std()In [ ]python · cell 60
python
def relu(x): return x.clamp_min(0.)In [ ]python · cell 61
python
l2 = relu(l1)
l2.mean(),l2.std()In [ ]python · cell 62
python
x = torch.randn(200, 100)
for i in range(50): x = relu(x @ (torch.randn(100,100) * 0.1))
x[0:5,0:5]In [ ]python · cell 63
python
x = torch.randn(200, 100)
for i in range(50): x = relu(x @ (torch.randn(100,100) * sqrt(2/100)))
x[0:5,0:5]In [ ]python · cell 64
python
x = torch.randn(200, 100)
y = torch.randn(200)In [ ]python · cell 65
python
w1 = torch.randn(100,50) * sqrt(2 / 100)
b1 = torch.zeros(50)
w2 = torch.randn(50,1) * sqrt(2 / 50)
b2 = torch.zeros(1)In [ ]python · cell 66
python
l1 = lin(x, w1, b1)
l2 = relu(l1)
l2.mean(), l2.std()In [ ]python · cell 67
python
def model(x):
l1 = lin(x, w1, b1)
l2 = relu(l1)
l3 = lin(l2, w2, b2)
return l3In [ ]python · cell 68
python
out = model(x)
out.shapeIn [ ]python · cell 69
python
def mse(output, targ): return (output.squeeze(-1) - targ).pow(2).mean()In [ ]python · cell 70
python
loss = mse(out, y)Gradients and the Backward Pass
In [ ]python · cell 72
python
def mse_grad(inp, targ):
# grad of loss with respect to output of previous layer
inp.g = 2. * (inp.squeeze() - targ).unsqueeze(-1) / inp.shape[0]In [ ]python · cell 73
python
def relu_grad(inp, out):
# grad of relu with respect to input activations
inp.g = (inp>0).float() * out.gIn [ ]python · cell 74
python
def lin_grad(inp, out, w, b):
# grad of matmul with respect to input
inp.g = out.g @ w.t()
w.g = inp.t() @ out.g
b.g = out.g.sum(0)Sidebar: SymPy
In [ ]python · cell 76
python
from sympy import symbols,diff
sx,sy = symbols('sx sy')
diff(sx**2, sx)End sidebar
In [ ]python · cell 78
python
def forward_and_backward(inp, targ):
# forward pass:
l1 = inp @ w1 + b1
l2 = relu(l1)
out = l2 @ w2 + b2
# we don't actually need the loss in backward!
loss = mse(out, targ)
# backward pass:
mse_grad(out, targ)
lin_grad(l2, out, w2, b2)
relu_grad(l1, l2)
lin_grad(inp, l1, w1, b1)Refactoring the Model
In [ ]python · cell 80
python
class Relu():
def __call__(self, inp):
self.inp = inp
self.out = inp.clamp_min(0.)
return self.out
def backward(self): self.inp.g = (self.inp>0).float() * self.out.gIn [ ]python · cell 81
python
class Lin():
def __init__(self, w, b): self.w,self.b = w,b
def __call__(self, inp):
self.inp = inp
self.out = inp@self.w + self.b
return self.out
def backward(self):
self.inp.g = self.out.g @ self.w.t()
self.w.g = self.inp.t() @ self.out.g
self.b.g = self.out.g.sum(0)In [ ]python · cell 82
python
class Mse():
def __call__(self, inp, targ):
self.inp = inp
self.targ = targ
self.out = (inp.squeeze() - targ).pow(2).mean()
return self.out
def backward(self):
x = (self.inp.squeeze()-self.targ).unsqueeze(-1)
self.inp.g = 2.*x/self.targ.shape[0]In [ ]python · cell 83
python
class Model():
def __init__(self, w1, b1, w2, b2):
self.layers = [Lin(w1,b1), Relu(), Lin(w2,b2)]
self.loss = Mse()
def __call__(self, x, targ):
for l in self.layers: x = l(x)
return self.loss(x, targ)
def backward(self):
self.loss.backward()
for l in reversed(self.layers): l.backward()In [ ]python · cell 84
python
model = Model(w1, b1, w2, b2)In [ ]python · cell 85
python
loss = model(x, y)In [ ]python · cell 86
python
model.backward()Going to PyTorch
In [ ]python · cell 88
python
class LayerFunction():
def __call__(self, *args):
self.args = args
self.out = self.forward(*args)
return self.out
def forward(self): raise Exception('not implemented')
def bwd(self): raise Exception('not implemented')
def backward(self): self.bwd(self.out, *self.args)In [ ]python · cell 89
python
class Relu(LayerFunction):
def forward(self, inp): return inp.clamp_min(0.)
def bwd(self, out, inp): inp.g = (inp>0).float() * out.gIn [ ]python · cell 90
python
class Lin(LayerFunction):
def __init__(self, w, b): self.w,self.b = w,b
def forward(self, inp): return inp@self.w + self.b
def bwd(self, out, inp):
inp.g = out.g @ self.w.t()
self.w.g = inp.t() @ self.out.g
self.b.g = out.g.sum(0)In [ ]python · cell 91
python
class Mse(LayerFunction):
def forward (self, inp, targ): return (inp.squeeze() - targ).pow(2).mean()
def bwd(self, out, inp, targ):
inp.g = 2*(inp.squeeze()-targ).unsqueeze(-1) / targ.shape[0]In [ ]python · cell 92
python
from torch.autograd import Function
class MyRelu(Function):
@staticmethod
def forward(ctx, i):
result = i.clamp_min(0.)
ctx.save_for_backward(i)
return result
@staticmethod
def backward(ctx, grad_output):
i, = ctx.saved_tensors
return grad_output * (i>0).float()In [ ]python · cell 93
python
import torch.nn as nn
class LinearLayer(nn.Module):
def __init__(self, n_in, n_out):
super().__init__()
self.weight = nn.Parameter(torch.randn(n_out, n_in) * sqrt(2/n_in))
self.bias = nn.Parameter(torch.zeros(n_out))
def forward(self, x): return x @ self.weight.t() + self.biasIn [ ]python · cell 94
python
lin = LinearLayer(10,2)
p1,p2 = lin.parameters()
p1.shape,p2.shapeIn [ ]python · cell 95
python
class Model(nn.Module):
def __init__(self, n_in, nh, n_out):
super().__init__()
self.layers = nn.Sequential(
nn.Linear(n_in,nh), nn.ReLU(), nn.Linear(nh,n_out))
self.loss = mse
def forward(self, x, targ): return self.loss(self.layers(x).squeeze(), targ)In [ ]python · cell 96
python
class Model(Module):
def __init__(self, n_in, nh, n_out):
self.layers = nn.Sequential(
nn.Linear(n_in,nh), nn.ReLU(), nn.Linear(nh,n_out))
self.loss = mse
def forward(self, x, targ): return self.loss(self.layers(x).squeeze(), targ)Conclusion
Questionnaire
- Write the Python code to implement a single neuron.
- Write the Python code to implement ReLU.
- Write the Python code for a dense layer in terms of matrix multiplication.
- Write the Python code for a dense layer in plain Python (that is, with list comprehensions and functionality built into Python).
- What is the "hidden size" of a layer?
- What does the
tmethod do in PyTorch? - Why is matrix multiplication written in plain Python very slow?
- In
matmul, why isac==br? - In Jupyter Notebook, how do you measure the time taken for a single cell to execute?
- What is "elementwise arithmetic"?
- Write the PyTorch code to test whether every element of
ais greater than the corresponding element ofb. - What is a rank-0 tensor? How do you convert it to a plain Python data type?
- What does this return, and why?
tensor([1,2]) + tensor([1]) - What does this return, and why?
tensor([1,2]) + tensor([1,2,3]) - How does elementwise arithmetic help us speed up
matmul? - What are the broadcasting rules?
- What is
expand_as? Show an example of how it can be used to match the results of broadcasting. - How does
unsqueezehelp us to solve certain broadcasting problems? - How can we use indexing to do the same operation as
unsqueeze? - How do we show the actual contents of the memory used for a tensor?
- When adding a vector of size 3 to a matrix of size 3×3, are the elements of the vector added to each row or each column of the matrix? (Be sure to check your answer by running this code in a notebook.)
- Do broadcasting and
expand_asresult in increased memory use? Why or why not? - Implement
matmulusing Einstein summation. - What does a repeated index letter represent on the left-hand side of einsum?
- What are the three rules of Einstein summation notation? Why?
- What are the forward pass and backward pass of a neural network?
- Why do we need to store some of the activations calculated for intermediate layers in the forward pass?
- What is the downside of having activations with a standard deviation too far away from 1?
- How can weight initialization help avoid this problem?
- What is the formula to initialize weights such that we get a standard deviation of 1 for a plain linear layer, and for a linear layer followed by ReLU?
- Why do we sometimes have to use the
squeezemethod in loss functions? - What does the argument to the
squeezemethod do? Why might it be important to include this argument, even though PyTorch does not require it? - What is the "chain rule"? Show the equation in either of the two forms presented in this chapter.
- Show how to calculate the gradients of
mse(lin(l2, w2, b2), y)using the chain rule. - What is the gradient of ReLU? Show it in math or code. (You shouldn't need to commit this to memory—try to figure it using your knowledge of the shape of the function.)
- In what order do we need to call the
*_gradfunctions in the backward pass? Why? - What is
__call__? - What methods must we implement when writing a
torch.autograd.Function? - Write
nn.Linearfrom scratch, and test it works. - What is the difference between
nn.Moduleand fastai'sModule?
Further Research
- Implement ReLU as a
torch.autograd.Functionand train a model with it. - If you are mathematically inclined, find out what the gradients of a linear layer are in mathematical notation. Map that to the implementation we saw in this chapter.
- Learn about the
unfoldmethod in PyTorch, and use it along with matrix multiplication to implement your own 2D convolution function. Then train a CNN that uses it. - Implement everything in this chapter using NumPy instead of PyTorch.
In [ ]python · cell 102
python