A fastai Learner from Scratch

This final chapter (other than the conclusion and the online chapters) is going to look a bit different. It contains far more code and far less prose than the previous chapters. We will introduce new Python keywords and libraries without discussing them. This chapter is meant to be the start of a significant research project for you. You see, we are going to implement many of the key pieces of the fastai and PyTorch APIs from scratch, building on nothing other than the components that we developed in <<chapter_foundations>>! The key goal here is to end up with your own Learner class, and some callbacks—enough to be able to train a model on Imagenette, including examples of each of the key techniques we've studied. On the way to building Learner, we will create our own version of Module, Parameter, and parallel DataLoader so you have a very good idea of what those PyTorch classes do.

The end-of-chapter questionnaire is particularly important for this chapter. This is where we will be pointing you in the many interesting directions that you could take, using this chapter as your starting point. We suggest that you follow along with this chapter on your computer, and do lots of experiments, web searches, and whatever else you need to understand what's going on. You've built up the skills and expertise to do this in the rest of this book, so we think you are going to do great!

最后一章（除了结论和在线章节）看起来会有点不同。它包含的代码比前几章多得多，文字却少得多。我们将在不讨论它们的情况下介绍新的Python关键字和库。这一章将成为您一个重要研究项目的开始。您看，我们将从头开始实现Fastai和PyTorch API的许多关键部分，仅基于我们在“chapter_foundations”中开发的组件！这里的关键目标是最终得到您自己的Learner类和一些回调——足以在Imagenette上训练模型，包括我们研究过的每种关键技术的示例。在构建Learner的过程中，我们将创建自己的模块、参数和并行DataLoader版本，这样您就可以很好地了解这些PyTorch类的作用。 本章结尾的问卷调查对本章尤其重要。这是我们将以本章为起点，为您指出许多有趣的方向。我们建议您在计算机上阅读本章，并进行大量实验、网络搜索以及其他任何您需要了解的事情。您在本书的其余部分已经积累了这样做的技能和专业知识，因此我们认为您会做得很好！

Let's begin by gathering (manually) some data.

让我们从（手动）收集一些数据开始。

Data

Have a look at the source to untar_data to see how it works. We'll use it here to access the 160-pixel version of Imagenette for use in this chapter:

看看untar_data的源代码，看看它是如何工作的。我们将在这里使用它来访问本章使用的160像素版本的Imagenette：

To access the image files, we can use get_image_files:

要访问图像文件，我们可以使用get_image_files：

Or we could do the same thing using just Python's standard library, with glob:

或者我们可以只使用Python的标准库来做同样的事情：

If you look at the source for get_image_files, you'll see it uses Python's os.walk; this is a faster and more flexible function than glob, so be sure to try it out.

We can open an image with the Python Imaging Library's Image class:

如果你查看get_image_files的源代码，你会发现它使用了Python的os.walk；这是一个比globb更快、更灵活的函数，所以一定要试一试。 我们可以使用Python Imaging Library的Image类打开图像：

That's going to be the basis of our independent variable. For our dependent variable, we can use Path.parent from pathlib. First we'll need our vocab:

这将是我们自变量的基础。对于因变量，我们可以使用path lib中的Path.parent。首先，我们需要我们的词汇表：

...and the reverse mapping, thanks to L.val2idx:

...和反向映射，感谢L.val2idx：

That's all the pieces we need to put together our Dataset.

这就是我们需要将数据集放在一起的所有部分。

Dataset

A Dataset in PyTorch can be anything that supports indexing (__getitem__) and len:

PyTorch中的数据集可以是支持索引（getitem）和len的任何内容：

We need a list of training and validation filenames to pass to Dataset.__init__:

我们需要一份训练和验证文件名的列表传递给Dataset.init：

Now we can try it out:

现在我们可以尝试一下：

As you see, our dataset is returning the independent and dependent variables as a tuple, which is just what we need. We'll need to be able to collate these into a mini-batch. Generally this is done with torch.stack, which is what we'll use here:

如您所见，我们的数据集将自变量和因变量作为元组返回，这正是我们所需要的。我们需要能够将这些整理成一个小批次。通常这是通过torch.stack完成的，这就是我们将在这里使用的：

Here's a mini-batch with two items, for testing our collate:

这是一个包含两个项目的小批量，用于测试我们的collate：

Now that we have a dataset and a collation function, we're ready to create DataLoader. We'll add two more things here: an optional shuffle for the training set, and a ProcessPoolExecutor to do our preprocessing in parallel. A parallel data loader is very important, because opening and decoding a JPEG image is a slow process. One CPU core is not enough to decode images fast enough to keep a modern GPU busy. Here's our DataLoader class:

现在我们有了数据集和归类功能，我们准备好创建DataLoader了。我们将在这里添加另外两件事：一个可选择的shuffle为训练集，以及并行进行预处理的ProcessPoolExecitor。并行数据加载器非常重要，因为打开和解码JPEG图像是一个缓慢的过程。一个CPU内核不足以快速解码图像以使现代GPU保持忙碌。这是我们的DataLoader类：

Let's try it out with our training and validation datasets:

让我们使用我们的训练和验证数据集来尝试一下：

This data loader is not much slower than PyTorch's, but it's far simpler. So if you're debugging a complex data loading process, don't be afraid to try doing things manually to help you see exactly what's going on.

For normalization, we'll need image statistics. Generally it's fine to calculate these on a single training mini-batch, since precision isn't needed here:

这个数据加载器并不比PyTorch慢多少，但要简单得多。因此，如果您正在调试复杂的数据加载过程，请不要害怕尝试手动操作以帮助您准确了解发生了什么。 对于规范化，我们需要图像统计数据。通常，在单个训练小批次上计算这些是可以的，因为这里不需要精度：

Our Normalize class just needs to store these stats and apply them (to see why the to_device is needed, try commenting it out, and see what happens later in this notebook):

我们的Normalize类只需要存储这些统计信息并应用它们（要了解为什么需要to_device，请尝试注释它，并查看本笔记本后面会发生什么）：

We always like to test everything we build in a notebook, as soon as we build it:

我们总是喜欢在构建笔记本时测试我们构建的所有内容：

Here tfm_x isn't just applying Normalize, but is also permuting the axis order from NHWC to NCHW (see <<chapter_convolutions>> if you need a reminder of what these acronyms refer to). PIL uses HWC axis order, which we can't use with PyTorch, hence the need for this permute.

这里tfm_x不仅仅是应用规范化，而且还将轴顺序从NHWC排列到NCHW（如果您需要提醒这些首字母缩略词指的是什么，请参阅“chapter_convolutions”）。PIL使用HWC轴顺序，我们不能在PyTorch中使用，因此需要这个permute。

That's all we need for the data for our model. So now we need the model itself!

这就是我们模型所需的数据。所以现在我们需要模型本身！

Module and Parameter

To create a model, we'll need Module. To create Module, we'll need Parameter, so let's start there. Recall that in <<chapter_collab>> we said that the Parameter class "doesn't actually add any functionality (other than automatically calling requires_grad_ for us). It's only used as a "marker" to show what to include in parameters." Here's a definition which does exactly that:

要创建一个模型，我们需要Module。要创建Module，我们需要Parameter，所以让我们从那里开始。回想一下，在“chapter_collab”中，我们说过Parameter类“实际上并没有添加任何功能（除了自动为我们调用requires_grad_）。它只用作“标记”来显示Parameters中包含的内容。”这是一个完全可以做到这一点的定义：

The implementation here is a bit awkward: we have to define the special __new__ Python method and use the internal PyTorch method _make_subclass because, as at the time of writing, PyTorch doesn't otherwise work correctly with this kind of subclassing or provide an officially supported API to do this. This may have been fixed by the time you read this, so look on the book's website to see if there are updated details.

Our Parameter now behaves just like a tensor, as we wanted:

这里的实现有点尴尬：我们必须定义特殊__new__Python方法并使用内部PyTorch方法_make_subclass因为在撰写本文时，PyTorch无法正常使用这种子类化或提供官方支持的API来执行此操作。在您阅读本文时，这可能已经修复，因此请查看本书的网站，看看是否有更新的详细信息。 我们的Parameter现在的行为就像张量，正如我们想要的：

Now that we have this, we can define Module:

现在我们有了这个，我们可以定义Module：

The key functionality is in the definition of parameters:

self.params + sum([m.parameters() for m in self.children], [])

This means that we can ask any Module for its parameters, and it will return them, including all its child modules (recursively). But how does it know what its parameters are? It's thanks to implementing Python's special __setattr__ method, which is called for us any time Python sets an attribute on a class. Our implementation includes this line:

if isinstance(v,Parameter): self.register_parameters(v)

As you see, this is where we use our new Parameter class as a "marker"—anything of this class is added to our params.

Python's __call__ allows us to define what happens when our object is treated as a function; we just call forward (which doesn't exist here, so it'll need to be added by subclasses). Before we do, we'll call a hook, if it's defined. Now you can see that PyTorch hooks aren't doing anything fancy at all—they're just calling any hooks that have been registered.

Other than these pieces of functionality, our Module also provides cuda and training attributes, which we'll use shortly.

Now we can create our first Module, which is ConvLayer:

关键功能在定义parameters： self.params + sum([m.parameters() for m in self.children], []) 这意味着我们可以向任何Module询问它的参数，它会返回它们，包括它的所有子模块（递归地）。但是它怎么知道它的参数是什么呢？这要归功于实现Python的特殊__setattr__方法，每当Python在类上设置属性时，我们都会调用它。我们的实现包括这一行： if isinstance(v,Parameter): self.register_parameters(v) 如您所见，这就是我们使用新Parameter类作为“标记”的地方——此类的任何内容都被添加到我们的params中。 Python的__call__允许我们定义当我们的对象被视为函数时会发生什么；我们只是调用forward（这里不存在，所以需要通过子类添加）。在我们这样做之前，我们将调用一个hook，如果它已经定义的话。现在你可以看到PyTorch hooks根本没有做任何花哨的事情——它们只是调用任何已经注册的hooks。 除了这些功能，我们的Module还提供了cuda和training属性，我们将很快使用这些属性。 现在我们可以创建我们的第一个Module，即ConvLayer：

We're not implementing F.conv2d from scratch, since you should have already done that (using unfold) in the questionnaire in <<chapter_foundations>>. Instead, we're just creating a small class that wraps it up along with bias and weight initialization. Let's check that it works correctly with Module.parameters:

我们不会从头开始实现F.conv2d，因为您应该已经在“chapter_foundations”的问卷中完成了（使用unfold）。相反，我们只是创建一个小类，将其与偏差和权重初始化一起包装起来。让我们检查它是否与Module.parameters正常工作：

And that we can call it (which will result in forward being called):

并且我们可以调用它（这将导致"forward"被调用）：

In the same way, we can implement Linear:

同样，我们可以实现Linear：

and test if it works:

并测试它是否有效：

Let's also create a testing module to check that if we include multiple parameters as attributes, they are all correctly registered:

让我们还创建一个测试模块来检查如果我们包含多个参数作为属性，它们都被正确注册：

Since we have a conv layer and a linear layer, each of which has weights and biases, we'd expect four parameters in total:

由于我们有一个conv层和一个线性层，每个层都有权重和偏差，因此我们预计总共有四个参数：

We should also find that calling cuda on this class puts all these parameters on the GPU:

我们还应该发现，在此类上调用cuda会将所有这些参数放在GPU上：

We can now use those pieces to create a CNN.

我们现在可以用这些片段来创建CNN。

Simple CNN

As we've seen, a Sequential class makes many architectures easier to implement, so let's make one:

正如我们所看到的，Sequential类使许多架构更容易实现，因此让我们制作一个：

The forward method here just calls each layer in turn. Note that we have to use the register_modules method we defined in Module, since otherwise the contents of layers won't appear in parameters.

这里的forward方法只是依次调用每个层。请注意，我们必须使用我们在Module中定义的register_modules方法，否则layers的内容不会出现在parameters中。

important: All The Code is Here: Remember that we're not using any PyTorch functionality for modules here; we're defining everything ourselves. So if you're not sure what register_modules does, or why it's needed, have another look at our code for Module to see what we wrote!

重要提示：所有代码都在这里：请记住，我们这里没有为模块使用任何PyTorch功能；我们自己定义一切。因此，如果您不确定register_modules做什么，或者为什么需要它，请再看一下我们的Module代码，看看我们写了什么！

We can create a simplified AdaptivePool that only handles pooling to a 1×1 output, and flattens it as well, by just using mean:

我们可以创建一个简化的AdaptivePool，它只处理1×1输出的池化，并通过使用means将其展平：

That's enough for us to create a CNN!

这足以让我们创建一个CNN！

Let's see if our parameters are all being registered correctly:

让我们看看我们的参数是否都被正确注册：

Now we can try adding a hook. Note that we've only left room for one hook in Module; you could make it a list, or use something like Pipeline to run a few as a single function:

现在我们可以尝试添加一个hook。请注意，我们在模块中只为一个hook留出了空间；您可以将其设为列表，或者使用Pipeline之类的东西将几个作为单个函数运行：

We have data and model. Now we need a loss function.

我们有数据和模型。现在我们需要一个损失函数。

Loss

We've already seen how to define "negative log likelihood":

我们已经看到了如何定义“负对数似然”：

Well actually, there's no log here, since we're using the same definition as PyTorch. That means we need to put the log together with softmax:

实际上，这里没有日志，因为我们使用与PyTorch相同的定义。这意味着我们需要将日志与softmax放在一起：

Combining these gives us our cross-entropy loss:

结合这些给出了我们的交叉熵损失：

Note that the formula:

$\log \left ( \frac{a}{b} \right ) = \log(a) - \log(b)$

gives a simplification when we compute the log softmax, which was previously defined as (x.exp()/(x.exp().sum(-1))).log():

注意公式： $\log \left ( \frac{a}{b} \right ) = \log(a) - \log(b)$ 当我们计算日志softmax,以前定义为'(x.exp()/(x.exp(). sum(-1))). log()'时给出了简化：

Then, there is a more stable way to compute the log of the sum of exponentials, called the LogSumExp trick. The idea is to use the following formula:

$\log \left ( \sum_{j=1}^{n} e^{x_{j}} \right ) = \log \left ( e^{a} \sum_{j=1}^{n} e^{x_{j}-a} \right ) = a + \log \left ( \sum_{j=1}^{n} e^{x_{j}-a} \right )$

where $a$ is the maximum of $x_{j}$.

Here's the same thing in code:

然后，有一种更稳定的方法来计算指数和的对数，称为[LogSumExp]（https://en.wikipedia.org/wiki/LogSumExp）技巧。想法是使用以下公式： $\log \left ( \sum_{j=1}^{n} e^{x_{j}} \right ) = \log \left ( e^{a} \sum_{j=1}^{n} e^{x_{j}-a} \right ) = a + \log \left ( \sum_{j=1}^{n} e^{x_{j}-a} \right )$ 其中$a$是$x_{j}$的最大值. 代码中是一样的：

We'll put that into a function:

我们将把它放入一个函数中：

so we can use it for our log_softmax function:

所以我们可以将其用于我们的log_softmax功能：

Which gives the same result as before:

这给出了与之前相同的结果：

We can use these to create cross_entropy:

我们可以使用这些来创建cross_entropy：

Let's now combine all those pieces together to create a Learner.

现在让我们将所有这些片段组合在一起创建一个Learner。

Learner

We have data, a model, and a loss function; we only need one more thing before we can fit a model, and that's an optimizer! Here's SGD:

我们有数据、模型和损失函数；在拟合模型之前，我们只需要一件事，那就是优化器！这是SGD：

As we've seen in this book, life is easier with a Learner. The Learner class needs to know our training and validation sets, which means we need DataLoaders to store them. We don't need any other functionality, just a place to store them and access them:

正如我们在本书中看到的，使用Learner会更轻松。Learner类需要知道我们的训练和验证集，这意味着我们需要DataLoaders来存储它们。我们不需要任何其他功能，只需要一个存储和访问它们的地方：

Now we're ready to create our Learner class:

现在我们准备创建我们的Learner类：

This is the largest class we've created in the book, but each method is quite small, so by looking at each in turn you should be able to follow what's going on.

The main method we'll be calling is fit. This loops with:

for self.epoch in range(n_epochs)

and at each epoch calls self.one_epoch for each of train=True and then train=False. Then self.one_epoch calls self.one_batch for each batch in dls.train or dls.valid, as appropriate (after wrapping the DataLoader in fastprogress.progress_bar. Finally, self.one_batch follows the usual set of steps to fit one mini-batch that we've seen throughout this book.

Before and after each step, Learner calls self, which calls __call__ (which is standard Python functionality). __call__ uses getattr(cb,name) on each callback in self.cbs, which is a Python built-in function that returns the attribute (a method, in this case) with the requested name. So, for instance, self('before_fit') will call cb.before_fit() for each callback where that method is defined.

As you can see, Learner is really just using our standard training loop, except that it's also calling callbacks at appropriate times. So let's define some callbacks!

这是我们在书中创建的最大的类，但是每个方法都很小，所以通过依次查看每个方法，您应该能够了解发生了什么。 我们将调用的主要方法是fit。这循环使用： for self.epoch in range(n_epochs) 在每个epoch中，调用self.one_epoch，分别为训练=True和训练=False。然后self.one_epochdls.train或dls.valid中的每个批次调用self.one_batch（在将DataLoader包装fastprogress.progress_bar之后）。最后，self.one_batch遵循我们在本书中看到的一组常见步骤来适应一个小批次。 在每个步骤之前和之后，Learner调用self，它调用__call__（这是标准的Python功能）。__call__在self.cbs中的每个回调上使用getattr（cb， name），这是一个Python内置函数，它返回带有请求名称的属性（在本例中是方法）。因此，例如，在定义该方法的每个回调中，self（'before_fit'）将调用cb.before_fit（）。 如您所见，Learner实际上只是使用我们的标准训练循环，只是它也在适当的时候调用回调。所以让我们定义一些回调！

Callbacks

In Learner.__init__ we have:

for cb in cbs: cb.learner = self

In other words, every callback knows what learner it is used in. This is critical, since otherwise a callback can't get information from the learner, or change things in the learner. Because getting information from the learner is so common, we make that easier by defining Callback as a subclass of GetAttr, with a default attribute of learner:

在Learner中。__init__我们有： for cb in cbs: cb.learner = self 换句话说，每个回调都知道它在哪个学习者中使用。这很关键，因为否则回调无法从学习者那里获取信息，也无法更改学习者中的内容。因为从学习者那里获取信息是如此普遍，所以我们通过将Callback定义为GetAttr的子类来简化它，默认属性为learner：

GetAttr is a fastai class that implements Python's standard __getattr__ and __dir__ methods for you, such that any time you try to access an attribute that doesn't exist, it passes the request along to whatever you have defined as _default.

GetAttr是一个快速类，它为您实现了Python的标准__getattr__和__dir__方法，因此每当您尝试访问不存在的属性时，它都会将请求传递给您定义为_default的任何内容。

For instance, we want to move all model parameters to the GPU automatically at the start of fit. We could do this by defining before_fit as self.learner.model.cuda(); however, because learner is the default attribute, and we have SetupLearnerCB inherit from Callback (which inherits from GetAttr), we can remove the .learner and just call self.model.cuda():

例如，我们希望在fit开始时自动将所有模型参数移动到GPU。我们可以通过将before_fit定义为self.learner.model.cuda（）来做到这一点；但是，因为learner是默认属性，并且我们有SetupLearnerCB继承自Callback（继承自GetAttr），我们可以删除. learner并只调用self.model.cuda（）：

In SetupLearnerCB we also move each mini-batch to the GPU, by calling to_device(self.batch) (we could also have used the longer to_device(self.learner.batch). Note however that in the line self.learner.batch = tfm_x(xb),yb we can't remove .learner, because here we're setting the attribute, not getting it.

Before we try our Learner out, let's create a callback to track and print progress. Otherwise we won't really know if it's working properly:

在SetupLearnerCB中，我们还通过调用to_device（self.batch）将每个小批处理移动到GPU（我们也可以使用更长的to_device（self.learner.batch）。但是请注意，在self.learner.batch=tfm_x（xb），yb行中，我们不能删除. learner，因为这里我们设置的是属性，而不是获取它。 在我们尝试我们的Learner之前，让我们创建一个回调来跟踪和打印进度。否则我们不会真正知道它是否正常工作：

Now we're ready to use our Learner for the first time!

现在我们准备好第一次使用我们的Learner了！

It's quite amazing to realize that we can implement all the key ideas from fastai's Learner in so little code! Let's now add some learning rate scheduling.

意识到我们可以在如此少的代码中实现Fastai Learner的所有关键思想真是令人惊讶！现在让我们添加一些学习率调度。

Scheduling the Learning Rate

If we're going to get good results, we'll want an LR finder and 1cycle training. These are both annealing callbacks—that is, they are gradually changing hyperparameters as we train. Here's LRFinder:

如果我们要获得好的结果，我们需要一个LR查找器和1周期训练。这两个都是退火回调——也就是说，它们在我们训练时逐渐改变超参数。这是LRFinder：

This shows how we're using CancelFitException, which is itself an empty class, only used to signify the type of exception. You can see in Learner that this exception is caught. (You should add and test CancelBatchException, CancelEpochException, etc. yourself.) Let's try it out, by adding it to our list of callbacks:

这显示了我们如何使用取消FitException，它本身是一个空类，仅用于表示异常的类型。您可以在Learner中看到此异常被捕获。（您应该自己添加和测试CancelBatchException、CancelEpochException等。）让我们尝试一下，将其添加到我们的回调列表中：

And take a look at the results:

让我们看看结果：

Now we can define our OneCycle training callback:

现在我们可以定义我们的OneCycle训练回调：

We'll try an LR of 0.1:

我们将尝试0.1的LR：

Let's fit for a while and see how it looks (we won't show all the output in the book—try it in the notebook to see the results):

让我们适应一段时间，看看它看起来如何（我们不会在书中显示所有输出-在笔记本中尝试查看结果）：

Finally, we'll check that the learning rate followed the schedule we defined (as you see, we're not using cosine annealing here):

最后，我们将检查学习率是否遵循我们定义的时间表（如您所见，我们这里没有使用余弦退火）：

Conclusion

We have explored how the key concepts of the fastai library are implemented by re-implementing them in this chapter. Since it's mostly full of code, you should definitely try to experiment with it by looking at the corresponding notebook on the book's website. Now that you know how it's built, as a next step be sure to check out the intermediate and advanced tutorials in the fastai documentation to learn how to customize every bit of the library.

我们已经探索了如何通过在本章中重新实现快速库的关键概念来实现它们。由于它大部分都是代码，您绝对应该通过查看本书网站上相应的笔记本来尝试使用它。现在您知道它是如何构建的，作为下一步，请务必查看快速留档中的中级和高级教程，以了解如何自定义库的每一点。

Questionnaire

tip: Experiments: For the questions here that ask you to explain what some function or class is, you should also complete your own code experiments.

提示：实验：对于这里要求您解释某个函数或类是什么的问题，您还应该完成自己的代码实验。

1. What is glob?
2. How do you open an image with the Python imaging library?
3. What does L.map do?
4. What does Self do?
5. What is L.val2idx?
6. What methods do you need to implement to create your own Dataset?
7. Why do we call convert when we open an image from Imagenette?
8. What does ~ do? How is it useful for splitting training and validation sets?
9. Does ~ work with the L or Tensor classes? What about NumPy arrays, Python lists, or pandas DataFrames?
10. What is ProcessPoolExecutor?
11. How does L.range(self.ds) work?
12. What is __iter__?
13. What is first?
14. What is permute? Why is it needed?
15. What is a recursive function? How does it help us define the parameters method?
16. Write a recursive function that returns the first 20 items of the Fibonacci sequence.
17. What is super?
18. Why do subclasses of Module need to override forward instead of defining __call__?
19. In ConvLayer, why does init depend on act?
20. Why does Sequential need to call register_modules?
21. Write a hook that prints the shape of every layer's activations.
22. What is "LogSumExp"?
23. Why is log_softmax useful?
24. What is GetAttr? How is it helpful for callbacks?
25. Reimplement one of the callbacks in this chapter without inheriting from Callback or GetAttr.
26. What does Learner.__call__ do?
27. What is getattr? (Note the case difference to GetAttr!)
28. Why is there a try block in fit?
29. Why do we check for model.training in one_batch?
30. What is store_attr?
31. What is the purpose of TrackResults.before_epoch?
32. What does model.cuda do? How does it work?
33. Why do we need to check model.training in LRFinder and OneCycle?
34. Use cosine annealing in OneCycle.

1.什么是glob？ 2.如何使用Python成像库打开图像？ 3.L.map是做什么的？ 4.self是做什么的？ 5.什么是L.val2idx？ 6.创建自己的数据集需要实现哪些方法？ 7.当我们从Imagenette打开图像时，为什么我们调用转换？ 8.做什么？它对拆分训练集和验证集有什么用？ 9.是否适用于L或Tensor类？NumPy数组、Python列表或熊猫DataFrames呢？ 10.什么是ProcessPoolExecitor？ 11.L.range（self.ds）是如何工作的？ 12.什么是__iter__？ 13.什么是first？ 14.什么是Permute？为什么需要它？ 15.什么是递归函数？它如何帮助我们定义参数方法？ 16.编写一个递归函数，返回斐波那契序列的前20项。 17.什么是super？ 18.为什么模块的子类需要向前覆盖而不是定义__call__？ 19.在ConvLayer中，为什么init依赖于act？ 20.为什么Sequential需要调用register_modules？ 21.写一个hook，打印每一层激活的形状。 22.什么是“LogSumExp”？ 23.为什么log_softmax有用？ 24.什么是GetAttr？它对回调有什么帮助？ 25.在不继承Callback或GetAttr的情况下重新实现本章中的一个回调。 26.学习者。__call__做什么？ 27.什么是getattr？（注意GetAttr的大小写区别！） 28.为什么有适合的try块？ 29.我们为什么要检查one_batch的model.training？ 30.什么是store_attr？ 31.TrackResults.before_epoch的目的是什么？ 32.model.cuda是做什么的？它是怎么工作的？ 33.为什么我们需要在LRFinder和OneCycle中检查model.training？ 34.在OneCycle中使用余弦退火。

Further Research

1. Write resnet18 from scratch (refer to <<chapter_resnet>> as needed), and train it with the Learner in this chapter.
2. Implement a batchnorm layer from scratch and use it in your resnet18.
3. Write a Mixup callback for use in this chapter.
4. Add momentum to SGD.
5. Pick a few features that you're interested in from fastai (or any other library) and implement them in this chapter.
6. Pick a research paper that's not yet implemented in fastai or PyTorch and implement it in this chapter.

• Port it over to fastai.
• Submit a pull request to fastai, or create your own extension module and release it.
• Hint: you may find it helpful to use nbdev to create and deploy your package.

1.从头开始编写resnet18（根据需要参考“chapter_resnet”），并在本章中使用Learner对其进行训练。 2.从头开始实现一个批处理规范层并在resnet18中使用它。 3.编写一个Mixup回调以在本章中使用。 4.为SGD增加动力。 5.选择一些你感兴趣的特性，并在本章中实现它们。 6.选择一篇尚未在Fastai或PyTorch中实现的研究论文，并在本章中实现它。 把它移植到Fastai。 向Fastai提交拉取请求，或者创建自己的扩展模块并释放它。 提示：您可能会发现使用nbdev来创建和部署包很有帮助。