Training a State-of-the-Art Model

训练最先进的模型

This chapter introduces more advanced techniques for training an image classification model and getting state-of-the-art results. You can skip it if you want to learn more about other applications of deep learning and come back to it later—knowledge of this material will not be assumed in later chapters.

We will look at what normalization is, a powerful data augmentation technique called mixup, the progressive resizing approach and test time augmentation. To show all of this, we are going to train a model from scratch (not using transfer learning) using a subset of ImageNet called Imagenette. It contains a subset of 10 very different categories from the original ImageNet dataset, making for quicker training when we want to experiment.

This is going to be much harder to do well than with our previous datasets because we're using full-size, full-color images, which are photos of objects of different sizes, in different orientations, in different lighting, and so forth. So, in this chapter we're going to introduce some important techniques for getting the most out of your dataset, especially when you're training from scratch, or using transfer learning to train a model on a very different kind of dataset than the pretrained model used.

本章介绍了训练图像分类模型和获得业内最佳表现的更高级技术。如果你想了解更多关于深度学习的其他应用，你可以跳过它，稍后再回来——在后面的章节中不会涉及本材料的内容。 我们将了解什么是规范化，一种称为mixup的强大数据增强技术，它是一种渐进地调整大小和测试时间增强地方法。为了展示这些内容，我们将使用名为Imagenette的ImageNet子集从头开始训练模型（不使用迁移学习）。它包含与原始ImageNet数据集非常不同的10个类别的子集，以便在我们想要进行实验时更快地训练。 这将比我们以前的数据集要困难得多，因为我们使用全尺寸、全彩色图像，这些图像是不同尺寸、不同方向、不同光线下的物体的照片，等等。因此，在本章中，我们将介绍一些重要的技术来充分利用您的数据集，尤其是当您从头开始训练时，或者使用迁移学习在与使用的预训练模型非常不同的数据集上训练模型。

Imagenette

影像

When fast.ai first started there were three main datasets that people used for building and testing computer vision models:

• ImageNet:: 1.3 million images of various sizes around 500 pixels across, in 1,000 categories, which took a few days to train
• MNIST:: 50,000 28×28-pixel grayscale handwritten digits
• CIFAR10:: 60,000 32×32-pixel color images in 10 classes

The problem was that the smaller datasets didn't actually generalize effectively to the large ImageNet dataset. The approaches that worked well on ImageNet generally had to be developed and trained on ImageNet. This led to many people believing that only researchers with access to giant computing resources could effectively contribute to developing image classification algorithms.

We thought that seemed very unlikely to be true. We had never actually seen a study that showed that ImageNet happen to be exactly the right size, and that other datasets could not be developed which would provide useful insights. So we thought we would try to create a new dataset that researchers could test their algorithms on quickly and cheaply, but which would also provide insights likely to work on the full ImageNet dataset.

About three hours later we had created Imagenette. We selected 10 classes from the full ImageNet that looked very different from one another. As we had hoped, we were able to quickly and cheaply create a classifier capable of recognizing these classes. We then tried out a few algorithmic tweaks to see how they impacted Imagenette. We found some that worked pretty well, and tested them on ImageNet as well—and we were very pleased to find that our tweaks worked well on ImageNet too!

There is an important message here: the dataset you get given is not necessarily the dataset you want. It's particularly unlikely to be the dataset that you want to do your development and prototyping in. You should aim to have an iteration speed of no more than a couple of minutes—that is, when you come up with a new idea you want to try out, you should be able to train a model and see how it goes within a couple of minutes. If it's taking longer to do an experiment, think about how you could cut down your dataset, or simplify your model, to improve your experimentation speed. The more experiments you can do, the better!

Let's get started with this dataset:

当fast.ai最初开始时，人们用于构建和测试计算机视觉模型的主要数据集有三个： ImageNet:: 130万各种大小的图像，大约500像素宽，1000个类别，需要几天的时间来训练 MNIST:: 50,000个28×28像素灰度手写数字 CIFAR10:: 10类60,000张32×32像素彩色图像

问题是较小的数据集实际上并不能有效地推广到大型ImageNet数据集。在ImageNet上运行良好的方法通常必须在ImageNet上开发和训练。这导致许多人认为，只有能够访问庞大计算资源的研究人员才能有效地为开发图像分类算法做出贡献。

我们认为这似乎不太可能是真的。我们从未真正看到过一项研究表明ImageNet恰好是正确的大小，并且无法开发其他数据集来提供有用的见解。因此，我们认为我们应该尝试创建一个新的数据集，研究人员可以快速且廉价地测试他们的算法，但也可以提供可能适用于完整ImageNet数据集的见解。

大约三个小时后，我们创建了Imagenette。我们从完整的ImageNet中选择了10个看起来彼此非常不同的类。正如我们所希望的那样，我们能够快速且廉价地创建一个能够识别这些类的分类器。然后，我们尝试了一些算法调整，看看它们如何影响Imagenette。我们找到了一些运行良好的算法，并在ImageNet上进行了测试——我们非常高兴地发现我们的调整在ImageNet上也运行良好！

这里有一个重要的信息：你得到的数据集不一定是你想要的数据集。它尤其不可能是你想在其中进行开发和原型设计的数据集。你的目标应该是迭代速度不超过几分钟——也就是说，当你想出一个你想尝试的新想法时，你应该能够训练一个模型，并在几分钟内看看它是如何进行的。如果做一个实验需要更长的时间，想想如何减少你的数据集，或者简化你的模型，以提高你的实验速度。你能做的实验越多越好！ 让我们开始使用此数据集：

First we'll get our dataset into a DataLoaders object, using the presizing trick introduced in <<chapter_pet_breeds>>:

首先，我们将使用<>中引入的缩放技巧将我们的数据集放入DataLoaders对象中

and do a training run that will serve as a baseline:

并进行培训运行，作为基线：

That's a good baseline, since we are not using a pretrained model, but we can do better. When working with models that are being trained from scratch, or fine-tuned to a very different dataset than the one used for the pretraining, there are some additional techniques that are really important. In the rest of the chapter we'll consider some of the key approaches you'll want to be familiar with. The first one is normalizing your data.

这是一个很好的基线，因为我们没有使用预训练的模型，但是我们可以做得更好。当使用从头开始训练的模型时，或者微调到与预训练使用的数据集非常不同的数据集时，有一些额外的技术非常重要。在本章的其余部分，我们将介绍一些您想要熟悉的关键方法。第一个是规范化您的数据。

Normalization

规范化

When training a model, it helps if your input data is normalized—that is, has a mean of 0 and a standard deviation of 1. But most images and computer vision libraries use values between 0 and 255 for pixels, or between 0 and 1; in either case, your data is not going to have a mean of 0 and a standard deviation of 1.

Let's grab a batch of our data and look at those values, by averaging over all axes except for the channel axis, which is axis 1:

在训练模型时，如果您的输入数据是标准化的，即均值为0，均方差为1，这会有所帮助。但是大多数图像和计算机视觉库对像素使用0到255之间的值，或者0到1之间的值；在这两种情况下，您的数据都不会有0的均值和1的均方差。

让我们获取一批数据并通过在除通道轴（轴1）之外的所有轴上取平均值来查看这些值：

As we expected, the mean and standard deviation are not very close to the desired values. Fortunately, normalizing the data is easy to do in fastai by adding the Normalize transform. This acts on a whole mini-batch at once, so you can add it to the batch_tfms section of your data block. You need to pass to this transform the mean and standard deviation that you want to use; fastai comes with the standard ImageNet mean and standard deviation already defined. (If you do not pass any statistics to the Normalize transform, fastai will automatically calculate them from a single batch of your data.)

Let's add this transform (using imagenet_stats as Imagenette is a subset of ImageNet) and take a look at one batch now:

正如我们所预料的，均值和均方差并不非常接近所需的值。幸运的是，通过添加Normize变换，数据的标准化很容易在Fastai中完成。这一次会对整个小批次起作用，因此您可以将其添加到数据块的batch_tfms部分。您需要将您要使用的均值和均方差传递给此转换；Fastai附带已定义的标准ImageNet均值和方差。（如果您不将任何统计数据传递给Normaliz变换，Fastai将自动从您的单批次数据中计算它们。）

让我们添加这个转换（使用imagenet_stats因为Imagenette是ImageNet的一个子集），现在看看一个批次：

Let's check what effect this had on training our model:

让我们检查一下这对训练我们的模型有什么影响：

Although it only helped a little here, normalization becomes especially important when using pretrained models. The pretrained model only knows how to work with data of the type that it has seen before. If the average pixel value was 0 in the data it was trained with, but your data has 0 as the minimum possible value of a pixel, then the model is going to be seeing something very different to what is intended!

This means that when you distribute a model, you need to also distribute the statistics used for normalization, since anyone using it for inference, or transfer learning, will need to use the same statistics. By the same token, if you're using a model that someone else has trained, make sure you find out what normalization statistics they used, and match them.

We didn't have to handle normalization in previous chapters because when using a pretrained model through vision_learner, the fastai library automatically adds the proper Normalize transform; the model has been pretrained with certain statistics in Normalize (usually coming from the ImageNet dataset), so the library can fill those in for you. Note that this only applies with pretrained models, which is why we need to add this information manually here, when training from scratch.

All our training up until now has been done at size 224. We could have begun training at a smaller size before going to that. This is called progressive resizing.

尽管它在这里只起了一点作用，但在使用预训练模型时，规范化变得特别重要。预训练模型只知道如何处理它以前见过的类型的数据。如果它训练的数据中的平均像素值为0，但您的数据将0作为像素的最小可能值，那么模型将看到与预期非常不同的东西！

这意味着当您分发模型时，您还需要分发用于规范化的统计信息，因为任何使用它进行推理或迁移学习的人都需要使用相同的统计信息。同样，如果您使用的是其他人训练过的模型，请确保您找出他们使用的规范化统计信息，并与它们进行匹配。

我们不必在前面的章节中处理规范化，因为当通过vision_learner使用预训练模型时，fastai库会自动添加适当的规范化转换；模型已经在规范化中使用某些统计数据进行了预训练（通常来自ImageNet数据集），因此库可以为您填写这些数据。请注意，这仅适用于预训练模型，这就是为什么我们需要在从头开始训练时在此处手动添加此信息。

到目前为止，我们所有的训练都是在224尺寸上进行的。在开始之前，我们可以先从更小的尺寸开始训练。这叫做渐进式调整。

Progressive Resizing

渐进式调整

When fast.ai and its team of students won the DAWNBench competition in 2018, one of the most important innovations was something very simple: start training using small images, and end training using large images. Spending most of the epochs training with small images, helps training complete much faster. Completing training using large images makes the final accuracy much higher. We call this approach progressive resizing.

当fast.ai和它的学生团队在2018年赢得DAWNBench比赛时，最重要的创新之一是非常简单的事情：开始使用小图像进行训练，结束使用大图像进行训练。将大部分时间用小图像进行训练，有助于更快地完成训练。使用大图像完成训练使最终的准确性更高。我们称这种方法为渐进式调整大小。

jargon: progressive resizing: Gradually using larger and larger images as you train.

行话：渐进式调整大小：在训练时逐渐使用越来越大的图像。

As we have seen, the kinds of features that are learned by convolutional neural networks are not in any way specific to the size of the image—early layers find things like edges and gradients, and later layers may find things like noses and sunsets. So, when we change image size in the middle of training, it doesn't mean that we have to find totally different parameters for our model.

But clearly there are some differences between small images and big ones, so we shouldn't expect our model to continue working exactly as well, with no changes at all. Does this remind you of something? When we developed this idea, it reminded us of transfer learning! We are trying to get our model to learn to do something a little bit different from what it has learned to do before. Therefore, we should be able to use the fine_tune method after we resize our images.

There is an additional benefit to progressive resizing: it is another form of data augmentation. Therefore, you should expect to see better generalization of your models that are trained with progressive resizing.

To implement progressive resizing it is most convenient if you first create a get_dls function which takes an image size and a batch size as we did in the section before, and returns your DataLoaders:

Now you can create your DataLoaders with a small size and use fit_one_cycle in the usual way, training for a few less epochs than you might otherwise do:

正如我们所看到的，卷积神经网络学习的特征类型与图像的大小无关——早期的图层会发现边缘和梯度之类的东西，而后期的图层可能会发现鼻子和日落之类的东西。因此，当我们在训练过程中改变图像大小时，这并不意味着我们必须为我们的模型找到完全不同的参数。

但是很明显，小图像和大图像之间存在一些差异，所以我们不应该期望我们的模型也能完全一样地继续工作，一点变化都没有。这是否让你想起了什么？当我们提出这个想法时，它让我们想起了迁移学习！我们正试图让我们的模型学会做一些与它以前学会做的有点不同的事情。因此，我们应该能够在调整图像大小后使用fine_tune方法。

渐进式调整大小还有一个额外的好处：它是另一种形式的数据增强。因此，您应该期望看到使用渐进式调整大小训练的模型更好地泛化。 要实现渐进式调整大小，最方便的方法是首先创建一个get_dls函数，它接受图像大小和批处理大小，就像我们在前面部分所做的那样，并返回您的DataLoaders：

现在，您可以创建一个小尺寸的DataLoader，并以通常的方式使用fit_one_cycle，比其他方式少训练几个epoch：

Then you can replace the DataLoaders inside the Learner, and fine-tune:

然后您可以替换Learner中的DataLoaders，并微调：

As you can see, we're getting much better performance, and the initial training on small images was much faster on each epoch.

You can repeat the process of increasing size and training more epochs as many times as you like, for as big an image as you wish—but of course, you will not get any benefit by using an image size larger than the size of your images on disk.

Note that for transfer learning, progressive resizing may actually hurt performance. This is most likely to happen if your pretrained model was quite similar to your transfer learning task and dataset and was trained on similar-sized images, so the weights don't need to be changed much. In that case, training on smaller images may damage the pretrained weights.

On the other hand, if the transfer learning task is going to use images that are of different sizes, shapes, or styles than those used in the pretraining task, progressive resizing will probably help. As always, the answer to "Will it help?" is "Try it!"

Another thing we could try is applying data augmentation to the validation set. Up until now, we have only applied it on the training set; the validation set always gets the same images. But maybe we could try to make predictions for a few augmented versions of the validation set and average them. We'll consider this approach next.

正如你所看到的，我们的表现越来越好，小图像的初始训练在每个epoch都要快得多。

只要您愿意，您可以重复增加大小和训练更多epoch的过程。-但是，使用大于磁盘上图像大小的尺寸不会获得任何好处。 请注意，对于迁移学习，渐进式调整大小实际上可能会损害性能。如果您的预训练模型与您的迁移学习任务和数据集非常相似，并且在类似大小的图像上进行了训练，因此权重不需要改变太多，则最有可能发生这种情况。在这种情况下，在较小的图像上进行训练可能会损坏预训练的权重。

另一方面，如果迁移学习任务要使用与预训练任务中使用的图像不同大小、形状或样式的图像，渐进式调整大小可能会有所帮助。“会有帮助吗？”的答案是“试试看！”

我们可以尝试的另一件事是将数据增强应用于验证集。到目前为止，我们只将其应用于训练集；验证集总是得到相同的图像。但是也许我们可以尝试对验证集的一些增强版本进行预测并对其进行平均。我们接下来将考虑这种方法。

Test Time Augmentation

测试时间增强

We have been using random cropping as a way to get some useful data augmentation, which leads to better generalization, and results in a need for less training data. When we use random cropping, fastai will automatically use center cropping for the validation set—that is, it will select the largest square area it can in the center of the image, without going past the image's edges.

This can often be problematic. For instance, in a multi-label dataset sometimes there are small objects toward the edges of an image; these could be entirely cropped out by center cropping. Even for problems such as our pet breed classification example, it's possible that some critical feature necessary for identifying the correct breed, such as the color of the nose, could be cropped out.

One solution to this problem is to avoid random cropping entirely. Instead, we could simply squish or stretch the rectangular images to fit into a square space. But then we miss out on a very useful data augmentation, and we also make the image recognition more difficult for our model, because it has to learn how to recognize squished and squeezed images, rather than just correctly proportioned images.

Another solution is to not just center crop for validation, but instead to select a number of areas to crop from the original rectangular image, pass each of them through our model, and take the maximum or average of the predictions. In fact, we could do this not just for different crops, but for different values across all of our test time augmentation parameters. This is known as test time augmentation (TTA).

我们一直在使用随机裁剪来作为一种获得一些有用数据增强的方法，这会导致更好的泛化，并且需要更少的训练数据。当我们使用随机裁剪时，Fastai会自动将中心裁剪用于验证集——也就是说，它会选择图像中心最大的正方形区域，而不会超过图像的边缘。

这通常是有问题的。例如，在多标签数据集中，有时图像边缘有小对象；这些可以通过中心裁剪完全裁剪掉。即使对于像我们的宠物品种分类示例这样的问题，也可能会裁剪掉一些识别正确品种所需的关键特征，例如鼻子的颜色。

这个问题的一个解决方案是完全避免随机裁剪。相反，我们可以简单地挤压或拉伸矩形图像以适应方形空间。但这样我们就错过了一个非常有用的数据增强，我们也使图像识别对我们的模型来说更加困难，因为它必须学习如何识别压扁和挤压的图像，而不仅仅是正确比例的图像。

jargon: test time augmentation (TTA): During inference or validation, creating multiple versions of each image, using data augmentation, and then taking the average or maximum of the predictions for each augmented version of the image.

术语：测试时间增强（TTA）：在推断或验证期间，使用数据增强创建每个图像的多个版本，然后为每个增强版本的图像获取预测的平均值或最大值。

Depending on the dataset, test time augmentation can result in dramatic improvements in accuracy. It does not change the time required to train at all, but will increase the amount of time required for validation or inference by the number of test-time-augmented images requested. By default, fastai will use the unaugmented center crop image plus four randomly augmented images.

You can pass any DataLoader to fastai's tta method; by default, it will use your validation set:

根据数据集的不同，测试时间增强可以显着提高准确性。它根本不会改变训练所需的时间，但会通过请求的测试时间增强图像的数量来增加验证或推断所需的时间。默认情况下，Fastai将使用未增强的中心裁剪图像加上四个随机增强的图像。

您可以将任何DataLoader传递给Fastai的tta方法；默认情况下，它将使用您的验证集：

As we can see, using TTA gives us good a boost in performance, with no additional training required. However, it does make inference slower—if you're averaging five images for TTA, inference will be five times slower.

We've seen examples of how data augmentation helps train better models. Let's now focus on a new data augmentation technique called Mixup.

正如我们所看到的，使用TTA可以很好地提高性能，而无需额外的训练。然而，它确实会使推理变慢——如果您为TTA平均五张图像，推理将慢五倍。

我们已经看到了数据增强如何帮助训练更好模型的示例。现在让我们专注于一种名为Mixup的新数据增强技术。

Mixup

混合

Mixup, introduced in the 2017 paper "mixup: Beyond Empirical Risk Minimization" by Hongyi Zhang et al., is a very powerful data augmentation technique that can provide dramatically higher accuracy, especially when you don't have much data and don't have a pretrained model that was trained on data similar to your dataset. The paper explains: "While data augmentation consistently leads to improved generalization, the procedure is dataset-dependent, and thus requires the use of expert knowledge." For instance, it's common to flip images as part of data augmentation, but should you flip only horizontally, or also vertically? The answer is that it depends on your dataset. In addition, if flipping (for instance) doesn't provide enough data augmentation for you, you can't "flip more." It's helpful to have data augmentation techniques where you can "dial up" or "dial down" the amount of change, to see what works best for you.

Mixup works as follows, for each image:

1. Select another image from your dataset at random.
2. Pick a weight at random.
3. Take a weighted average (using the weight from step 2) of the selected image with your image; this will be your independent variable.
4. Take a weighted average (with the same weight) of this image's labels with your image's labels; this will be your dependent variable.

In pseudocode, we're doing this (where t is the weight for our weighted average):

image2,target2 = dataset[randint(0,len(dataset)]
t = random_float(0.5,1.0)
new_image = t * image1 + (1-t) * image2
new_target = t * target1 + (1-t) * target2

For this to work, our targets need to be one-hot encoded. The paper describes this using the equations shown in <> where $\lambda$ is the same as t in our pseudocode:

Mixup由张弘毅等人在2017年的论文《mixup：超越经验风险最小化》中介绍，是一种非常强大的数据增强技术，可以提供更高的准确性，尤其是当您没有太多数据并且没有预训练模型时，该模型是根据与您的数据集相似的数据训练的。该论文解释说：“虽然数据增强始终会导致提高泛化，但该过程依赖于数据集，因此需要使用知识。”例如，作为数据增强的一部分翻转图像是很常见的，但您应该只水平翻转还是垂直翻转？答案是这取决于您的数据集。此外，如果翻转（例如）不能为您提供足够的数据增强，您就不能“翻转更多”。拥有“向上”或“向下”调节变化数量的数据增强技术很有帮助，您可以看看什么最适合您。 对于每个图像，Mixup的工作原理如下： 从数据集中随机选择另一个图像。 随机选择一个权重。 将所选图像与您的图像进行加权平均（使用步骤2中的权重）；这将是您的自变量。 将此图像的标签与您的图像标签进行加权平均（权重相同）；这将是您的因变量。 在伪代码中，我们这样做（其中t是加权平均值的权重）：

image2,target2 = dataset[randint(0,len(dataset)] t = random_float(0.5,1.0) new_image = t * image1 + (1-t) * image2 new_target = t * target1 + (1-t) * target2

为此，我们的目标需要进行一热编码。本文使用<>中显示的方程来描述这一点，其中与我们的伪代码中的t相同：

Sidebar: Papers and Math

侧边栏：论文和数学

We're going to be looking at more and more research papers from here on in the book. Now that you have the basic jargon, you might be surprised to discover how much of them you can understand, with a little practice! One issue you'll notice is that Greek letters, such as $\lambda$, appear in most papers. It's a very good idea to learn the names of all the Greek letters, since otherwise it's very hard to read the papers to yourself, and remember them (or to read code based on them, since code often uses the names of the Greek letters spelled out, such as lambda).

The bigger issue with papers is that they use math, instead of code, to explain what's going on. If you don't have much of a math background, this will likely be intimidating and confusing at first. But remember: what is being shown in the math, is something that will be implemented in code. It's just another way of talking about the same thing! After reading a few papers, you'll pick up more and more of the notation. If you don't know what a symbol is, try looking it up in Wikipedia's list of mathematical symbols or drawing it in Detexify, which (using machine learning!) will find the name of your hand-drawn symbol. Then you can search online for that name to find out what it's for.

从这本书开始，我们将看到越来越多的研究论文。现在你已经了解了基本的行话，你可能会惊讶地发现，只要稍加练习，你就能理解其中的内容！你会注意到的一个问题是，希腊字母，如𝜆 ，出现在大多数论文中。学习所有希腊字母的名称是一个非常好的主意，否则很难自己阅读论文并记住它们（或者阅读基于它们的代码，因为代码通常使用拼写出来的希腊字母的名称，如lambda）。

论文更大的问题是它们使用数学而不是代码来解释发生了什么。如果你没有太多的数学背景，一开始这可能会让人害怕和困惑。但是请记住：数学中显示的是可以用代码实现的东西。这只是谈论同样的事情的另一种方式！阅读几篇论文后，你会学到越来越多的符号。如果你不知道符号是什么，试着在维基百科的数学符号列表中查找它，或者在Detexify中绘制它，它（使用机器学习！）会找到你手绘符号的名称。然后您可以在线搜索该名称以了解它的用途。

End sidebar

侧边栏结束

<<mixup_example>> shows what it looks like when we take a linear combination of images, as done in Mixup.

<>显示当我们获取图像的线性组合时的样子，如Mixup中所做的那样。

The third image is built by adding 0.3 times the first one and 0.7 times the second. In this example, should the model predict "church" or "gas station"? The right answer is 30% church and 70% gas station, since that's what we'll get if we take the linear combination of the one-hot-encoded targets. For instance, suppose we have 10 classes and "church" is represented by the index 2 and "gas station" is represented by the index 7, the one-hot-encoded representations are:

[0, 0, 1, 0, 0, 0, 0, 0, 0, 0] and [0, 0, 0, 0, 0, 0, 0, 1, 0, 0]

so our final target is:

[0, 0, 0.3, 0, 0, 0, 0, 0.7, 0, 0]

第三个图像是通过添加第一个的0.3倍和第二个的0.7倍来构建的。在这个例子中，模型应该预测“教堂”还是“加油站”？正确的答案是30%教堂和70%加油站，因为这是我们将得到的，如果我们采用one-hot-编码目标的线性组合。例如，假设我们有10个类，“教堂”由索引2表示，“加油站”由索引7表示，one-hot-编码表示为： [0,0,1,0,0,0,0,0,0]和[0,0,0,0,0,0,1,0,0] 所以我们的最终目标是： [0,0,0.3,0,0,0,0,0,0,0]

This all done for us inside fastai by adding a callback to our Learner. Callbacks are what is used inside fastai to inject custom behavior in the training loop (like a learning rate schedule, or training in mixed precision). We'll be learning all about callbacks, including how to make your own, in <<chapter_accel_sgd>>. For now, all you need to know is that you use the cbs parameter to Learner to pass callbacks.

Here is how we train a model with Mixup:

model = xresnet50(n_out=dls.c)
learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), 
                metrics=accuracy, cbs=MixUp())
learn.fit_one_cycle(5, 3e-3)

通过向我们的Learner添加回调，这一切都在fastai完成。回调是在fastai内部用于在训练循环中注入自定义行为的东西（如学习速率计划或混合精度训练）。我们将学习所有关于回调的知识，包括如何在<>中创建自己的回调。现在，您只需要知道您使用Learner的cbs参数来传递回调。

以下是我们如何使用Mixup训练模型：

model = xresnet50(n_out=dls.c) learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), metrics=accuracy, cbs=MixUp()) learn.fit_one_cycle(5, 3e-3)

What happens when we train a model with data that's "mixed up" in this way? Clearly, it's going to be harder to train, because it's harder to see what's in each image. And the model has to predict two labels per image, rather than just one, as well as figuring out how much each one is weighted. Overfitting seems less likely to be a problem, however, because we're not showing the same image in each epoch, but are instead showing a random combination of two images.

Mixup requires far more epochs to train to get better accuracy, compared to other augmentation approaches we've seen. You can try training Imagenette with and without Mixup by using the examples/train_imagenette.py script in the fastai repo. At the time of writing, the leaderboard in the Imagenette repo is showing that Mixup is used for all leading results for trainings of >80 epochs, and for fewer epochs Mixup is not being used. This is in line with our experience of using Mixup too.

One of the reasons that Mixup is so exciting is that it can be applied to types of data other than photos. In fact, some people have even shown good results by using Mixup on activations inside their models, not just on inputs—this allows Mixup to be used for NLP and other data types too.

There's another subtle issue that Mixup deals with for us, which is that it's not actually possible with the models we've seen before for our loss to ever be perfect. The problem is that our labels are 1s and 0s, but the outputs of softmax and sigmoid can never equal 1 or 0. This means training our model pushes our activations ever closer to those values, such that the more epochs we do, the more extreme our activations become.

With Mixup we no longer have that problem, because our labels will only be exactly 1 or 0 if we happen to "mix" with another image of the same class. The rest of the time our labels will be a linear combination, such as the 0.7 and 0.3 we got in the church and gas station example earlier.

One issue with this, however, is that Mixup is "accidentally" making the labels bigger than 0, or smaller than 1. That is to say, we're not explicitly telling our model that we want to change the labels in this way. So, if we want to make the labels closer to, or further away from 0 and 1, we have to change the amount of Mixup—which also changes the amount of data augmentation, which might not be what we want. There is, however, a way to handle this more directly, which is to use label smoothing.

当我们用这样“混合”的数据训练一个模型时会发生什么？显然，训练会更难，因为很难看到每张图像中的内容。模型必须预测每张图像的两个标签，而不仅仅是一个，并计算出每个标签的权重。然而，过度拟合似乎不太可能成为问题，因为我们在每个时代显示的不是相同的图像，而是显示两个图像的随机组合。

与我们见过的其他增强方法相比，Mixup需要训练更多的纪元才能获得更好的准确性。您可以尝试使用[快速存储库]（https://github.com/fastai/fastai）中的*示例/train_imagenette.py*脚本来训练有和没有Mixup的Imagenette。在撰写本文时，[Imagenette存储库]（https://github.com/fastai/imagenette/）中的排行榜显示，对于>80个纪元的训练，Mixup用于所有主要结果，对于更少的纪元，Mixup未被使用。这也符合我们使用Mixup的经验。

Mixup如此令人兴奋的原因之一是它可以应用于照片以外的数据类型。事实上，有些人甚至通过在模型内部的激活上使用Mixup来显示良好的结果，而不仅仅是在输入上——这允许Mixup也用于NLP和其他数据类型。

Mixup为我们处理了另一个微妙的问题，那就是我们以前见过的模型实际上不可能让我们的损失永远完美。问题是我们的标签是1和0，但是softmax和sigmoid的输出永远不能等于1或0。这意味着训练我们的模型会使我们的激活更接近这些值，这样我们做的时间越多，我们的激活就变得越极端。

使用Mixup，我们不再有这个问题，因为如果我们碰巧与同一类的另一张图像“混合”，我们的标签只会正好是1或0。其余时间，我们的标签将是线性组合，例如我们之前在教堂和加油站示例中得到的0.7和0.3。

然而，这样做的一个问题是，Mixup“意外地”让标签大于0或小于1。也就是说，我们没有明确地告诉我们的模型我们想以这种方式更改标签。因此，如果我们想让标签更接近或远离0和1，我们必须改变Mixup的数量——这也会改变数据增强的数量，这可能不是我们想要的。然而，有一种方法可以更直接地处理这个问题，那就是使用标签平滑。

Label Smoothing

标签平滑

In the theoretical expression of loss, in classification problems, our targets are one-hot encoded (in practice we tend to avoid doing this to save memory, but what we compute is the same loss as if we had used one-hot encoding). That means the model is trained to return 0 for all categories but one, for which it is trained to return 1. Even 0.999 is not "good enough", the model will get gradients and learn to predict activations with even higher confidence. This encourages overfitting and gives you at inference time a model that is not going to give meaningful probabilities: it will always say 1 for the predicted category even if it's not too sure, just because it was trained this way.

This can become very harmful if your data is not perfectly labeled. In the bear classifier we studied in <<chapter_production>>, we saw that some of the images were mislabeled, or contained two different kinds of bears. In general, your data will never be perfect. Even if the labels were manually produced by humans, they could make mistakes, or have differences of opinions on images that are harder to label.

Instead, we could replace all our 1s with a number a bit less than 1, and our 0s by a number a bit more than 0, and then train. This is called label smoothing. By encouraging your model to be less confident, label smoothing will make your training more robust, even if there is mislabeled data. The result will be a model that generalizes better.

This is how label smoothing works in practice: we start with one-hot-encoded labels, then replace all 0s with $\frac{\epsilon}{N}$ (that's the Greek letter epsilon, which is what was used in the paper that introduced label smoothing and is used in the fastai code), where $N$ is the number of classes and $\epsilon$ is a parameter (usually 0.1, which would mean we are 10% unsure of our labels). Since we want the labels to add up to 1, replace the 1 by $1-\epsilon + \frac{\epsilon}{N}$. This way, we don't encourage the model to predict something overconfidently. In our Imagenette example where we have 10 classes, the targets become something like (here for a target that corresponds to the index 3):

[0.01, 0.01, 0.01, 0.91, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01]

In practice, we don't want to one-hot encode the labels, and fortunately we won't need to (the one-hot encoding is just good to explain what label smoothing is and visualize it).

在损失的理论表达中，在分类问题中，我们的目标是一热编码（在实践中，我们倾向于避免这样做以节省内存，但我们计算的损失与我们使用一热编码时相同）。这意味着模型被训练为所有类别都返回0，只有一个类别被训练为返回1。即使0.999也不够好，模型也会得到梯度，并学会以更高的信心预测激活。这会鼓励过度拟合，并在推理时给你一个不会给出有意义的概率的模型：即使它不太确定，它也总是说预测类别为1，只是因为它是以这种方式训练的。

如果您的数据没有完美地标记，这可能会变得非常有害。在我们在<>中研究的熊分类器中，我们看到一些图像被错误地标记，或者包含两种不同种类的熊。一般来说，您的数据永远不会是完美的。即使标签是由人类手动制作的，他们也可能会出错，或者对更难标记的图像有意见分歧。

相反，我们可以用一个略小于1的数字替换所有的1，用一个略大于0的数字替换我们的0，然后进行训练。这称为标签平滑。通过鼓励您的模型不太自信，标签平滑将使您的训练更健壮，即使有错误标记的数据。结果将是一个泛化更好的模型。 这就是标签平滑在实践中的工作方式：我们从one-Hot编码的标签开始，然后将所有0替换为$\frac{\epsilon}{N}$（这是希腊字母epsilon，这是引入标签平滑的论文中使用的内容，并在快速代码中使用），其中是类的数量，是一个参数（通常为0.1，这意味着我们对我们的标签有10%的不确定性）。因为我们希望标签加起来为1，所以将1替换为$1-\epsilon + \frac{\epsilon}{N}$. 这样，我们就不会鼓励模型过于自信地预测某事。在我们的Imagenette示例中，我们有10个类，目标如下所示（这里是对应于索引3的目标）： [0.01,0.01,0.91,0.01,0.01,0.01,0.01,0.01] 在实践中，我们不想对标签进行一热编码，幸运的是我们不需要（一热编码很好地解释了什么是标签平滑并将其可视化）。

Sidebar: Label Smoothing, the Paper

侧边栏：标签平滑，论文

Here is how the reasoning behind label smoothing was explained in the paper by Christian Szegedy et al.:

: This maximum is not achievable for finite $z_k$ but is approached if $z_y\gg z_k$ for all $k\neq y$—that is, if the logit corresponding to the ground-truth label is much great than all other logits. This, however, can cause two problems. First, it may result in over-fitting: if the model learns to assign full probability to the ground-truth label for each training example, it is not guaranteed to generalize. Second, it encourages the differences between the largest logit and all others to become large, and this, combined with the bounded gradient $\frac{\partial\ell}{\partial z_k}$, reduces the ability of the model to adapt. Intuitively, this happens because the model becomes too confident about its predictions.

以下是Christian Szegedy等人在论文中解释标签平滑背后的推理： ：这个最大值对于有限的是无法实现的，但是如果 𝑧𝑦≫𝑧𝑘 所有的k≠y——也就是说，如果ground-truth对应的logit比所有其他logit都大得多，就会接近这个最大值。然而，这可能会导致两个问题。首先，它可能会导致过度拟合：如果模型学会为每个训练示例分配ground-truth的全部概率，它就不能保证泛化。其次，它鼓励最大logit和所有其他logit之间的差异变得很大，而这与有界梯度∂ℓ/∂𝑧𝑘相结合 , 降低了模型的适应能力。直观地说，这是因为模型对其预测变得过于自信。

Let's practice our paper-reading skills to try to interpret this. "This maximum" is refering to the previous part of the paragraph, which talked about the fact that 1 is the value of the label for the positive class. So it's not possible for any value (except infinity) to result in 1 after sigmoid or softmax. In a paper, you won't normally see "any value" written; instead it will get a symbol, which in this case is $z_k$. This shorthand is helpful in a paper, because it can be referred to again later and the reader will know what value is being discussed.

Then it says "if $z_y\gg z_k$ for all $k\neq y$." In this case, the paper immediately follows the math with an English description, which is handy because you can just read that. In the math, the $y$ is refering to the target ($y$ is defined earlier in the paper; sometimes it's hard to find where symbols are defined, but nearly all papers will define all their symbols somewhere), and $z_y$ is the activation corresponding to the target. So to get close to 1, this activation needs to be much higher than all the others for that prediction.

Next, consider the statement "if the model learns to assign full probability to the ground-truth label for each training example, it is not guaranteed to generalize." This is saying that making $z_y$ really big means we'll need large weights and large activations throughout our model. Large weights lead to "bumpy" functions, where a small change in input results in a big change to predictions. This is really bad for generalization, because it means just one pixel changing a bit could change our prediction entirely!

Finally, we have "it encourages the differences between the largest logit and all others to become large, and this, combined with the bounded gradient $\frac{\partial\ell}{\partial z_k}$, reduces the ability of the model to adapt." The gradient of cross-entropy, remember, is basically output - target. Both output and target are between 0 and 1, so the difference is between -1 and 1, which is why the paper says the gradient is "bounded" (it can't be infinite). Therefore our SGD steps are bounded too. "Reduces the ability of the model to adapt" means that it is hard for it to be updated in a transfer learning setting. This follows because the difference in loss due to incorrect predictions is unbounded, but we can only take a limited step each time.

让我们练习我们的论文阅读技巧来尝试解释这一点。“这个最大值”指的是段落的前一部分，其中谈到了1是正类标签的值。因此，任何值（无穷大除外）都不可能在sigmoid或softmax之后导致1。在论文中，您通常不会看到写入“任何值”；相反，它会得到一个符号，在本例中是𝑧𝑘。这个速记在论文中很有帮助，因为稍后可以再次引用它，读者将知道讨论的是什么值。

然后它会说“如果 𝑧𝑦≫𝑧𝑘 𝑘≠𝑦 ." 在这种情况下，论文会立即在数学后面加上英文描述，这很方便，因为你可以直接阅读。在数学中y，指的是目标（y在论文前面定义；有时很难找到符号的定义位置，但几乎所有论文都会在某个地方定义所有符号），Z𝑦是与目标对应的激活。因此，为了接近1，这个激活需要比该预测的所有其他激活高得多。

接下来，考虑一下“如果模型学会为每个训练示例分配完全概率给真值标签，则不能保证泛化。”这是说让Zy变得非常大意味着我们需要在整个模型中使用大权重和大激活。大权重会导致“颠簸”函数，其中输入的微小变化会导致预测的巨大变化。这对泛化真的很不利，因为这意味着只要一个像素稍微改变一点就可以完全改变我们的预测！ 最后，我们有“它鼓励最大logit和所有其他logit之间的差异变大，这与有界梯度相结合 ∂ℓ/∂𝑧𝑘 , 降低了模型的适应能力。”请记住，交叉熵的梯度基本上是输出-目标。输出和目标都在0和1之间，所以差异在-1和1之间，这就是为什么论文说梯度是“有界的”（它不可能是无限的）。因此我们的SGD步骤也是有界的。“降低模型的适应能力”意味着它很难在迁移学习环境中更新。这是因为不正确的预测导致的损失差异是无限的，但我们每次只能采取有限的步骤。

End sidebar

侧边栏结束

To use this in practice, we just have to change the loss function in our call to Learner:

model = xresnet50(n_out=dls.c)
learn = Learner(dls, model, loss_func=LabelSmoothingCrossEntropy(), 
                metrics=accuracy)
learn.fit_one_cycle(5, 3e-3)

Like with Mixup, you won't generally see significant improvements from label smoothing until you train more epochs. Try it yourself and see: how many epochs do you have to train before label smoothing shows an improvement?

要在实践中使用它，我们只需要在调用Learner时更改损失函数：

model = xresnet50(n_out=dls.c) learn = Learner(dls, model, loss_func=LabelSmoothingCrossEntropy(), metrics=accuracy) learn.fit_one_cycle(5, 3e-3)

与Mixup一样，在训练更多的epoch之前，您通常不会看到标签平滑的显着改进。亲自尝试一下，看看：在标签平滑显示改进之前，您需要训练多少个epoch？

Conclusion

总结

You have now seen everything you need to train a state-of-the-art model in computer vision, whether from scratch or using transfer learning. Now all you have to do is experiment on your own problems! See if training longer with Mixup and/or label smoothing avoids overfitting and gives you better results. Try progressive resizing, and test time augmentation.

Most importantly, remember that if your dataset is big, there is no point prototyping on the whole thing. Find a small subset that is representative of the whole, like we did with Imagenette, and experiment on it.

In the next three chapters, we will look at the other applications directly supported by fastai: collaborative filtering, tabular modeling and working with text. We will go back to computer vision in the next section of the book, with a deep dive into convolutional neural networks in <<chapter_convolutions>>.

您现在已经看到了在计算机视觉中训练最先进模型所需的一切，无论是从头开始还是使用迁移学习。现在您所要做的就是对自己的问题进行实验！看看使用Mixup和/或标签平滑进行更长时间的训练是否可以避免过度拟合并为您提供更好的结果。尝试渐进式调整大小和测试时间增加。

最重要的是，请记住，如果您的数据集很大，则没有必要对整个事物进行原型设计。找到一个代表整体的小子集，就像我们对Imagenette所做的那样，并对其进行实验。

在接下来的三章中，我们将看看Fastai直接支持的其他应用程序：协同过滤、表格建模和处理文本。我们将在本书的下一节回到计算机视觉，在<>中深入探讨卷积神经网络。

Questionnaire

1. What is the difference between ImageNet and Imagenette? When is it better to experiment on one versus the other?
2. What is normalization?
3. Why didn't we have to care about normalization when using a pretrained model?
4. What is progressive resizing?
5. Implement progressive resizing in your own project. Did it help?
6. What is test time augmentation? How do you use it in fastai?
7. Is using TTA at inference slower or faster than regular inference? Why?
8. What is Mixup? How do you use it in fastai?
9. Why does Mixup prevent the model from being too confident?
10. Why does training with Mixup for five epochs end up worse than training without Mixup?
11. What is the idea behind label smoothing?
12. What problems in your data can label smoothing help with?
13. When using label smoothing with five categories, what is the target associated with the index 1?
14. What is the first step to take when you want to prototype quick experiments on a new dataset?

Further Research

1. Use the fastai documentation to build a function that crops an image to a square in each of the four corners, then implement a TTA method that averages the predictions on a center crop and those four crops. Did it help? Is it better than the TTA method of fastai?
2. Find the Mixup paper on arXiv and read it. Pick one or two more recent articles introducing variants of Mixup and read them, then try to implement them on your problem.
3. Find the script training Imagenette using Mixup and use it as an example to build a script for a long training on your own project. Execute it and see if it helps.
4. Read the sidebar "Label Smoothing, the Paper", look at the relevant section of the original paper and see if you can follow it. Don't be afraid to ask for help!