Data Munging with fastai's Mid-Level API

用fastai的中间层API处理数据

We have seen what Tokenizer and Numericalize do to a collection of texts, and how they're used inside the data block API, which handles those transforms for us directly using the TextBlock. But what if we want to only apply one of those transforms, either to see intermediate results or because we have already tokenized texts? More generally, what can we do when the data block API is not flexible enough to accommodate our particular use case? For this, we need to use fastai's mid-level API for processing data. The data block API is built on top of that layer, so it will allow you to do everything the data block API does, and much much more.

我们已经看到了Tokenizer和Numericalize对文本集合的作用，以及它们是如何在数据块API内使用的，它直接使用TextBlock为我们处理这些转换。但是，如果我们只想应用这些转换中的一个，或者是为了看到中间的结果，或者是因为我们已经对文本进行了标记，那该怎么办？更广泛地说，当数据块API不够灵活以适应我们的特定用例时，我们可以做什么？为此，我们需要使用fastai的中间层API来处理数据。数据块API是建立在该层之上的，所以它将允许你做数据块API所做的一切，以及更多更多。

Going Deeper into fastai's Layered API

深入了解fastai的分层API

The fastai library is built on a layered API. In the very top layer there are applications that allow us to train a model in five lines of codes, as we saw in <<chapter_intro>>. In the case of creating DataLoaders for a text classifier, for instance, we used the line:

fastai库是建立在一个分层的API上。在最顶层，有一些应用程序允许我们用五行代码训练一个模型，正如我们在<>中看到的那样。例如，在为一个文本分类器创建DataLoaders的情况下，我们使用了这一行。

The factory method TextDataLoaders.from_folder is very convenient when your data is arranged the exact same way as the IMDb dataset, but in practice, that often won't be the case. The data block API offers more flexibility. As we saw in the last chapter, we can get the same result with:

当你的数据与IMDb数据集的排列方式完全相同时，工厂方法TextDataLoaders.from_folder是非常方便的，但在实践中，情况往往不会是这样。数据块API提供了更多的灵活性。正如我们在上一章中所看到的，我们可以通过以下方式得到同样的结果。

But it's sometimes not flexible enough. For debugging purposes, for instance, we might need to apply just parts of the transforms that come with this data block. Or we might want to create a DataLoaders for some application that isn't directly supported by fastai. In this section, we'll dig into the pieces that are used inside fastai to implement the data block API. Understanding these will enable you to leverage the power and flexibility of this mid-tier API.

但它有时不够灵活。例如，出于调试的目的，我们可能只需要应用这个数据块中的部分转换。或者我们可能想为一些不被fastai直接支持的应用程序创建一个DataLoaders。在本节中，我们将深入研究fastai内部用来实现数据块API的部分。了解这些将使你能够利用这个中间层API的能力和灵活性。

note: Mid-Level API: The mid-level API does not only contain functionality for creating DataLoaders. It also has the callback system, which allows us to customize the training loop any way we like, and the general optimizer. Both will be covered in <<chapter_accel_sgd>>.

注：中层API:中层API不仅包含创建DataLoaders的功能。它还有回调系统，它允许我们以任何方式定制训练循环，以及一般的优化器。两者都将在<<chapter_accel_sgd>>中介绍。

Transforms

变换

When we studied tokenization and numericalization in the last chapter, we started by grabbing a bunch of texts:

当我们在上一章研究标记化和数值化时，我们先抓了一堆文本：

We then showed how to tokenize them with a Tokenizer:

然后，我们展示了如何用一个Tokenizer对它们进行标记：

and how to numericalize, including automatically creating the vocab for our corpus:

以及如何将其数值化，包括为我们的语料库自动创建词汇:

The classes also have a decode method. For instance, Numericalize.decode gives us back the string tokens:

这些类也有一个decode。例如，Numericalize.decode给我们返回的是字符串标记:

and Tokenizer.decode turns this back into a single string (it may not, however, be exactly the same as the original string; this depends on whether the tokenizer is reversible, which the default word tokenizer is not at the time we're writing this book):

并且Tokenizer.decode将其转回一个单一的字符串（然而，它可能不完全与原始字符串相同；这取决于标记器是否是可逆的，在我们写这本书时，默认的单词标记器不是这样的）：

decode is used by fastai's show_batch and show_results, as well as some other inference methods, to convert predictions and mini-batches into a human-understandable representation.

For each of tok or num in the preceding example, we created an object, called the setup method (which trains the tokenizer if needed for tok and creates the vocab for num), applied it to our raw texts (by calling the object as a function), and then finally decoded the result back to an understandable representation. These steps are needed for most data preprocessing tasks, so fastai provides a class that encapsulates them. This is the Transform class. Both Tokenize and Numericalize are Transforms.

In general, a Transform is an object that behaves like a function and has an optional setup method that will initialize some inner state (like the vocab inside num) and an optional decode that will reverse the function (this reversal may not be perfect, as we saw with tok).

A good example of decode is found in the Normalize transform that we saw in <<chapter_sizing_and_tta>>: to be able to plot the images its decode method undoes the normalization (i.e., it multiplies by the standard deviation and adds back the mean). On the other hand, data augmentation transforms do not have a decode method, since we want to show the effects on images to make sure the data augmentation is working as we want.

A special behavior of Transforms is that they always get applied over tuples. In general, our data is always a tuple (input,target) (sometimes with more than one input or more than one target). When applying a transform on an item like this, such as Resize, we don't want to resize the tuple as a whole; instead, we want to resize the input (if applicable) and the target (if applicable) separately. It's the same for batch transforms that do data augmentation: when the input is an image and the target is a segmentation mask, the transform needs to be applied (the same way) to the input and the target.

We can see this behavior if we pass a tuple of texts to tok:

decode被fastai的show_batch和show_results以及其他一些推理方法所使用，用于将预测和小型批次转换为人类可理解的表示。

对于前面例子中的tok或num，我们创建了一个对象，调用setup方法（如果tok需要的话，训练标记器，num则创建词汇表），将其应用于我们的原始文本（通过调用对象作为一个函数），然后最后将结果解码为可理解的表示。大多数数据预处理任务都需要这些步骤，所以fastai提供了一个封装这些步骤的类。这就是Transform类。Tokenize和Numericalize都是Transforms。

一般来说，Transform是一个行为类似于函数的对象，它有一个可选的setup方法，将初始化一些内部状态（比如num里面的词汇），还有一个可选的decode，将反转函数（这个反转可能并不完美，正如我们在tok中看到的那样）。

decode的一个很好的例子是我们在<>中看到的Normalize变换：为了能够绘制图像，它的解码方法取消了归一化（即，它乘以标准差并加回平均值）。另一方面，数据增强变换没有decode方法，因为我们想显示对图像的影响，以确保数据增强是按我们的要求进行的。

Transforms的一个特殊行为是它们总是被应用于元组。一般来说，我们的数据总是一个元组（input,target）（有时有多个输入或多个目标）。当在这样的项目上应用变换时，比如Resize，我们不想把元组作为一个整体来调整；相反，我们想分别调整输入（如果适用）和目标（如果适用）的大小。这对于做数据增强的批量变换也是一样的：当输入是一个图像，目标是一个分割掩码时，变换需要（以同样的方式）应用于输入和目标。

如果我们把一个文本的元组传递给tok，我们就可以看到这种行为:

Writing Your Own Transform

编写你自己的变换

If you want to write a custom transform to apply to your data, the easiest way is to write a function. As you can see in this example, a Transform will only be applied to a matching type, if a type is provided (otherwise it will always be applied). In the following code, the :int in the function signature means that f only gets applied to ints. That's why tfm(2.0) returns 2.0, but tfm(2) returns 3 here:

如果你想写一个自定义的变换来应用于你的数据，最简单的方法是写一个函数。正如你在这个例子中看到的，如果提供了一个类型，Transform将只应用于一个匹配的类型（否则它将一直被应用）。在下面的代码中，函数签名中的:int意味着f只被应用于ints。这就是为什么tfm(2.0)返回2.0，但tfm(2)在这里返回3。

Here, f is converted to a Transform with no setup and no decode method.

Python has a special syntax for passing a function (like f) to another function (or something that behaves like a function, known as a callable in Python), called a decorator. A decorator is used by prepending a callable with @ and placing it before a function definition (there are lots of good online tutorials about Python decorators, so take a look at one if this is a new concept for you). The following is identical to the previous code:

在这里，f被转换为一个没有setup也没有decode方法的Transform。

Python 有一种特殊的语法，用于将一个函数 (如 f) 传递给另一个函数 (或行为类似于函数的东西，在 Python 中被称为可调用函数)，称为装饰器。装饰器的使用方法是在可调用函数前加@，并把它放在函数定义之前 (关于Python装饰器有很多很好的在线教程，如果这对你来说是个新概念，可以看一看)。下面的代码与前面的代码相同:

If you need either setup or decode, you will need to subclass Transform to implement the actual encoding behavior in encodes, then (optionally), the setup behavior in setups and the decoding behavior in decodes:

如果你需要setup或decode，你将需要对Transform进行子类化，在encodes中实现实际的编码行为，然后（可选择）在setups中实现设置行为，在decodes中实现解码行为。

Here, NormalizeMean will initialize some state during the setup (the mean of all elements passed), then the transformation is to subtract that mean. For decoding purposes, we implement the reverse of that transformation by adding the mean. Here is an example of NormalizeMean in action:

在这里，NormalizeMean将在设置过程中初始化一些状态（所有传递的元素的平均值），然后变换是减去这个平均值。出于解码的目的，我们通过添加平均值来实现该转换的反向。下面是一个NormalizeMean的操作例子：

Note that the method called and the method implemented are different, for each of these methods:

[options="header"]
|======
| Class | To call | To implement
| `nn.Module` (PyTorch) | `()` (i.e., call as function) | `forward`
| `Transform` | `()` | `encodes`
| `Transform` | `decode()` | `decodes`
| `Transform` | `setup()` | `setups`
|======

So, for instance, you would never call setups directly, but instead would call setup. The reason for this is that setup does some work before and after calling setups for you. To learn more about Transforms and how you can use them to implement different behavior depending on the type of the input, be sure to check the tutorials in the fastai docs.

请注意，对于这些方法中的每一个，调用的方法和实现的方法是不同的:

[options="header"]
|======
| Class | To call | To implement
| `nn.Module` (PyTorch) | `()` (i.e., call as function) | `forward`
| `Transform` | `()` | `encodes`
| `Transform` | `decode()` | `decodes`
| `Transform` | `setup()` | `setups`
|======

因此，举例来说，你永远不会直接调用setups，而是调用setup。原因是setup在调用setups之前和之后为你做一些工作。要了解更多关于Transforms的信息，以及如何使用它们来实现取决于输入类型的不同行为，请务必查看fastai文档中的教程。

Pipeline

流水线

To compose several transforms together, fastai provides the Pipeline class. We define a Pipeline by passing it a list of Transforms; it will then compose the transforms inside it. When you call Pipeline on an object, it will automatically call the transforms inside, in order:

为了将几个变换组合在一起，fastai提供了Pipeline类。我们通过传递一个Transforms列表来定义一个Pipeline；然后它将在其中组合这些变换。当你在一个对象上调用Pipeline时，它将自动按顺序调用里面的变换。

And you can call decode on the result of your encoding, to get back something you can display and analyze:

你可以对你的编码结果调用decode，以得到你可以显示和分析的东西:

The only part that doesn't work the same way as in Transform is the setup. To properly set up a Pipeline of Transforms on some data, you need to use a TfmdLists.

唯一与Transform中的工作方式不同的部分是设置。要在一些数据上正确地设置一个Transform的Pipeline，你需要使用TfmdLists。

TfmdLists and Datasets: Transformed Collections

TfmdLists和数据集:变换后的集合

Your data is usually a set of raw items (like filenames, or rows in a DataFrame) to which you want to apply a succession of transformations. We just saw that a succession of transformations is represented by a Pipeline in fastai. The class that groups together this Pipeline with your raw items is called TfmdLists.

你的数据通常是一组原始项目（如文件名，或DataFrame中的行），你想对其应用一系列的变换。我们刚刚看到，一连串的转换是由fastai中的Pipeline表示的。将这个 Pipeline和你的原始项目组合在一起的类叫做TfmdLists。

TfmdLists

Here is the short way of doing the transformation we saw in the previous section:

下面是我们在上一节中看到的进行变换的简短方法:

At initialization, the TfmdLists will automatically call the setup method of each Transform in order, providing them not with the raw items but the items transformed by all the previous Transforms in order. We can get the result of our Pipeline on any raw element just by indexing into the TfmdLists:

在初始化时，TfmdLists会自动按顺序调用每个Transform的setup方法，为它们提供的不是原始项目，而是按顺序由所有先前的Transform变换的项目。我们可以通过对TfmdLists的索引来获得我们在任何原始元素上的管道结果:

And the TfmdLists knows how to decode for show purposes:

而TfmdLists知道如何解码以达到展示目的:

In fact, it even has a show method:

事实上，它甚至有一个show的方法。

The TfmdLists is named with an "s" because it can handle a training and a validation set with a splits argument. You just need to pass the indices of which elements are in the training set, and which are in the validation set:

TfmdLists以"s"命名，因为它可以用一个splits参数处理训练集和验证集。你只需要传递哪些元素在训练集里，哪些在验证集里的索引。

You can then access them through the train and valid attributes:

然后你可以通过train和valid属性来访问它们:

If you have manually written a Transform that performs all of your preprocessing at once, turning raw items into a tuple with inputs and targets, then TfmdLists is the class you need. You can directly convert it to a DataLoaders object with the dataloaders method. This is what we will do in our Siamese example later in this chapter.

In general, though, you will have two (or more) parallel pipelines of transforms: one for processing your raw items into inputs and one to process your raw items into targets. For instance, here, the pipeline we defined only processes the raw text into inputs. If we want to do text classification, we also have to process the labels into targets.

For this we need to do two things. First we take the label name from the parent folder. There is a function, parent_label, for this:

如果你手动编写了一个Transform，一次性执行所有的预处理，把原始项目变成一个带有输入和目标的元组，那么TfmdLists就是你需要的类。你可以用dataloaders方法直接把它转换成DataLoaders对象。这就是我们在本章后面的Siamese例子中要做的。

不过一般来说，你会有两条（或更多）平行的变换管道：一条用于将你的原始项目处理成输入，一条用于将你的原始项目处理成目标。例如，在这里，我们定义的管道只将原始文本处理成输入。如果我们想做文本分类，我们还必须将标签处理成目标。

Then we need a Transform that will grab the unique items and build a vocab with them during setup, then transform the string labels into integers when called. fastai provides this for us; it's called Categorize:

然后我们需要一个Transform，在设置过程中抓取独特的项并建立一个词汇表，然后在调用时将字符串标签转换为整数。fastai为我们提供了这个;它被称为Categorize:

To do the whole setup automatically on our list of files, we can create a TfmdLists as before:

为了在我们的文件列表上自动进行整个设置，我们可以像以前一样创建一个TfmdLists:

But then we end up with two separate objects for our inputs and targets, which is not what we want. This is where Datasets comes to the rescue.

但这样一来，我们的输入和目标就会出现两个独立的对象，这不是我们想要的。这就是Datasets的用武之地。

Datasets

数据集

Datasets will apply two (or more) pipelines in parallel to the same raw object and build a tuple with the result. Like TfmdLists, it will automatically do the setup for us, and when we index into a Datasets, it will return us a tuple with the results of each pipeline:

Datasets将对同一个原始对象并行地应用两个（或多个）管道，并将结果建立一个元组。像TfmdLists一样，它将自动为我们进行设置，当我们对Datasets进行索引时，它将返回一个包含每个管道结果的元组。

Like a TfmdLists, we can pass along splits to a Datasets to split our data between training and validation sets:

像TfmdLists一样，我们可以将splits 传递给Datasets，在训练集和验证集之间分割我们的数据:

It can also decode any processed tuple or show it directly:

它还可以对任何处理过的元组进行解码或直接显示:

The last step is to convert our Datasets object to a DataLoaders, which can be done with the dataloaders method. Here we need to pass along a special argument to take care of the padding problem (as we saw in the last chapter). This needs to happen just before we batch the elements, so we pass it to before_batch:

最后一步是将我们的Datasets对象转换为DataLoaders，这可以通过dataloaders方法完成。这里我们需要传递一个特殊的参数来处理填充问题（正如我们在上一章中看到的）。这需要在我们批处理元素之前发生，所以我们把它传给before_batch:

dataloaders directly calls DataLoader on each subset of our Datasets. fastai's DataLoader expands the PyTorch class of the same name and is responsible for collating the items from our datasets into batches. It has a lot of points of customization, but the most important ones that you should know are:

• after_item:: Applied on each item after grabbing it inside the dataset. This is the equivalent of item_tfms in DataBlock.
• before_batch:: Applied on the list of items before they are collated. This is the ideal place to pad items to the same size.
• after_batch:: Applied on the batch as a whole after its construction. This is the equivalent of batch_tfms in DataBlock.

dataloaders直接调用DataLoader在我们的Datasets的每个子集上。 fastai的DataLoader扩展了同名的PyTorch类，负责将我们的数据集的项目整理成批。它有很多定制点，但最重要的是你应该知道:

• after_item:: 在数据集中抓取每个项目后应用。这相当于DataBlock中的item_tfms。

• before_batch:: 应用于整理前的项目列表。这是一个理想的地方，可以将项目填充到相同的尺寸。

• after_batch:: 在批处理构建后，应用于整个批处理。这相当于DataBlock中的batch_tfms。

As a conclusion, here is the full code necessary to prepare the data for text classification:

作为结论，这里是为文本分类准备数据所需的全部代码:

The two differences from the previous code are the use of GrandparentSplitter to split our training and validation data, and the dl_type argument. This is to tell dataloaders to use the SortedDL class of DataLoader, and not the usual one. SortedDL constructs batches by putting samples of roughly the same lengths into batches.

This does the exact same thing as our previous DataBlock:

与之前代码的两个不同点是使用GrandparentSplitter来分割我们的训练和验证数据，以及dl_type参数。这是为了告诉dataloaders使用DataLoader的SortedDL类，而不是通常的。SortedDL通过将长度大致相同的样本放入批次来构造批次。

这和我们之前的DataBlock做的事情完全一样:

But now, you know how to customize every single piece of it!

Let's practice what we just learned about this mid-level API for data preprocessing, using a computer vision example now.

但现在，你知道如何定制它的每一个部分了!

让我们练习一下我们刚刚学到的关于这个数据预处理的中级API，现在用一个计算机视觉的例子。

Applying the Mid-Level Data API: SiamesePair

应用中间层的数据API:SiamesePair

A Siamese model takes two images and has to determine if they are of the same class or not. For this example, we will use the Pet dataset again and prepare the data for a model that will have to predict if two images of pets are of the same breed or not. We will explain here how to prepare the data for such a model, then we will train that model in <<chapter_arch_details>>.

First things first, let's get the images in our dataset:

一个Siamese模型需要两张图片，并必须确定它们是否属于同一类别。在这个例子中，我们将再次使用宠物数据集，为一个模型准备数据，该模型必须预测两张宠物图片是否属于同一品种。我们将在这里解释如何为这样一个模型准备数据，然后我们将在<<chapter_arch_details>>中训练这个模型。

首先，让我们在我们的数据集中获取图像:

If we didn't care about showing our objects at all, we could directly create one transform to completely preprocess that list of files. We will want to look at those images though, so we need to create a custom type. When you call the show method on a TfmdLists or a Datasets object, it will decode items until it reaches a type that contains a show method and use it to show the object. That show method gets passed a ctx, which could be a matplotlib axis for images, or a row of a DataFrame for texts.

Here we create a SiameseImage object that subclasses fastuple and is intended to contain three things: two images, and a Boolean that's True if the images are of the same breed. We also implement the special show method, such that it concatenates the two images with a black line in the middle. Don't worry too much about the part that is in the if test (which is to show the SiameseImage when the images are Python images, not tensors); the important part is in the last three lines:

如果我们根本不关心显示我们的对象，我们可以直接创建一个变换来完全预处理那个文件列表。不过我们会想看看这些图片，所以我们需要创建一个自定义类型。当你在TfmdLists或Datasets对象上调用show方法时，它将对项目进行解码，直到到达一个包含show方法的类型，并使用它来显示对象。这个显示方法会被传递给一个ctx，对于图像来说，ctx可以是matplotlib的一个轴，对于文本来说，ctx可以是DataFrame的一行。

在这里，我们创建了一个SiameseImage对象，该对象子类为fastuple，目的是包含三样东西：两张图片，以及一个布尔值，如果图片是同一类的，则为True。我们还实现了特殊的show方法，这样它就把两张图片用一条黑线串联起来。不要太担心在if测试中的部分（即当图像是Python图像而不是张量时显示SiameseImage）；重要的部分在最后三行:

Let's create a first SiameseImage and check our show method works:

让我们创建第一个SiameseImage并检查我们的show方法是否有效:

We can also try with a second image that's not from the same class:

我们也可以用第二张不属于同一类别的图片来试试:

The important thing with transforms that we saw before is that they dispatch over tuples or their subclasses. That's precisely why we chose to subclass fastuple in this instance—this way we can apply any transform that works on images to our SiameseImage and it will be applied on each image in the tuple:

我们之前看到的变换的重要之处在于，它们在元组或其子类上进行调度。这正是我们在这个例子中选择子类fastuple的原因--这样我们就可以将任何适用于图像的变换应用于我们的SiameseImage，并且它将被应用于元组中的每张图像:

Here the Resize transform is applied to each of the two images, but not the Boolean flag. Even if we have a custom type, we can thus benefit from all the data augmentation transforms inside the library.

We are now ready to build the Transform that we will use to get our data ready for a Siamese model. First, we will need a function to determine the classes of all our images:

在这里，调整大小的变换被应用于两张图片中的每一张，但不是布尔标志。即使我们有一个自定义的类型，我们也可以因此受益于库内所有的数据增强变换。

我们现在已经准备好构建我们将用来为Siamese模型准备好数据的Transform。首先，我们将需要一个函数来确定我们所有图像的类别:

For each image our tranform will, with a probability of 0.5, draw an image from the same class and return a SiameseImage with a true label, or draw an image from another class and return a SiameseImage with a false label. This is all done in the private _draw function. There is one difference between the training and validation sets, which is why the transform needs to be initialized with the splits: on the training set we will make that random pick each time we read an image, whereas on the validation set we make this random pick once and for all at initialization. This way, we get more varied samples during training, but always the same validation set:

对于每张图片，我们的Tranform将以0.5的概率从同一类别中抽取一张图片，并返回一个带有真实标签的SiameseImage，或者从另一个类别中抽取一张图片，并返回一个带有错误标签的SiameseImage。这些都是在私有的_draw函数中完成的。训练集和验证集之间有一个区别，这就是为什么Transform需要用分片来初始化：在训练集上，我们将在每次读取图像时进行随机抽取，而在验证集上，我们在初始化时一次性地进行这种随机抽取。这样一来，我们在训练过程中就能得到更多不同的样本，但始终是同一个验证集。

We can then create our main transform:

然后，我们可以创建我们的主变换:

In the mid-level API for data collection we have two objects that can help us apply transforms on a set of items, TfmdLists and Datasets. If you remember what we have just seen, one applies a Pipeline of transforms and the other applies several Pipelines of transforms in parallel, to build tuples. Here, our main transform already builds the tuples, so we use TfmdLists:

在数据收集的中层API中，我们有两个对象可以帮助我们在一组项目上应用变换，即TfmdLists和Datasets。如果你还记得我们刚刚看到的，一个是应用变换的管道，另一个是平行地应用几个变换的管道，以建立图元。在这里，我们的主转换已经建立了图元，所以我们使用TfmdLists：

And we can finally get our data in DataLoaders by calling the dataloaders method. One thing to be careful of here is that this method does not take item_tfms and batch_tfms like a DataBlock. The fastai DataLoader has several hooks that are named after events; here what we apply on the items after they are grabbed is called after_item, and what we apply on the batch once it's built is called after_batch:

而我们最终可以通过调用dataloaders方法在DataLoaders中获得我们的数据。这里需要注意的一点是，这个方法不像DataBlock那样接受item_tfms和batch_tfms。fastai DataLoader有几个以事件命名的钩子；在这里，我们在抓取项目后应用的东西被称为after_item，而我们在批处理建立后应用的东西被称为after_batch：

Note that we need to pass more transforms than usual—that's because the data block API usually adds them automatically:

• ToTensor is the one that converts images to tensors (again, it's applied on every part of the tuple).
• IntToFloatTensor converts the tensor of images containing integers from 0 to 255 to a tensor of floats, and divides by 255 to make the values between 0 and 1.

请注意，我们需要传递比平时更多的变换--这是因为数据块API通常会自动添加这些变换：

• ToTensor是将图像转换为张量（同样，它适用于元组的每个部分）。
• IntToFloatTensor将包含0到255的整数的图像张量转换为浮点数的张量，然后除以255，使数值在0和1之间。

We can now train a model using this DataLoaders. It will need a bit more customization than the usual model provided by vision_learner since it has to take two images instead of one, but we will see how to create such a model and train it in <<chapter_arch_dtails>>.

我们现在可以使用这个DataLoaders训练一个模型。与vision_learner提供的通常模型相比，它需要更多的定制，因为它必须接受两张图片而不是一张，但我们将在<<chapter_arch_dtails>>中看到如何创建这样一个模型并训练它。

Conclusion

结论

fastai provides a layered API. It takes one line of code to grab the data when it's in one of the usual settings, making it easy for beginners to focus on training a model without spending too much time assembling the data. Then, the high-level data block API gives you more flexibility by allowing you to mix and match some building blocks. Underneath it, the mid-level API gives you greater flexibility to apply any transformations on your items. In your real-world problems, this is probably what you will need to use, and we hope it makes the step of data-munging as easy as possible.

fastai提供了一个分层的API。当数据处于通常的设置中时，只需要一行代码就可以抓取数据，这使得初学者很容易专注于训练一个模型，而不需要花太多的时间来组装数据。然后，高层数据块API给你更多的灵活性，允许你混合和匹配一些构建块。在它的下面，中级的API给你更大的灵活性，可以在你的项目上应用任何变换。在你的实际问题中，这可能是你需要使用的，我们希望它能使数据拼装这一步尽可能的简单。

Questionnaire

调查问卷

1. Why do we say that fastai has a "layered" API? What does it mean?
2. Why does a Transform have a decode method? What does it do?
3. Why does a Transform have a setup method? What does it do?
4. How does a Transform work when called on a tuple?
5. Which methods do you need to implement when writing your own Transform?
6. Write a Normalize transform that fully normalizes items (subtract the mean and divide by the standard deviation of the dataset), and that can decode that behavior. Try not to peek!
7. Write a Transform that does the numericalization of tokenized texts (it should set its vocab automatically from the dataset seen and have a decode method). Look at the source code of fastai if you need help.
8. What is a Pipeline?
9. What is a TfmdLists?
10. What is a Datasets? How is it different from a TfmdLists?
11. Why are TfmdLists and Datasets named with an "s"?
12. How can you build a DataLoaders from a TfmdLists or a Datasets?
13. How do you pass item_tfms and batch_tfms when building a DataLoaders from a TfmdLists or a Datasets?
14. What do you need to do when you want to have your custom items work with methods like show_batch or show_results?
15. Why can we easily apply fastai data augmentation transforms to the SiamesePair we built?

1. 为什么我们说fastai有一个"layered"的API？它是什么意思？
2. 为什么一个Transform有一个decode方法？它的作用是什么？
3. 为什么Transform有一个setup方法？它的作用是什么？
4. 当在一个元组上调用Transform时，它是如何工作的？
5. 在编写你自己的Transform时，你需要实现哪些方法？
6. 编写一个Normalize变换，使项目完全正常化（减去数据集的平均数并除以标准差），并1. 能对该行为进行解码。尽量不要偷看!
7. 编写一个Transform，对标记化的文本进行数值化处理（它应该根据所看到的数据集自动设置其词汇，并有一个decode方法）。如果你需要帮助，请看fastai的源代码。
8. 什么是Pipeline？
9. 什么是TfmdLists？
10. 什么是Datasets？它与TfmdLists有什么不同？
11. 为什么TfmdLists和Datasets以 "s "命名？
12. 如何从TfmdLists或Datasets建立一个DataLoaders？
13. 当从TfmdLists或Datasets建立DataLoaders时，如何传递item_tfms和batch_tfms？
14. 当你想让你的自定义项目与show_batch或show_results等方法一起工作时，你需要做什么？
15. 为什么我们可以很容易地将fastai数据增强转换应用于我们建立的SiamesePair？

Further Research

进一步调查

1. Use the mid-level API to prepare the data in DataLoaders on your own datasets. Try this with the Pet dataset and the Adult dataset from Chapter 1.
2. Look at the Siamese tutorial in the fastai documentation to learn how to customize the behavior of show_batch and show_results for new type of items. Implement it in your own project.

1. 使用中层API在你自己的数据集上准备DataLoaders中的数据。用第一章中的宠物数据集和成人数据集试试。
2. 看看fastai文档中的Siamese教程，学习如何为新类型的项目定制show_batch和show_results。在你自己的项目中实施它。

Understanding fastai's Applications: Wrap Up

了解fastai的应用:总结

Congratulations—you've completed all of the chapters in this book that cover the key practical parts of training models and using deep learning! You know how to use all of fastai's built-in applications, and how to customize them using the data block API and loss functions. You even know how to create a neural network from scratch, and train it! (And hopefully you now know some of the questions to ask to make sure your creations help improve society too.)

The knowledge you already have is enough to create full working prototypes of many types of neural network applications. More importantly, it will help you understand the capabilities and limitations of deep learning models, and how to design a system that's well adapted to them.

In the rest of this book we will be pulling apart those applications, piece by piece, to understand the foundations they are built on. This is important knowledge for a deep learning practitioner, because it is what allows you to inspect and debug models that you build and create new applications that are customized for your particular projects.

祝贺你--你已经完成了本书的所有章节，这些章节涵盖了训练模型和使用深度学习的关键实践部分 你知道如何使用fastai的所有内置应用程序，以及如何使用数据块API和损失函数来定制它们。你甚至知道如何从头开始创建一个神经网络，并对其进行训练 (希望你现在知道一些要问的问题，以确保你的创作也有助于改善社会）。

你已经掌握的知识足以为许多类型的神经网络应用创建完整的工作原型。更重要的是，它将帮助你了解深度学习模型的能力和限制，以及如何设计一个很好地适应它们的系统。

在本书的其余部分，我们将把这些应用一块一块地拆开，以了解它们所建立的基础。这对一个深度学习从业者来说是很重要的知识，因为它可以让你检查和调试你建立的模型，并创建为你的特定项目定制的新应用程序。