A Language Model from Scratch

一个来自Scratch的语言模型

We're now ready to go deep... deep into deep learning! You already learned how to train a basic neural network, but how do you go from there to creating state-of-the-art models? In this part of the book we're going to uncover all of the mysteries, starting with language models.

You saw in <<chapter_nlp>> how to fine-tune a pretrained language model to build a text classifier. In this chapter, we will explain to you what exactly is inside that model, and what an RNN is. First, let's gather some data that will allow us to quickly prototype our various models.

我们现在准备好深入...深入深度学习！您已经学习了如何训练基本的神经网络，但是您如何从那里开始创建最先进的模型？在本书的这一部分，我们将揭开所有的谜团，从语言模型开始。 您在<>中看到了如何微调预训练的语言模型以构建文本分类器。在本章中，我们将向您解释该模型内部到底是什么，以及RNN是什么。首先，让我们收集一些数据，使我们能够快速原型化我们的各种模型。

The Data

数据

Whenever we start working on a new problem, we always first try to think of the simplest dataset we can that will allow us to try out methods quickly and easily, and interpret the results. When we started working on language modeling a few years ago we didn't find any datasets that would allow for quick prototyping, so we made one. We call it Human Numbers, and it simply contains the first 10,000 numbers written out in English.

每当我们开始处理一个新问题时，我们总是首先尝试想出我们能想到的最简单的数据集，这将使我们能够快速轻松地尝试我们的方法，并解释结果。当我们几年前开始研究语言建模时，我们没有找到任何可以快速原型化的数据集，所以我们做了一个。我们称之为人类数字，它只是包含用英语写出来的前10,000个数字。

j: One of the most common practical mistakes I see even amongst highly experienced practitioners is failing to use appropriate datasets at appropriate times during the analysis process. In particular, most people tend to start with datasets that are too big and too complicated.

J：即使在经验丰富的从业者中，我也看到最常见的实际错误之一是在分析过程中未能在适当的时间使用适当的数据集。特别是，大多数人倾向于从太大太复杂的数据集开始。

We can download, extract, and take a look at our dataset in the usual way:

我们可以以通常的方式下载、提取和查看我们的数据集：

Let's open those two files and see what's inside. At first we'll join all of the texts together and ignore the train/valid split given by the dataset (we'll come back to that later):

让我们打开这两个文件，看看里面有什么。首先，我们将所有文本连接在一起，忽略数据集给出的训练集和验证集。（我们稍后会回到这个问题）：

We take all those lines and concatenate them in one big stream. To mark when we go from one number to the next, we use a . as a separator:

我们将所有这些线连接在一个大流中。当我们从一个数字到下一个数字时，我们使用.作为分隔符：

We can tokenize this dataset by splitting on spaces:

我们可以使用空格划分此数据集：

To numericalize, we have to create a list of all the unique tokens (our vocab):

要进行数值化，我们必须创建包含所有唯一令牌（我们的词汇表）的列表：

Then we can convert our tokens into numbers by looking up the index of each in the vocab:

然后我们可以通过在词汇表中查找每个词的索引将我们的标记转换为数字：

Now that we have a small dataset on which language modeling should be an easy task, we can build our first model.

现在我们有了一个小的数据集，语言模型化应该是一项简单的任务，我们可以构建我们的第一个模型。

Our First Language Model from Scratch

我们从零开始的第一个语言模型

One simple way to turn this into a neural network would be to specify that we are going to predict each word based on the previous three words. We could create a list of every sequence of three words as our independent variables, and the next word after each sequence as the dependent variable.

We can do that with plain Python. Let's do it first with tokens just to confirm what it looks like:

将其转化为神经网络的一个简单方法是指定我们将根据前三个单词预测每个单词。我们可以创建一个列表，将三个单词的每个序列作为自变量，将每个序列后的下一个单词作为因变量。 我们可以用普通Python 编程语言编程语言来做到这一点。让我们首先使用令牌来确认它的外观：

Now we will do it with tensors of the numericalized values, which is what the model will actually use:

现在我们将使用数值的张量来完成，这是模型实际使用的：

We can batch those easily using the DataLoader class. For now we will split the sequences randomly:

我们可以使用DataLoader类轻松批处理它们。现在我们将随机拆分序列：

We can now create a neural network architecture that takes three words as input, and returns a prediction of the probability of each possible next word in the vocab. We will use three standard linear layers, but with two tweaks.

The first tweak is that the first linear layer will use only the first word's embedding as activations, the second layer will use the second word's embedding plus the first layer's output activations, and the third layer will use the third word's embedding plus the second layer's output activations. The key effect of this is that every word is interpreted in the information context of any words preceding it.

The second tweak is that each of these three layers will use the same weight matrix. The way that one word impacts the activations from previous words should not change depending on the position of a word. In other words, activation values will change as data moves through the layers, but the layer weights themselves will not change from layer to layer. So, a layer does not learn one sequence position; it must learn to handle all positions.

Since layer weights do not change, you might think of the sequential layers as "the same layer" repeated. In fact, PyTorch makes this concrete; we can just create one layer, and use it multiple times.

我们现在可以创建一个神经网络结构，将三个单词作为输入，并返回词汇中每个可能的下一个单词的概率预测。我们将使用三个标准线性的层，但有两个调整。 第一个调整是第一个线性的层将只使用第一个单词的嵌入作为激活，第二层将使用第二个单词的嵌入加上第一层的输出激活，第三层将使用第三个单词的嵌入加上第二层的输出激活。这样做的关键效果是每个单词都在它之前的任何单词的信息上下文中被解释。 第二个调整是这三层中的每一层都将使用相同的权矩阵。一个单词影响前一个单词激活的方式不应根据单词的位置而改变。换句话说，激活值会随着数据在层中移动而改变，但层权重本身不会因层而改变。因此，一个层不会学习一个序列位置；它必须学会处理所有位置。 由于层权重不会改变，您可能会认为顺序层是重复的“同一层”。事实上，PyTorch使这变得具体；我们可以只创建一层，并多次使用它。

Our Language Model in PyTorch

PyTorch中的语言模型

We can now create the language model module that we described earlier:

我们现在可以创建前面描述的语言模型模块：

As you see, we have created three layers:

• The embedding layer (i_h, for input to hidden)
• The linear layer to create the activations for the next word (h_h, for hidden to hidden)
• A final linear layer to predict the fourth word (h_o, for hidden to output)

This might be easier to represent in pictorial form, so let's define a simple pictorial representation of basic neural networks. <<img_simple_nn>> shows how we're going to represent a neural net with one hidden layer.

如您所见，我们创建了三层： 嵌入层（i_h，用于输入隐藏） 线性的层创建下一个单词的激活（h_h，从隐藏到隐藏） 最后一个线性的层来预测第四个单词（h_o，隐藏到输出） 这可能更容易以图形形式表示，因此让我们定义基本神经网络的简单图形表示。<>显示了我们将如何表示具有一个隐含层的神经网络。

Each shape represents activations: rectangle for input, circle for hidden (inner) layer activations, and triangle for output activations. We will use those shapes (summarized in <<img_shapes>>) in all the diagrams in this chapter.

每个形状代表激活：矩形用于输入，圆形用于隐藏（内部）层激活，三角形用于输出激活。我们将在本章的所有图表中使用这些形状（总结在<>中）。

An arrow represents the actual layer computation—i.e., the linear layer followed by the activation function. Using this notation, <<lm_rep>> shows what our simple language model looks like.

箭头表示实际的层计算——即线性的层，后跟激活函数。使用这种表示法，<>显示了我们的简单语言模型的样子。

To simplify things, we've removed the details of the layer computation from each arrow. We've also color-coded the arrows, such that all arrows with the same color have the same weight matrix. For instance, all the input layers use the same embedding matrix, so they all have the same color (green).

Let's try training this model and see how it goes:

为了简化，我们从每个箭头中删除了层计算的细节。我们还对箭头进行了颜色编码，使得所有具有相同颜色的箭头都具有相同的权矩阵。例如，所有输入层使用相同的嵌入矩阵，因此它们都具有相同的颜色（绿色）。 让我们试着训练这个模型，看看它是如何进行的：

To see if this is any good, let's check what a very simple model would give us. In this case we could always predict the most common token, so let's find out which token is most often the target in our validation set:

为了看看这是否有任何好处，让我们检查一个非常简单的模型会给我们什么。在这种情况下，我们总是可以预测最常见的令牌，所以让我们找出哪个令牌是我们验证集中最常见的目标：

The most common token has the index 29, which corresponds to the token thousand. Always predicting this token would give us an accuracy of roughly 15%, so we are faring way better!

最常见的令牌有索引29，对应于令牌千。总是预测这个令牌会给我们大约15%的准确率，所以我们做得更好！

A: My first guess was that the separator would be the most common token, since there is one for every number. But looking at tokens reminded me that large numbers are written with many words, so on the way to 10,000 you write "thousand" a lot: five thousand, five thousand and one, five thousand and two, etc. Oops! Looking at your data is great for noticing subtle features and also embarrassingly obvious ones.

A：我的第一个猜测是分隔符将是最常见的令牌，因为每个数字都有一个。但是看着令牌提醒我，大的数字是用很多单词写的，所以在到达10,000的路上，你会写很多“千”： 5000、51000、52000，等等。哎呀！查看你的数据非常有助于注意微妙的特征和令人尴尬的明显特征。

This is a nice first baseline. Let's see how we can refactor it with a loop.

这是一个很好的第一个基线。让我们看看如何用循环重构它。

Our First Recurrent Neural Network

我们的第一个循环神经网络

Looking at the code for our module, we could simplify it by replacing the duplicated code that calls the layers with a for loop. As well as making our code simpler, this will also have the benefit that we will be able to apply our module equally well to token sequences of different lengths—we won't be restricted to token lists of length three:

看看我们模块的代码，我们可以通过用for循环替换调用层的重复代码来简化它。除了使我们的代码更简单之外，这还有一个好处，那就是我们将能够同样好地将我们的模块应用于不同长度的令牌序列——我们不会局限于长度为3的令牌列表：

Let's check that we get the same results using this refactoring:

让我们检查使用此重构是否得到相同的结果：

We can also refactor our pictorial representation in exactly the same way, as shown in <<basic_rnn>> (we're also removing the details of activation sizes here, and using the same arrow colors as in <<lm_rep>>).

我们还可以以完全相同的方式重构图形表示，如<>中所示（我们还在此处删除了激活大小的详细信息，并使用与<>中相同的箭头颜色）。

You will see that there is a set of activations that are being updated each time through the loop, stored in the variable h—this is called the hidden state.

您将看到有一组激活每次都通过循环更新，存储在变量h中——这称为隐状态。

Jargon: hidden state: The activations that are updated at each step of a recurrent neural network.

行话：隐状态：在循环神经网络的每一步更新的激活。

A neural network that is defined using a loop like this is called a recurrent neural network (RNN). It is important to realize that an RNN is not a complicated new architecture, but simply a refactoring of a multilayer neural network using a for loop.

A: My true opinion: if they were called "looping neural networks," or LNNs, they would seem 50% less daunting!

使用这样的循环定义的神经网络称为循环神经网络（RNN）。重要的是要意识到RNN不是复杂的新架构，而只是使用for循环重构多层神经网络。 A：我的真实观点是：如果它们被称为“循环神经网络”或LNNs，它们看起来可怕的程度会减少50%！

Now that we know what an RNN is, let's try to make it a little bit better.

现在我们知道了什么是RNN，让我们试着把它做得更好一点。

Improving the RNN

改进RNN

Looking at the code for our RNN, one thing that seems problematic is that we are initializing our hidden state to zero for every new input sequence. Why is that a problem? We made our sample sequences short so they would fit easily into batches. But if we order the samples correctly, those sample sequences will be read in order by the model, exposing the model to long stretches of the original sequence.

Another thing we can look at is having more signal: why only predict the fourth word when we could use the intermediate predictions to also predict the second and third words?

Let's see how we can implement those changes, starting with adding some state.

看看我们的RNN代码，有一件事似乎有问题，那就是我们正在为每个新的输入序列初始化我们的隐状态为零。为什么会有问题？我们缩短了样本序列，这样它们就可以很容易地成批。但是如果我们正确排序样本，这些样本序列将被模型按顺序读取，从而使模型暴露在原始序列的很长一段中。 我们可以考虑的另一件事是有更多的信号：当我们可以使用中间预测来预测第二个和第三个单词时，为什么只预测第四个单词？ 让我们看看如何实现这些更改，从添加一些状态开始。

Maintaining the State of an RNN

维护RNN的状态

Because we initialize the model's hidden state to zero for each new sample, we are throwing away all the information we have about the sentences we have seen so far, which means that our model doesn't actually know where we are up to in the overall counting sequence. This is easily fixed; we can simply move the initialization of the hidden state to __init__.

But this fix will create its own subtle, but important, problem. It effectively makes our neural network as deep as the entire number of tokens in our document. For instance, if there were 10,000 tokens in our dataset, we would be creating a 10,000-layer neural network.

To see why this is the case, consider the original pictorial representation of our recurrent neural network in <<lm_rep>>, before refactoring it with a for loop. You can see each layer corresponds with one token input. When we talk about the representation of a recurrent neural network before refactoring with the for loop, we call this the unrolled representation. It is often helpful to consider the unrolled representation when trying to understand an RNN.

The problem with a 10,000-layer neural network is that if and when you get to the 10,000th word of the dataset, you will still need to calculate the derivatives all the way back to the first layer. This is going to be very slow indeed, and very memory-intensive. It is unlikely that you'll be able to store even one mini-batch on your GPU.

The solution to this problem is to tell PyTorch that we do not want to back propagate the derivatives through the entire implicit neural network. Instead, we will just keep the last three layers of gradients. To remove all of the gradient history in PyTorch, we use the detach method.

Here is the new version of our RNN. It is now stateful, because it remembers its activations between different calls to forward, which represent its use for different samples in the batch:

因为我们为每个新样本初始化模型的隐状态为零，所以我们丢弃了迄今为止所看到的关于句子的所有信息，这意味着我们的模型实际上不知道我们在整个计数序列中的位置。这很容易修复；我们可以简单地将隐状态的初始化移动到__init__。

但是这个修复会产生一个微妙但重要的问题。它有效地使我们的神经网络与文档中的所有标记一样深。例如，如果我们的数据集中有10,000个标记，我们将创建一个10,000层的神经网络。

要了解为什么会出现这种情况，请考虑<>中循环神经网络的原始图形表示，然后再使用for循环对其进行重构。您可以看到每一层对应一个令牌输入。当我们在使用for循环重构之前讨论循环神经网络的表示时，我们称之为展开表示。在尝试理解RNN时考虑展开表示通常很有帮助。

10,000层神经网络的问题是，如果并且当您到达数据集的第10,000个单词时，您仍然需要计算导数直到回到第一层。这确实会非常慢，并且非常占用内存。您不太可能在GPU上存储哪怕一个迷你批次。

这个问题的解决方案是告诉机器学习库，我们不想通过整个隐式神经网络反向传播导数。相反，我们将只保留最后三层梯度。为了删除机器学习库中的所有梯度历史，我们使用分离方法。 这是我们RNN的新版本。它现在是有状态的，因为它会记住它在不同的转发调用之间的激活，这代表它对批处理中不同样本的使用：

This model will have the same activations whatever sequence length we pick, because the hidden state will remember the last activation from the previous batch. The only thing that will be different is the gradients computed at each step: they will only be calculated on sequence length tokens in the past, instead of the whole stream. This approach is called backpropagation through time (BPTT).

无论我们选择什么序列长度，这个模型都将具有相同的激活，因为隐状态将记住前一批的最后一次激活。唯一不同的是在每一步计算的梯度：它们只会在过去的序列长度标记上计算，而不是整个流。这种方法称为通过时间的反向传播算法（BPTT）。

jargon: Back propagation through time (BPTT): Treating a neural net with effectively one layer per time step (usually refactored using a loop) as one big model, and calculating gradients on it in the usual way. To avoid running out of memory and time, we usually use truncated BPTT, which "detaches" the history of computation steps in the hidden state every few time steps.

行话：通过时间反向传播（BPTT）：将每个时间步有效地一层（通常使用循环重构）的神经网络视为一个大模型，并以通常的方式计算其上的梯度。为了避免运行内存溢出和时间，我们通常使用截断的BPTT，它每隔几个时间步“分离”隐状态中计算步骤的历史。

To use LMModel3, we need to make sure the samples are going to be seen in a certain order. As we saw in <<chapter_nlp>>, if the first line of the first batch is our dset[0] then the second batch should have dset[1] as the first line, so that the model sees the text flowing.

LMDataLoader was doing this for us in <<chapter_nlp>>. This time we're going to do it ourselves.

To do this, we are going to rearrange our dataset. First we divide the samples into m = len(dset) // bs groups (this is the equivalent of splitting the whole concatenated dataset into, for example, 64 equally sized pieces, since we're using bs=64 here). m is the length of each of these pieces. For instance, if we're using our whole dataset (although we'll actually split it into train versus valid in a moment), that will be:

要使用LMModel3，我们需要确保以一定的顺序看到样本。正如我们在<>中看到的，如果第一批的第一行是我们的dset[0]，那么第二批应该有dset[1]作为第一行，以便模型看到文本流动。 LMDataLoader在<>中为我们做了这件事。这次我们要自己做。 为此，我们将重新排列我们的数据集。首先，我们将样本划分为m=len（dset）//bs组（这相当于将整个串联数据集拆分为64个相同大小的块，因为我们在这里使用bs=64）。m是这些片段中每个片段的长度。例如，如果我们使用我们的整个数据集（尽管我们实际上会将其拆分为训练与有效），那将是：

The first batch will be composed of the samples:

(0, m, 2*m, ..., (bs-1)*m)

the second batch of the samples:

(1, m+1, 2*m+1, ..., (bs-1)*m+1)

and so forth. This way, at each epoch, the model will see a chunk of contiguous text of size 3*m (since each text is of size 3) on each line of the batch.

The following function does that reindexing:

第一个批次由以下的样本组成： (0, m, 2*m, ..., (bs-1)m) 第二个批次由一下的样本组成： (1, m+1, 2m+1, ..., (bs-1)m+1) 第四层也是如此。用这种方法，在每一个批次，模型将在批处理的每一行上看到一块大小为3m的连续文本（因为每个文本的大小都是3）。 以下函数执行此重新索引操作：

Then we just pass drop_last=True when building our DataLoaders to drop the last batch that does not have a shape of bs. We also pass shuffle=False to make sure the texts are read in order:

然后，在构建DataLoader时，我们只需传递drop_last=True以删除最后一批没有bs形状的文本。我们还传递shuffle=False以确保按顺序读取文本：

The last thing we add is a little tweak of the training loop via a Callback. We will talk more about callbacks in <<chapter_accel_sgd>>; this one will call the reset method of our model at the beginning of each epoch and before each validation phase. Since we implemented that method to zero the hidden state of the model, this will make sure we start with a clean state before reading those continuous chunks of text. We can also start training a bit longer:

我们添加的最后一件事是通过回调对训练循环进行一点调整。我们将在<>中更多地讨论回调；这个将在每个epoch的开始和每个验证阶段之前调用模型的重置方法。由于我们实现了该方法以将模型的隐状态归零，这将确保我们在阅读那些连续的文本块之前从干净的状态开始。我们还可以开始更长时间的训练：

This is already better! The next step is to use more targets and compare them to the intermediate predictions.

这已经更好了！下一步是使用更多的目标，并将它们与中间预测进行比较。

Creating More Signal

创造更多信号

Another problem with our current approach is that we only predict one output word for each three input words. That means that the amount of signal that we are feeding back to update weights with is not as large as it could be. It would be better if we predicted the next word after every single word, rather than every three words, as shown in <<stateful_rep>>.

我们当前方法的另一个问题是，我们每三个输入词只预测一个输出词。这意味着，我们反馈给更新权重的信号量没有可能的大。如果我们在每个单词之后预测下一个单词，而不是每三个单词预测一次，会更好，如<>所示。

This is easy enough to add. We need to first change our data so that the dependent variable has each of the three next words after each of our three input words. Instead of 3, we use an attribute, sl (for sequence length), and make it a bit bigger:

这很容易添加。我们需要首先更改我们的数据，以便因变量在我们的三个输入词之后都有三个接下来的词。我们使用属性sl（表示序列长度）而不是3，并使其大一点：

Looking at the first element of seqs, we can see that it contains two lists of the same size. The second list is the same as the first, but offset by one element:

查看seqs的第一个元素，我们可以看到它包含两个大小相同的列表。第二个列表与第一个相同，但偏移量为一个元素：

Now we need to modify our model so that it outputs a prediction after every word, rather than just at the end of a three-word sequence:

现在我们需要修改我们的模型，以便它在每个单词之后输出预测，而不仅仅是在三个单词序列的末尾：

This model will return outputs of shape bs x sl x vocab_sz (since we stacked on dim=1). Our targets are of shape bs x sl, so we need to flatten those before using them in F.cross_entropy:

此模型将返回形状bs x sl xvocab_sz的输出（因为我们堆叠在dim=1上）。我们的目标是形状bs x sl，因此我们需要在将它们用于F.cross_entropy之前将它们展平：

We can now use this loss function to train the model:

现在我们可以使用损失函数来训练模型

We need to train for longer, since the task has changed a bit and is more complicated now. But we end up with a good result... At least, sometimes. If you run it a few times, you'll see that you can get quite different results on different runs. That's because effectively we have a very deep network here, which can result in very large or very small gradients. We'll see in the next part of this chapter how to deal with this.

Now, the obvious way to get a better model is to go deeper: we only have one linear layer between the hidden state and the output activations in our basic RNN, so maybe we'll get better results with more.

我们需要训练更长时间，因为任务已经发生了一些变化，现在变得更加复杂了。但是我们最终会得到一个好结果...至少有时是这样。如果你运行几次，你会发现你可以在不同的运行中得到完全不同的结果。这是因为实际上我们这里有一个非常深的网络，这可能会导致非常大或非常小的梯度。我们将在本章的下一部分看到如何处理这个问题。

现在，获得更好模型的明显方法是深入研究：我们在基本RNN中的隐状态和输出激活之间只有一个线性层，所以也许我们会得到更好的结果。

Multilayer RNNs

多层RNN

In a multilayer RNN, we pass the activations from our recurrent neural network into a second recurrent neural network, like in <<stacked_rnn_rep>>.

在多层RNN中，我们将循环神经网络的激活传递到第二个循环神经网络，如<>中。

The unrolled representation is shown in <<unrolled_stack_rep>> (similar to <<lm_rep>>).

展开的表示显示在<>中（类似于<>）。

Let's see how to implement this in practice.

现在我们来看看怎样在实际中实现。

The Model

模型

We can save some time by using PyTorch's RNN class, which implements exactly what we created earlier, but also gives us the option to stack multiple RNNs, as we have discussed:

我们可以通过使用 PyTorch的RNN类来节省一些时间，它完全实现了我们之前创建的内容，但也为我们提供了堆叠多个RNN的选项，正如我们已经讨论过的：

Now that's disappointing... our previous single-layer RNN performed better. Why? The reason is that we have a deeper model, leading to exploding or vanishing activations.

现在这令人失望...我们之前的单层RNN表现更好。为什么？原因是我们有一个更深的模型，导致爆炸或消失激活。

Exploding or Disappearing Activations

爆炸或消失激活

In practice, creating accurate models from this kind of RNN is difficult. We will get better results if we call detach less often, and have more layers—this gives our RNN a longer time horizon to learn from, and richer features to create. But it also means we have a deeper model to train. The key challenge in the development of deep learning has been figuring out how to train these kinds of models.

The reason this is challenging is because of what happens when you multiply by a matrix many times. Think about what happens when you multiply by a number many times. For example, if you multiply by 2, starting at 1, you get the sequence 1, 2, 4, 8,... after 32 steps you are already at 4,294,967,296. A similar issue happens if you multiply by 0.5: you get 0.5, 0.25, 0.125… and after 32 steps it's 0.00000000023. As you can see, multiplying by a number even slightly higher or lower than 1 results in an explosion or disappearance of our starting number, after just a few repeated multiplications.

Because matrix multiplication is just multiplying numbers and adding them up, exactly the same thing happens with repeated matrix multiplications. And that's all a deep neural network is —each extra layer is another matrix multiplication. This means that it is very easy for a deep neural network to end up with extremely large or extremely small numbers.

This is a problem, because the way computers store numbers (known as "floating point") means that they become less and less accurate the further away the numbers get from zero. The diagram in <<float_prec>>, from the excellent article "What You Never Wanted to Know About Floating Point but Will Be Forced to Find Out", shows how the precision of floating-point numbers varies over the number line.

在实践中，从这种RNN创建准确的模型是很困难的。如果我们更少地调用分离，并拥有更多的层，我们将获得更好的结果——这给了我们的RNN更长的时间范围来学习，以及更丰富的特征来创建。但这也意味着我们有更深层次的模型要训练。深度学习发展的关键挑战一直是弄清楚如何训练这类模型。

这之所以具有挑战性，是因为当你多次乘以一个矩阵时会发生什么。想想当你多次乘以一个数字时会发生什么。例如，如果你乘以2，从1开始，你得到的序列是1,2,4,8，...32步后你已经是4,294,967,296了。如果你乘以0.5也会发生类似的问题：你得到0.5,0.25,0.125...32步后是0.00000000023。正如你所看到的，乘以一个比1稍高或稍低的数字会导致我们的起始数爆炸或消失，只需重复几次乘法。

因为矩阵乘法只是将数字相乘并相加，重复的矩阵乘法也会发生完全相同的事情。这就是深度神经网络的全部——每多一层都是另一个矩阵乘法。这意味着深度神经网络很容易得到非常大或非常小的数字。

这是一个问题，因为计算机存储数字的方式（称为“浮点”）意味着数字离零越远，它们就变得越来越不准确。<>中的图表来自优秀文章“你从未想过要知道的关于浮点但将被迫找出”，显示了浮点数的查准率/精确度如何在数线上变化。

This inaccuracy means that often the gradients calculated for updating the weights end up as zero or infinity for deep networks. This is commonly referred to as the vanishing gradients or exploding gradients problem. It means that in SGD, the weights are either not updated at all or jump to infinity. Either way, they won't improve with training.

Researchers have developed a number of ways to tackle this problem, which we will be discussing later in the book. One option is to change the definition of a layer in a way that makes it less likely to have exploding activations. We'll look at the details of how this is done in <<chapter_convolutions>>, when we discuss batch normalization, and <<chapter_resnet>>, when we discuss ResNets, although these details don't generally matter in practice (unless you are a researcher that is creating new approaches to solving this problem). Another strategy for dealing with this is by being careful about initialization, which is a topic we'll investigate in <<chapter_foundations>>.

For RNNs, there are two types of layers that are frequently used to avoid exploding activations: gated recurrent units (GRUs) and long short-term memory (LSTM) layers. Both of these are available in PyTorch, and are drop-in replacements for the RNN layer. We will only cover LSTMs in this book; there are plenty of good tutorials online explaining GRUs, which are a minor variant on the LSTM design.

这种不准确性意味着，对于深度网络，为更新权重而计算的梯度通常最终为零或无穷大。这通常被称为梯度消失或梯度爆炸问题。这意味着在SGD中，权重要么根本不更新，要么跳到无穷大。无论哪种方式，它们都不会通过训练得到改善。

研究人员已经开发了许多方法来解决这个问题，我们将在本书的后面讨论。一种选择是改变层的定义，使其不太可能发生爆炸式激活。当我们讨论批处理规范化时，我们将在<>和<>中查看如何做到这一点的细节，尽管这些细节在实践中通常并不重要（除非您是正在创建新方法来解决这个问题的研究人员）。另一种解决这个问题的策略是小心初始化，这是我们将在<>中研究的主题。

对于RNN，有两种类型的层经常用于避免爆炸激活：门控循环单元（GRU）和长时记忆（LSTM）层。这两种都可以在PyTorch中使用，并且是RNN层的直接替代品。我们将在本书中只介绍LSTM；网上有很多很好的教程解释GRU，这是LSTM设计的一个小变体。

LSTM

LSTM is an architecture that was introduced back in 1997 by Jürgen Schmidhuber and Sepp Hochreiter. In this architecture, there are not one but two hidden states. In our base RNN, the hidden state is the output of the RNN at the previous time step. That hidden state is then responsible for two things:

• Having the right information for the output layer to predict the correct next token
• Retaining memory of everything that happened in the sentence

Consider, for example, the sentences "Henry has a dog and he likes his dog very much" and "Sophie has a dog and she likes her dog very much." It's very clear that the RNN needs to remember the name at the beginning of the sentence to be able to predict he/she or his/her.

In practice, RNNs are really bad at retaining memory of what happened much earlier in the sentence, which is the motivation to have another hidden state (called cell state) in the LSTM. The cell state will be responsible for keeping long short-term memory, while the hidden state will focus on the next token to predict. Let's take a closer look at how this is achieved and build an LSTM from scratch.

LSTM是一个由Jürgen Schmidhuber和Sepp Hochreiter于1997年引入的架构。在这个架构中，不是一个而是两个隐藏状态。在我们的基础RNN中，隐状态是RNN在前一个时间步的输出。然后，隐状态负责两件事：

为输出层提供正确的信息来预测正确的下一个令牌 保留对句子中发生的一切的记忆

例如，考虑句子“亨利有一只狗，他非常喜欢他的狗”和“索菲有一只狗，她非常喜欢她的狗”很明显，RNN需要记住句子开头的名字才能预测他/她或他/她。

在实践中，RNN真的不擅长保留对句子中更早发生的事情的记忆，这是LSTM中另一个隐状态（称为单元格状态）的动机。单元格状态将负责保持长时记忆，而隐状态将专注于下一个要预测的令牌。让我们仔细看看这是如何实现的，并从头开始构建LSTM。

Building an LSTM from Scratch

从头开始构建LSTM

In order to build an LSTM, we first have to understand its architecture. <> shows its inner structure.

为了构建LSTM，我们首先必须了解它的架构。<>显示了它的内部结构。

In this picture, our input $x_{t}$ enters on the left with the previous hidden state ($h_{t-1}$) and cell state ($c_{t-1}$). The four orange boxes represent four layers (our neural nets) with the activation being either sigmoid ($\sigma$) or tanh. tanh is just a sigmoid function rescaled to the range -1 to 1. Its mathematical expression can be written like this:

$\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x}+e^{-x}} = 2 \sigma(2x) - 1$

where $\sigma$ is the sigmoid function. The green circles are elementwise operations. What goes out on the right is the new hidden state ($h_{t}$) and new cell state ($c_{t}$), ready for our next input. The new hidden state is also used as output, which is why the arrow splits to go up.

Let's go over the four neural nets (called gates) one by one and explain the diagram—but before this, notice how very little the cell state (at the top) is changed. It doesn't even go directly through a neural net! This is exactly why it will carry on a longer-term state.

First, the arrows for input and old hidden state are joined together. In the RNN we wrote earlier in this chapter, we were adding them together. In the LSTM, we stack them in one big tensor. This means the dimension of our embeddings (which is the dimension of $x_{t}$) can be different than the dimension of our hidden state. If we call those n_in and n_hid, the arrow at the bottom is of size n_in + n_hid; thus all the neural nets (orange boxes) are linear layers with n_in + n_hid inputs and n_hid outputs.

The first gate (looking from left to right) is called the forget gate. Since it’s a linear layer followed by a sigmoid, its output will consist of scalars between 0 and 1. We multiply this result by the cell state to determine which information to keep and which to throw away: values closer to 0 are discarded and values closer to 1 are kept. This gives the LSTM the ability to forget things about its long-term state. For instance, when crossing a period or an xxbos token, we would expect to it to (have learned to) reset its cell state.

The second gate is called the input gate. It works with the third gate (which doesn't really have a name but is sometimes called the cell gate) to update the cell state. For instance, we may see a new gender pronoun, in which case we'll need to replace the information about gender that the forget gate removed. Similar to the forget gate, the input gate decides which elements of the cell state to update (values close to 1) or not (values close to 0). The third gate determines what those updated values are, in the range of –1 to 1 (thanks to the tanh function). The result is then added to the cell state.

The last gate is the output gate. It determines which information from the cell state to use to generate the output. The cell state goes through a tanh before being combined with the sigmoid output from the output gate, and the result is the new hidden state.

In terms of code, we can write the same steps like this:

在这张图中，我们的输入在左侧输入之前的隐藏状态( ℎ𝑡−1 )和单元格状态( 𝑐𝑡−1 )。四个橙色框代表四层（我们的神经网络），激活要么是sigmoid ( 𝜎 ) 要么是tanh。tanh只是一个重新缩放到-1到1范围内的sigmoid函数。它的数学表达式可以这样写：

$\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x}+e^{-x}} = 2 \sigma(2x) - 1$

其中𝜎是sigmoid函数。绿色圆圈是元素操作。右侧输出的是新的隐藏状态 ( ℎ𝑡 ) 和新的单元格状态 ( 𝑐𝑡 ), 为我们的下一个输入做好准备。新的隐藏状态也用作输出，这就是箭头向上分裂的原因。 让我们一个接一个地检查四个神经网络（称为门）并解释图表——但在此之前，请注意细胞状态（在顶部）变化很小。它甚至没有直接通过神经网络！这正是为什么它会持续更长时间的状态。 首先，输入和旧隐藏状态的箭头连接在一起。在本章前面写的RNN中，我们将它们相加在一起。在LSTM中，我们将它们堆叠在一个大张量中。这意味着我们嵌入的维度（即的维度）可能与我们隐藏状态的维度不同。如果我们调用这些n_in和n_hid，底部的箭头大小为n_in+n_hid；因此所有神经网络（橙色框）都是线性层，具有n_in+n_hid输入和n_hid输出。 第一个门（从左到右）被称为遗忘门。由于它是一个线性层，后跟一个sigmoid，它的输出将由0到1之间的标量组成。我们将此结果乘以单元格状态以确定哪些信息要保留，哪些信息要丢弃：更接近0的值被丢弃，更接近1的值被保留。这使LSTM能够忘记有关其长期状态的事情。例如，当跨越一个句点或xxbos标记时，我们希望它（已经学会）重置其单元格状态。

第二个门称为输入门。它与第三个门（实际上没有名字，但有时称为单元格门）一起更新单元格状态。例如，我们可能会看到一个新的性别代词，在这种情况下，我们需要替换忘记门删除的有关性别的信息。与忘记门类似，输入门决定要更新单元格状态的哪些元素（值接近1）或不更新（值接近0）。第三个门确定这些更新的值是什么，范围在-1到1之间（感谢tanh函数）。然后将结果添加到单元格状态。 最后一个门是输出门。它决定使用来自单元状态的哪些信息来生成输出。单元状态在与输出门的sigmoid输出组合之前经过tanh，结果是新的隐藏状态。

在代码方面，我们可以编写如下相同的步骤：

In practice, we can then refactor the code. Also, in terms of performance, it's better to do one big matrix multiplication than four smaller ones (that's because we only launch the special fast kernel on the GPU once, and it gives the GPU more work to do in parallel). The stacking takes a bit of time (since we have to move one of the tensors around on the GPU to have it all in a contiguous array), so we use two separate layers for the input and the hidden state. The optimized and refactored code then looks like this:

在实践中，我们可以重构代码。此外，就性能而言，做一个大矩阵乘法比做四个小矩阵乘法更好（这是因为我们只在GPU上启动一次特殊的快速内核，它让GPU有更多的工作要并行完成）。堆叠需要一点时间（因为我们必须在GPU上移动一个张量以将其全部放在一个连续的数组中），所以我们使用两个单独的层来输入和隐状态。优化和重构的代码如下所示： 在实践中，我们可以重构代码。此外，就性能而言，做一个大矩阵乘法比做四个小矩阵乘法更好（这是因为我们只在GPU上启动一次特殊的快速内核，它让GPU有更多的工作要并行完成）。堆叠需要一点时间（因为我们必须在GPU上移动一个张量以将其全部放在一个连续的数组中），所以我们使用两个单独的层来输入和隐藏状态。优化和重构的代码如下所示： ...

Here we use the PyTorch chunk method to split our tensor into four pieces. It works like this:

这里我们使用PyTorch的chunk方法将我们的张量分成四块。它是这样工作的：

Let's now use this architecture to train a language model!

现在让我们使用这种架构来训练语言模型！

Training a Language Model Using LSTMs

使用LSTMs训练一个语言模型

Here is the same network as LMModel5, using a two-layer LSTM. We can train it at a higher learning rate, for a shorter time, and get better accuracy:

这是与LMModel5相同的网络，使用两层LSTM。我们可以以更高的学习率、更短的时间训练它，并获得更好的准确率：

Now that's better than a multilayer RNN! We can still see there is a bit of overfitting, however, which is a sign that a bit of regularization might help.

这比多层RNN更好！然而，我们仍然可以看到有一点过拟合，这是一点正则化可能有所帮助的迹象。

Regularizing an LSTM

正则化一个LSTM

Recurrent neural networks, in general, are hard to train, because of the problem of vanishing activations and gradients we saw before. Using LSTM (or GRU) cells makes training easier than with vanilla RNNs, but they are still very prone to overfitting. Data augmentation, while a possibility, is less often used for text data than for images because in most cases it requires another model to generate random augmentations (e.g., by translating the text into another language and then back into the original language). Overall, data augmentation for text data is currently not a well-explored space.

However, there are other regularization techniques we can use instead to reduce overfitting, which were thoroughly studied for use with LSTMs in the paper "Regularizing and Optimizing LSTM Language Models" by Stephen Merity, Nitish Shirish Keskar, and Richard Socher. This paper showed how effective use of dropout, activation regularization, and temporal activation regularization could allow an LSTM to beat state-of-the-art results that previously required much more complicated models. The authors called an LSTM using these techniques an AWD-LSTM. We'll look at each of these techniques in turn.

一般来说，循环神经网络很难训练，因为我们之前看到的激活和梯度消失的问题。使用LSTM（或GRU）细胞比使用普通RNN更容易训练，但它们仍然非常容易过拟合。数据增强虽然是一种可能性，但经常用于图像而不是文本，因为在大多数情况下，它需要另一个模型来生成随机增强（例如，通过将文本翻译成另一种语言，然后返回到原始语言）。总的来说，文本数据的数据目前不是一个很好探索的空间。

Dropout

Dropout is a regularization technique that was introduced by Geoffrey Hinton et al. in Improving neural networks by preventing co-adaptation of feature detectors. The basic idea is to randomly change some activations to zero at training time. This makes sure all neurons actively work toward the output, as seen in <<img_dropout>> (from "Dropout: A Simple Way to Prevent Neural Networks from Overfitting" by Nitish Srivastava et al.).

Hinton used a nice metaphor when he explained, in an interview, the inspiration for dropout:

: I went to my bank. The tellers kept changing and I asked one of them why. He said he didn’t know but they got moved around a lot. I figured it must be because it would require cooperation between employees to successfully defraud the bank. This made me realize that randomly removing a different subset of neurons on each example would prevent conspiracies and thus reduce overfitting.

In the same interview, he also explained that neuroscience provided additional inspiration:

: We don't really know why neurons spike. One theory is that they want to be noisy so as to regularize, because we have many more parameters than we have data points. The idea of dropout is that if you have noisy activations, you can afford to use a much bigger model.

This explains the idea behind why dropout helps to generalize: first it helps the neurons to cooperate better together, then it makes the activations more noisy, thus making the model more robust.

We can see, however, that if we were to just zero those activations without doing anything else, our model would have problems training: if we go from the sum of five activations (that are all positive numbers since we apply a ReLU) to just two, this won't have the same scale. Therefore, if we apply dropout with a probability p, we rescale all activations by dividing them by 1-p (on average p will be zeroed, so it leaves 1-p), as shown in <<img_dropout1>>.

This is a full implementation of the dropout layer in PyTorch (although PyTorch's native layer is actually written in C, not Python):

The bernoulli_ method is creating a tensor of random zeros (with probability p) and ones (with probability 1-p), which is then multiplied with our input before dividing by 1-p. Note the use of the training attribute, which is available in any PyTorch nn.Module, and tells us if we are doing training or inference.

note: Do Your Own Experiments: In previous chapters of the book we'd be adding a code example for bernoulli_ here, so you can see exactly how it works. But now that you know enough to do this yourself, we're going to be doing fewer and fewer examples for you, and instead expecting you to do your own experiments to see how things work. In this case, you'll see in the end-of-chapter questionnaire that we're asking you to experiment with bernoulli_—but don't wait for us to ask you to experiment to develop your understanding of the code we're studying; go ahead and do it anyway!

Using dropout before passing the output of our LSTM to the final layer will help reduce overfitting. Dropout is also used in many other models, including the default CNN head used in fastai.vision, and is available in fastai.tabular by passing the ps parameter (where each "p" is passed to each added Dropout layer), as we'll see in <<chapter_arch_details>>.

Dropout has different behavior in training and validation mode, which we specified using the training attribute in Dropout. Calling the train method on a Module sets training to True (both for the module you call the method on and for every module it recursively contains), and eval sets it to False. This is done automatically when calling the methods of Learner, but if you are not using that class, remember to switch from one to the other as needed.

Activation Regularization and Temporal Activation Regularization

Activation regularization (AR) and temporal activation regularization (TAR) are two regularization methods very similar to weight decay, discussed in <<chapter_collab>>. When applying weight decay, we add a small penalty to the loss that aims at making the weights as small as possible. For activation regularization, it's the final activations produced by the LSTM that we will try to make as small as possible, instead of the weights.

To regularize the final activations, we have to store those somewhere, then add the means of the squares of them to the loss (along with a multiplier alpha, which is just like wd for weight decay):

loss += alpha * activations.pow(2).mean()

Temporal activation regularization is linked to the fact we are predicting tokens in a sentence. That means it's likely that the outputs of our LSTMs should somewhat make sense when we read them in order. TAR is there to encourage that behavior by adding a penalty to the loss to make the difference between two consecutive activations as small as possible: our activations tensor has a shape bs x sl x n_hid, and we read consecutive activations on the sequence length axis (the dimension in the middle). With this, TAR can be expressed as:

loss += beta * (activations[:,1:] - activations[:,:-1]).pow(2).mean()

alpha and beta are then two hyperparameters to tune. To make this work, we need our model with dropout to return three things: the proper output, the activations of the LSTM pre-dropout, and the activations of the LSTM post-dropout. AR is often applied on the dropped-out activations (to not penalize the activations we turned into zeros afterward) while TAR is applied on the non-dropped-out activations (because those zeros create big differences between two consecutive time steps). There is then a callback called RNNRegularizer that will apply this regularization for us.

Training a Weight-Tied Regularized LSTM

We can combine dropout (applied before we go into our output layer) with AR and TAR to train our previous LSTM. We just need to return three things instead of one: the normal output of our LSTM, the dropped-out activations, and the activations from our LSTMs. The last two will be picked up by the callback RNNRegularization for the contributions it has to make to the loss.

Another useful trick we can add from the AWD LSTM paper is weight tying. In a language model, the input embeddings represent a mapping from English words to activations, and the output hidden layer represents a mapping from activations to English words. We might expect, intuitively, that these mappings could be the same. We can represent this in PyTorch by assigning the same weight matrix to each of these layers:

self.h_o.weight = self.i_h.weight

In LMModel7, we include these final tweaks:

We can create a regularized Learner using the RNNRegularizer callback:

A TextLearner automatically adds those two callbacks for us (with those values for alpha and beta as defaults), so we can simplify the preceding line to:

We can then train the model, and add additional regularization by increasing the weight decay to 0.1:

Now this is far better than our previous model!

Conclusion

You have now seen everything that is inside the AWD-LSTM architecture we used in text classification in <<chapter_nlp>>. It uses dropout in a lot more places:

• Embedding dropout (inside the embedding layer, drops some random lines of embeddings)
• Input dropout (applied after the embedding layer)
• Weight dropout (applied to the weights of the LSTM at each training step)
• Hidden dropout (applied to the hidden state between two layers)

This makes it even more regularized. Since fine-tuning those five dropout values (including the dropout before the output layer) is complicated, we have determined good defaults and allow the magnitude of dropout to be tuned overall with the drop_mult parameter you saw in that chapter (which is multiplied by each dropout).

Another architecture that is very powerful, especially in "sequence-to-sequence" problems (that is, problems where the dependent variable is itself a variable-length sequence, such as language translation), is the Transformers architecture. You can find it in a bonus chapter on the book's website.

Questionnaire

1. If the dataset for your project is so big and complicated that working with it takes a significant amount of time, what should you do?
2. Why do we concatenate the documents in our dataset before creating a language model?
3. To use a standard fully connected network to predict the fourth word given the previous three words, what two tweaks do we need to make to our model?
4. How can we share a weight matrix across multiple layers in PyTorch?
5. Write a module that predicts the third word given the previous two words of a sentence, without peeking.
6. What is a recurrent neural network?
7. What is "hidden state"?
8. What is the equivalent of hidden state in LMModel1?
9. To maintain the state in an RNN, why is it important to pass the text to the model in order?
10. What is an "unrolled" representation of an RNN?
11. Why can maintaining the hidden state in an RNN lead to memory and performance problems? How do we fix this problem?
12. What is "BPTT"?
13. Write code to print out the first few batches of the validation set, including converting the token IDs back into English strings, as we showed for batches of IMDb data in <<chapter_nlp>>.
14. What does the ModelResetter callback do? Why do we need it?
15. What are the downsides of predicting just one output word for each three input words?
16. Why do we need a custom loss function for LMModel4?
17. Why is the training of LMModel4 unstable?
18. In the unrolled representation, we can see that a recurrent neural network actually has many layers. So why do we need to stack RNNs to get better results?
19. Draw a representation of a stacked (multilayer) RNN.
20. Why should we get better results in an RNN if we call detach less often? Why might this not happen in practice with a simple RNN?
21. Why can a deep network result in very large or very small activations? Why does this matter?
22. In a computer's floating-point representation of numbers, which numbers are the most precise?
23. Why do vanishing gradients prevent training?
24. Why does it help to have two hidden states in the LSTM architecture? What is the purpose of each one?
25. What are these two states called in an LSTM?
26. What is tanh, and how is it related to sigmoid?
27. What is the purpose of this code in LSTMCell: h = torch.cat([h, input], dim=1)
28. What does chunk do in PyTorch?
29. Study the refactored version of LSTMCell carefully to ensure you understand how and why it does the same thing as the non-refactored version.
30. Why can we use a higher learning rate for LMModel6?
31. What are the three regularization techniques used in an AWD-LSTM model?
32. What is "dropout"?
33. Why do we scale the acitvations with dropout? Is this applied during training, inference, or both?
34. What is the purpose of this line from Dropout: if not self.training: return x
35. Experiment with bernoulli_ to understand how it works.
36. How do you set your model in training mode in PyTorch? In evaluation mode?
37. Write the equation for activation regularization (in math or code, as you prefer). How is it different from weight decay?
38. Write the equation for temporal activation regularization (in math or code, as you prefer). Why wouldn't we use this for computer vision problems?
39. What is "weight tying" in a language model?

Further Research

1. In LMModel2, why can forward start with h=0? Why don't we need to say h=torch.zeros(...)?
2. Write the code for an LSTM from scratch (you may refer to <>).
3. Search the internet for the GRU architecture and implement it from scratch, and try training a model. See if you can get results similar to those we saw in this chapter. Compare your results to the results of PyTorch's built in GRU module.
4. Take a look at the source code for AWD-LSTM in fastai, and try to map each of the lines of code to the concepts shown in this chapter.