CNN Interpretation with CAM

Now that we know how to build up pretty much anything from scratch, let's use that knowledge to create entirely new (and very useful!) functionality: the class activation map. It gives us some insight into why a CNN made the predictions it did.

In the process, we'll learn about one handy feature of PyTorch we haven't seen before, the hook, and we'll apply many of the concepts introduced in the rest of the book. If you want to really test out your understanding of the material in this book, after you've finished this chapter, try putting it aside and recreating the ideas here yourself from scratch (no peeking!).

现在我们知道了如何从头开始构建几乎任何东西，让我们利用这些知识来创建全新的（并且非常有用的！）功能：类激活图。它让我们了解了CNN为什么会做出这样的预测。 在这个过程中，我们将了解机器学习库中我们以前从未见过的一个方便的特性，即hook，我们将应用本书其余部分中介绍的许多概念。如果你想真正测试你对本书材料的理解，在你读完这一章后，试着把它放在一边，自己从头开始重新创建这里的想法（不要偷看！）。

CAM and Hooks

The class activation map (CAM) was introduced by Bolei Zhou et al. in "Learning Deep Features for Discriminative Localization". It uses the output of the last convolutional layer (just before the average pooling layer) together with the predictions to give us a heatmap visualization of why the model made its decision. This is a useful tool for interpretation.

More precisely, at each position of our final convolutional layer, we have as many filters as in the last linear layer. We can therefore compute the dot product of those activations with the final weights to get, for each location on our feature map, the score of the feature that was used to make a decision.

We're going to need a way to get access to the activations inside the model while it's training. In PyTorch this can be done with a hook. Hooks are PyTorch's equivalent of fastai's callbacks. However, rather than allowing you to inject code into the training loop like a fastai Learner callback, hooks allow you to inject code into the forward and backward calculations themselves. We can attach a hook to any layer of the model, and it will be executed when we compute the outputs (forward hook) or during backpropagation (backward hook). A forward hook is a function that takes three things—a module, its input, and its output—and it can perform any behavior you want. (fastai also provides a handy HookCallback that we won't cover here, but take a look at the fastai docs; it makes working with hooks a little easier.)

To illustrate, we'll use the same cats and dogs model we trained in <<chapter_intro>>:

类激活图（CAM）由Bolei Zhou等人在“学习深度特征以进行判别定位”中介绍。它使用最后一个卷积神经网络的输出（就在平均汇聚层之前）与预测一起为我们提供模型为什么做出决定的热图可视化。这是一个有用的解释工具。 更准确地说，在我们最终卷积神经网络的每个位置，我们有和最后一个线性的层一样多的过滤器。因此，我们可以计算这些激活的点积和最终的权重，以便为特征图上的每个位置获得用于做出决定的特征的分数。 我们需要一种方法来在模型训练时访问模型内部的激活。在机器学习库中，这可以用hook来完成。hook相当于机器学习库的快速回调。但是，hook不允许您像快速“学习者”回调一样将代码注入训练循环，而是允许您将代码注入向前和向后计算本身。我们可以将hook附加到模型的任何层，它将在我们计算输出时执行（forward hook）或在反向传播算法期间执行（backward hook）。forward hook是一个包含三件事的函数——模块、输入和输出——它可以执行您想要的任何行为。（Fastai还提供了一个方便的“HookCallback”，我们不会在这里介绍，但请查看Fastai文档；它使使用hook变得更容易了。） 为了说明，我们将使用我们在<<chapter_intro>>中训练的相同猫和狗模型：

To start, we'll grab a cat picture and a batch of data:

首先，我们将获取一张猫的照片和一批数据：

For CAM we want to store the activations of the last convolutional layer. We put our hook function in a class so it has a state that we can access later, and just store a copy of the output:

对于CAM，我们希望存储最后一个卷积神经网络的激活。我们将hook函数放在一个类中，这样它就有一个我们可以稍后访问的状态，并且只存储输出的副本：

We can then instantiate a Hook and attach it to the layer we want, which is the last layer of the CNN body:

然后我们可以实例化一个Hook并将其附加到我们想要的层，即CNN主体的最后一层：

Now we can grab a batch and feed it through our model:

现在我们可以抓取一批并通过我们的模型提供：

And we can access our stored activations:

我们可以访问我们存储的激活：

Let's also double-check our predictions:

让我们也仔细检查一下我们的预测：

We know 0 (for False) is "dog," because the classes are automatically sorted in fastai, bu we can still double-check by looking at dls.vocab:

我们知道‘0’（为False）是“狗”，因为类是自动排序的，但是我们仍然可以通过查看dls.vocab来仔细检查：

So, our model is very confident this was a picture of a cat.

所以，我们的模型非常确信这是一张猫的照片。

To do the dot product of our weight matrix (2 by number of activations) with the activations (batch size by activations by rows by cols), we use a custom einsum:

要使用激活（批次大小，激活行，激活行）对权矩阵（2）进行点积，我们使用自定义einsum：

For each image in our batch, and for each class, we get a 7×7 feature map that tells us where the activations were higher and where they were lower. This will let us see which areas of the pictures influenced the model's decision.

For instance, we can find out which areas made the model decide this animal was a cat (note that we need to decode the input x since it's been normalized by the DataLoader, and we need to cast to TensorImage since at the time this book is written PyTorch does not maintain types when indexing—this may be fixed by the time you are reading this):

对于我们批次中的每个图像和每个类，我们会得到一个7×7的特征图，告诉我们哪里的激活更高，哪里的激活更低。这将让我们看到图片的哪些区域影响了模型的决定。 例如，我们可以找出哪些区域使模型决定这只动物是一只猫（请注意，我们需要解码输入x，因为它已被DataLoader归一化，并且我们需要转换为TensorImage，因为在本书编写时机器学习库在索引时不维护类型-这可能在您阅读本文时得到修复）：

The areas in bright yellow correspond to high activations and the areas in purple to low activations. In this case, we can see the head and the front paw were the two main areas that made the model decide it was a picture of a cat.

Once you're done with your hook, you should remove it as otherwise it might leak some memory:

亮黄色的区域对应高激活，紫色到低激活的区域。在这种情况下，我们可以看到头部和前爪是让模型决定这是一张猫的照片的两个主要区域。 完成hook后，您应该删除它，否则它可能会泄漏一些内存：

That's why it's usually a good idea to have the Hook class be a context manager, registering the hook when you enter it and removing it when you exit. A context manager is a Python construct that calls __enter__ when the object is created in a with clause, and __exit__ at the end of the with clause. For instance, this is how Python handles the with open(...) as f: construct that you'll often see for opening files without requiring an explicit close(f) at the end. If we define Hook as follows:

这就是为什么让Hook类成为上下文管理器通常是个好主意，在输入hook时注册hook，在退出时将其删除。上下文管理器是一种Python 编程语言编程语言构造，当对象在with子句中创建时调用__enter__，并在with子句末尾__exit__。例如，这就是Python 编程语言编程语言处理with open（...）as f：构造的方式，您经常会在打开文件时看到这种构造，而不需要在末尾显式关闭（f）。如果我们将Hook定义如下：

we can safely use it this way:

我们可以这样安全地使用它：

fastai provides this Hook class for you, as well as some other handy classes to make working with hooks easier.

This method is useful, but only works for the last layer. Gradient CAM is a variant that addresses this problem.

Fastai为您提供了这个Hook类，以及一些其他方便的类，以使使用Hook更容易。 这种方法很有用，但只适用于最后一层。梯度CAM是解决这个问题的变体。

Gradient CAM

The method we just saw only lets us compute a heatmap with the last activations, since once we have our features, we have to multiply them by the last weight matrix. This won't work for inner layers in the network. A variant introduced in the paper "Grad-CAM: Why Did You Say That? Visual Explanations from Deep Networks via Gradient-based Localization" in 2016 uses the gradients of the final activation for the desired class. If you remember a little bit about the backward pass, the gradients of the output of the last layer with respect to the input of that layer are equal to the layer weights, since it is a linear layer.

With deeper layers, we still want the gradients, but they won't just be equal to the weights anymore. We have to calculate them. The gradients of every layer are calculated for us by PyTorch during the backward pass, but they're not stored (except for tensors where requires_grad is True). We can, however, register a hook on the backward pass, which PyTorch will give the gradients to as a parameter, so we can store them there. For this we will use a HookBwd class that works like Hook, but intercepts and stores gradients instead of activations:

我们刚刚看到的方法只允许我们计算带有最后激活的热图，因为一旦我们有了我们的特征，我们就必须将它们乘以最后的权矩阵。这对网络中的内层不起作用。2016年论文[“Grad-CAM：你为什么这么说？来自深度网络的视觉解释通过基于梯度的本地化”]（https://arxiv.org/abs/1611.07450）中引入的一个变体使用了所需类的最终激活的梯度。如果你记得一点关于反向传递的信息，最后一层的输出相对于该层的输入的梯度等于层权重，因为它是线性的层。 对于更深的层，我们仍然需要梯度，但它们不再仅仅等于权重。我们必须计算它们。在向后传递期间，机器学习库为我们计算每一层的梯度，但它们不会被存储（除了“requires_grad”为“True”的张量）。但是，我们可以在向后传递上注册一个hook，机器学习库会将梯度作为参数提供给它，因此我们可以将它们存储在那里。为此，我们将使用一个类似于“Hook”的“HookBwd”类，但截取和存储梯度而不是激活：

Then for the class index 1 (for True, which is "cat") we intercept the features of the last convolutional layer as before, and compute the gradients of the output activations of our class. We can't just call output.backward(), because gradients only make sense with respect to a scalar (which is normally our loss) and output is a rank-2 tensor. But if we pick a single image (we'll use 0) and a single class (we'll use 1), then we can calculate the gradients of any weight or activation we like, with respect to that single value, using output[0,cls].backward(). Our hook intercepts the gradients that we'll use as weights:

然后对于类索引1（对于True，即“cat”），我们像以前一样截取最后一个卷积神经网络的特征，并计算我们类的输出激活的梯度。我们不能只调用output.backward（），因为梯度只对标量有意义（这通常是我们的损失），输出是秩-2张量。但是如果我们选择一个图像（我们将使用0）和一个类（我们将使用1），那么我们可以使用输出[0， cls]. back（）计算我们喜欢的任何权重或激活的梯度，相对于该单个值。我们的hook截取我们将用作权重的梯度：

The weights for our Grad-CAM are given by the average of our gradients across the feature map. Then it's exactly the same as before:

Grad-CAM的权重由特征图中梯度的平均值给出。然后它与之前完全相同：

The novelty with Grad-CAM is that we can use it on any layer. For example, here we use it on the output of the second-to-last ResNet group:

Grad-CAM的新颖之处在于我们可以在任何层上使用它。例如，这里我们在倒数第二个ResNet组的输出上使用它：

And we can now view the activation map for this layer:

我们现在可以查看该层的激活图：

Conclusion

Model interpretation is an area of active research, and we just scraped the surface of what is possible in this brief chapter. Class activation maps give us insight into why a model predicted a certain result by showing the areas of the images that were most responsible for a given prediction. This can help us analyze false positives and figure out what kind of data is missing in our training to avoid them.

模型解释是一个积极研究的领域，我们只是在这简短的一章中触及了可能的表面。类激活图通过显示对给定预测最负责的图像区域，让我们深入了解为什么模型预测了某个结果。这可以帮助我们分析假阳性，并弄清楚我们的训练中缺少什么样的数据来避免它们。

Questionnaire

1.What is a "hook" in PyTorch? 2.Which layer does CAM use the outputs of? 3.Why does CAM require a hook? 4.Look at the source code of the ActivationStats class and see how it uses hooks. 5.Write a hook that stores the activations of a given layer in a model (without peeking, if possible). 6.Why do we call eval before getting the activations? Why do we use no_grad? 7.Use torch.einsum to compute the "dog" or "cat" score of each of the locations in the last activation of the body of the model. 8.How do you check which order the categories are in (i.e., the correspondence of index->category)? 9.Why are we using decode when displaying the input image? 10. What is a "context manager"? What special methods need to be defined to create one? 11. Why can't we use plain CAM for the inner layers of a network? 12. Why do we need to register a hook on the backward pass in order to do Grad-CAM? 13. Why can't we call output.backward() when output is a rank-2 tensor of output activations per image per class?

1.什么是机器学习库中的“hook”？ 2.CAM使用哪一层的输出？ 3.为什么CAM需要hook？ 4.查看“ActivationStats”类的源代码，看看它是如何使用hooks的。 5.编写一个hook，将给定层的激活存储在模型中（如果可能，不要偷看）。 6.为什么我们在获得激活之前调用“val”？为什么我们使用“no_grad”？ 7.使用“torch.einsum”计算模型主体最后一次激活中每个位置的“狗”或“猫”分数。 8.您如何检查类别的顺序（即索引->类别的对应关系）？ 9.为什么我们在显示输入图像时使用“解码”？ 10.什么是“上下文管理器”？创建一个需要定义哪些特殊方法？ 11.为什么我们不能将普通CAM用于网络的内层？ 12.为什么我们需要在向后传递上注册一个hook才能进行Grad-CAM？ 13.为什么我们不能调用'output.backward（）'当'输出'是一个秩-2张量的输出激活每个图像每个类？

Further Research

1. Try removing keepdim and see what happens. Look up this parameter in the PyTorch docs. Why do we need it in this notebook?
2. Create a notebook like this one, but for NLP, and use it to find which words in a movie review are most significant in assessing the sentiment of a particular movie review.

1.尝试删除保留，看看会发生什么。在机器学习库文档中查找此参数。为什么我们需要它在这个笔记本中？ 2.创建一个像这样的笔记本，但对于NLP，并使用它来查找电影评论中的哪些单词在评估特定电影评论的情绪时最重要。