Debugging the Internals of Convolutional Networks
Abstract
The ﬁlters learned by Convolutional Neural Networks (CNNs) and the feature maps these ﬁlters compute are sensitive to convolution arithmetic.
Several architectural choices that dictate this arithmetic can result in feature-map artifacts.
These artifacts can interfere with the downstream task and impact the accuracy and robustness.
We provide a number of visual-debugging means to surface feature-map artifacts and to analyze how they emerge in CNNs.
Our means help analyze the impact of these artifacts on the weights learned by CNNs.
Guided by our analysis, model developers can make informed architectural choices that can veriﬁably mitigate harmful artifacts and improve the model’s accuracy and its shift robustness
Background and Motivation
The majority of work on visualizing convolutional neural networks (CNNs) has focused on understanding the features they learn [11, 21, 34].
Techniques such as feature saliency in a given input [3, 25, 28, 34], input optimization [33, 19, 21], and concept-based interpretation [10, 12] have improved our understanding of how CNNs work [24].
Furthermore, these techniques can expose certain limitations of CNNs such as decisions based on unreliable visual cues [14].
A number of techniques were proposed to explore various aspects of CNNs beyond interpreting the features and concepts they learn.
For example, recent work has focused on visualizing the weights [22, 31] in order to analyze the role of individual ﬁlters and to expose patterns in a cohort thereof.
Another piece of work has focused on branch specialization in CNNs [32].
Besides, several techniques have been proposed to analyze CNN predictions in order to expose patterns in erroneous ones [1, 6] and to subsequently shed light into the model’s behavior.
In this work, we focus on exposing subtle effects in CNN innards caused by several aspects of convolution arithmetic [9].
These effects directly manifest in the feature maps computed within the CNN, in the form of artifacts.
These effects can further manifest in the learned weights, in the form of asymmetries.
Both issues have been partly discussed in a recent work [2].
Our contribution lies in providing generic visual means to inspect the feature maps and the learned weights for such artifacts and asymmetries.
We demonstrate how debugging these effects help improve the accuracy and robustness of CNN-based models, besides deepening our understanding of their innards.
Visualizing Feature-Map Artifacts
A naive way to inspect feature maps inside CNNs for potential artifacts is to generate them for a number of images.
However, this poses challenges regarding scalability and ability to spot the artifacts, as we explain next.
2.1 Enabling Scalable Inspection
Each layer in a CNN typically computes hundreds of feature maps for every input.
To enable scalable analysis, we aggregate the feature maps per layer to provide an overview.
Figure 1 shows the mean feature map computed for a real scene in the CNN backbone of an SSD object detector [26].
Besides reducing the information overload, aggregation via averaging often helps surface systematic artifacts since it reinforces ones that recur in multiple maps while partially washing out benign areas in the maps.
The ﬁnal map in Figure 1, highlighted with a pink arrow, demonstrates an example of these artifacts.
We will elaborate later on how convolution arithmetic can incur such artifacts.
2.2 Making the Artifacts Stand Out
Aggregating the feature maps of a layer only partially helps surface systematic artifacts in these maps.
This is because the natural activation patterns can still interfere with these artifacts (Figure 1).
To improve the visibility of the artifacts, we feed a baseline input into the CNN instead of natural images.
We found the following baselines useful for that purpose:
Random inputs We generate k images uniformly at random and feed them to the CNN.
Figuredepicts the mean feature map per layer, when using one random input ( k = 1 ).
These maps largely retain a random activation pattern, except for line artifacts that emerge around the boundaries at deeper layers.
This makes these artifacts stand out better than when feeding a natural input.
When feeding multiple input samples ( k > 1) , we aggregate the feature maps per layer further over these samples which helps smoothening the mean maps.
Constant input With a constant input, the feature maps are largely constant.
This enables analyzing subtle artifacts in these maps such as line, grid, and checker-board patterns (Figure 4).
A zero-valued input (a gray image), a maximal input (a white image), or a minimal input (a black image) can be a suited choice, depending on the dataset and the model.
Simpliﬁed input The former two options eliminate selective and localized neuron activation that occurs in CNNs when feeding natural images, because they lack visual features that trigger such activation patterns.
To analyze these patterns we can add a visual stimulus to a constant input and feed it to the CNN.
Figure 3 demonstrates an example stimulus feed into a DeepLab-V3 semanticsegmentation model [7], trained on CityScapes [8].
This reveals artifacts that potentially impact how this stimulus is processed by the CNN.
For example, in Figure 3 the stimulus leads to nine localized activation areas at layer classifier_conv_1 , spaced out at an equal distance.
Interestingly, this distance is equal to the dilation factor applied at this layer.
2.3 Drilling-down on Demand
Aggregation is useful to detect feature-map artifacts and to analyze their progression over successive layers.
Besides, we enable users to drill-down into individual feature maps of a selected layer to closely examine how these artifacts emerge.
Figure 4 depicts a number of individual maps from layer classifier_conv_1 of the aforementioned DeepLab-V3 model, generated by feeding a zero-valued input.
While the majority of these maps are constant, several maps exhibit multiple line artifacts.
These artifacts also appear when feeding a natural input, and can hence interfere with the extracted features and impact the segmentation quality.
2.4 What Causes the Artifacts?
The architectural choices that dictate the CNN arithmetic are behind the artifacts we demonstrate in previous sections.
In particular, the padding mechanism, while fundamental, introduces an artiﬁcial boundary around the feature maps, especially under 0-padding.
Over multiple convolutional layers, this boundary interacts with neighboring values and can gradually interfere with the activation patterns inside the feature maps beyond the boundary.
This interference has been investigated in details in a recent work [2] that demonstrates its signiﬁcant impact on small object detection.
Another source of the artifacts is the downsampling mechanism used, such as pooling techniques and strided convolutions.
This mechanism can amplify boundary effects and accelerate their interference with the rest of the feature maps.
Moreover, the use of strides often incurs aliasing effects [35] and checkerboard artifacts [20].
Finally, downsampling can lead to skewness in the learned weights as we demonstrate in Section 3.
Besides the above-mentioned components of convolution arithmetic, dilation factors can also inﬂuence the emergence of feature-map artifacts and the patterns these artifacts take, as illustrated in Figure 3.
Likewise, the input size also has an inﬂuence on these artifacts, since it is involved in the convolution arithmetic.
Finally, non-arithmetic architectural choices can further amplify or dampen feature-map artifacts, such as normalization schemes and antialiasing [35].
2.5 Performance Implications of Feature-Map Artifacts
The artifacts presented in the previous sections can impact the accuracy and the robustness of CNNs.
Such impact varies depending on the task, the training data, and the architecture. For example, artifacts induced by zero padding were shown to cause blind spots in single-shot detectors (SSDs), hindering small object detection at impacted areas [2]. The same study demonstrates how using mirror padding in SSDs drastically reduce these artifacts, in turn, eliminating these blind spots. The authors demonstrate the average precision of an SSD-based trafﬁc light detector improved from 49 .
58% under zero padding to 57% under mirror padding.
Padding choices were further shown to impact the accuracy of image classiﬁcation [15] and semantic segmentation [15, 18].
Visualizing Asymmetries in the Learned Weights
Besides impacting model inference, feature-map artifacts can further impact the weights learned by the CNN during training.
In particular, these weights might adapt to accommodate the presence of these artifacts during training.
We aim to reveal such adaptation patterns in the learned weights.
For this purpose we compute the mean k × k patch in the input of each convolution layer, averaged over the spatial positions, the channels, and the instances of the training set as Figure 5 illustrates.
We also compute the mean k × k kernel in the weights of the corresponding layer.
Figure 6 demonstrates our analysis on two ResNet-50 models trained on ImageNet images using two different sizes of the input images, 224 × 224 and 225 × 225 .
In each case, we depict the mean 3 × 3 patch in the padded input of each 3 × 3 convolutional layer in ResNet-50, besides the mean kernel of the layer weights.
Main Observations
We notice that the majority of mean patches are symmetric about their centers.
Surprisingly, all salient exceptions appear when the model is trained on 224 × 224 images.
Moreover, patches the exhibit asymmetry are from the input to the three layers with stride = 2 (highlighted in blue) or the layers preceding them 1 .
Interestingly, the mean kernels of these stride-convolution layers exhibit similar asymmetries, which is highly evident at layer2.0.conv2 and layer2.0.conv2 .
Moreover, these asymmetries disappear when the model is trained with 225 × 225 images.
Explaining the Asymmetries
To explain the above observations, we closely examine how downsampling layers in both models process their input, as illustrated in Figure 7.
It is evident that these layers ignore parts of the padding applied when their input size is an even number.
This happens at every downsampling layer
when the
CNN input size is 224 × 224 .
The unconsumed padding occurs only at the right and bottom sides of the feature maps.
Since ResNet models ﬁll the padding areas with zeros, failing to consume the padding at the right and bottom sides does change the statistics of the layer input.
This explains why the mean 3 × 3 patches highlighted in Figure 6 exhibit asymmetries when the CNN input size is 224 × 224 input, thanks to the over-representation of zeros at the top and left borders.
On the other hand, when this size is 225 × 225 the padding is fully consumed at all sides of the feature maps, making the mean patches highly symmetric.
Since the convolution involves multiplying 3 × 3 patches with corresponding 3 × 3 weight kernels, any asymmetries that govern these patches in the training set are likely to be reﬂected in the learned weights.
This explains the strong asymmetries we observe in certain mean kernels when the CNN is training on 224 × 224 images 2 .
This also explains the high symmetry of the mean kernels when training on 225 × 225 images.
In summary, the skewness we exposed collectively in the learned weights at certain layers is the
CNN’s way to adapt to the uneven application of padding to the input of these layers.
This skewness is an artifact of CNN arithmetic: It does not capture inherent semantic information about the classiﬁcation task since it disappears when uneven application of padding is eliminated.
Ensuring even application of padding was shown to improve the accuracy and the translation invariance of the
CNN on ImageNet [2].
This demonstrates the value of exposing potential artifacts of CNN arithmetic.
Limitations Exposing asymmetries in the learned weights becomes challenging with kernels larger than 3 × 3 in size, unless windowing is applied [29].
Moreover, the asymmetries might not necessarily be an artifact of CNN arithmetic, but rather reﬂect semantic properties of the dataset such as dominant vertical strokes in handwritten digits or letters.
Analyzing the dataset features and examining the weight kernels under different hyperparameters can help determine possible reasons for asymmetry.
Lessons learned
We demonstrated how visual analysis of CNN internals can expose critical issues in CNNs that might otherwise go unnoticed.
In fact, it has been known that certain architectural choices can introduce artifacts, however, CNNs were assumed able to implicitly handle them.
We hope the following components of our analysis could inspire similar debugging and exposition techniques:
Baseline inputs Feeding a constant or near-constant input to a CNN helps simplify the internals it computes which, in turn, helps surface abnormalities in the internals.
Feeding random inputs can also help in exposing systematic bias in these internals.
Aggregation and Ensemble Analysis Computing the mean feature map and the mean kernel in a layer is crucial to expose systematic spatial bias and asymmetries at that layer.
Aggregation enables these patterns to emerge collectively, unlike the analysis of individual ﬁlters or feature maps.
Visual Representation Considering multiple visual representations of the model internals is helpful to iteratively reveal subtle patterns and variations that might inﬂuence the model behavior.
Figuredemonstrates how a naive depiction of the mean image in ImageNet (Figure 8a) falls short of revealing spatial variations in this image, unlike a heatmap (Figure 8b).
These variations help explain the banding effects in the learned weights, recently observed by Petrov et al [22] (Figure 8c), and conﬁrm the role of the training data in the emergence of these effects.
Insights From Signal Processing Several phenomena related to CNNs have already been studied in signal and image processing.
For example, zero padding has been known to cause image artifacts [16], with several alternatives proposed in the literature to mitigate them [17].
Likewise, insights from sampling theory have helped reducing aliasing effects in CNNs [35], and windowing techniques were shown to reduce spectral leakage and the artifacts it induces in CNN internals [29].  Related Work
A variety of prior art shares similar goals or employs similar analysis techniques to the ones we presented.
For example, image artifacts have been studied extensively in the context of generative models, focusing on how they develop in the model internals and how they can be mitigated [4, 30].
Likewise, Shocher et al [27] studied the impact of downsampling mechanisms on the learned ﬁlters. Qian et al [23] use mean attention maps to demonstrate aliasing issues in vision transformers. Finally, baseline inputs are widely used in in various feature-attribution algorithms that help explain model decisions [13] .  Conclusion
Convolution arithmetic can be a source of artifacts that impact the innards of CNNs.
These artifacts manifest both in the feature maps and in the weight kernels.
We presented visual means to expose these artifacts, to analyze why and how they emerge, and to understand their impact on the task.
Our open-source means leverage the power of visualization along with deliberate inputs and aggregation schemes.
We exempliﬁed several insights enabled by our analysis and demonstrated how they help model developers make informed decisions to mitigate the aforementioned artifacts and biases, and as a result improve the accuracy and robustness of their models.
Future work can dive deeper into how the effects we expose impact speciﬁc CNN tasks and whether similar effects appear in transformer-based models and hybrid CNN and transformer models.