Examining neural network behavior at the classiﬁcation boundary
Abstract
We evaluate the classiﬁcation of both human volunteers and various neural network models on a set of GAN-generated images that reﬂect the transition from one
MNIST class to another.
We ﬁnd that models that obtain the same test accuracy on the standard MNIST test data set exhibit different behavior on these images.
Further, we ﬁnd that although the number of misclassiﬁed images decreases with test accuracy, the spread in predictions over multiple runs on images that are difﬁcult to classify (for humans) also decreases with test accuracy.
Our results raise the question of how we want networks to behave on images that could plausibly belong to multiple classes and hint at the value of complementing test accuracy with other evaluation metrics.
Introduction
Two people manually classifying images may easily obtain the same accuracy on a test data set.
How do we determine which of the two is a better classiﬁer?
One option is to extend the challenge to a range of different data sets and tasks and evaluate their performance on each of these.
Examples of such benchmarks are the Visual Task Adaptation Benchmark (VTAB) [9] for vision and SuperGLUE [8] for natural language processing models.
However, the choice of these benchmark data sets may skew the direction of machine learning research as discussed in a recent preprint on “The Benchmark
Lottery” by Dehghani et al.
A further limitation that we want to discuss in this work is that across all these data sets the metric that we use to evaluate classiﬁers remains the same: the performance on a held out test set.
To be accepted in any of the leading machine learning journals, a paper that proposes a new neural network optimization scheme is generally expected to prove that its test accuracy on certain popular data sets, e.g., ImageNet for vision, matches (ideally exceeds) the state of the art.
This effectively constrains the development of optimization schemes to a class of algorithms that will perform well using the metric and benchmark data sets generally accepted by the community.
If we consider our search for new optimization schemes as being analogous to traversing a loss landscape, then overﬁtting on speciﬁc metric and benchmark choices will result in getting stuck in a local minimum, which is undesirable.
Although expanding our range of data sets and metrics does not entirely resolve the problem, we argue that more research on this topic will lead to better and more robust optimizers.
In this work we study whether a model’s accuracy on the standard test data set reﬂects its performance on a set of images that lie along the classiﬁcation boundary between two known classes.
We do this by comparing the classiﬁcations of different network architectures with identical test accuracies to the classiﬁcations provided by a group of human volunteers.
Our experiment raises the question of how we want networks to behave on images that could belong to either of two known classes and independently illustrates the potential limitations of using test accuracy as the sole metric for model evaluation.
[Table]
Experiment design
In this experiment we consider the behavior of different neural network models with identical test accuracies on images that capture the transition from one MNIST class to another.
We measure their performance and compare their predictions with the labels assigned by a set of human volunteers.
Creating the data.
We trained a Deep Convolutional Generative Adversarial Network (DCGAN) [6] on MNIST data [4].
We then used the trained DCGAN to generate images, selected ten images depicting the handwritten numbers 0 to 9, and stored the corresponding ten 100-dimensional vectors z 0 , . . .
, z 9 which formed the input to the generator.
To linearly interpolate between any given z i and z j (for i (cid:54)
=
j ) we used the following equation:
[Equation]
To create our data set we considered 15 examples consisting of 15 pairs of ( z i , z j ), where we set i <
j and had each handwritten number represented three times, i.e., each z k for k ∈ { 0 , 1 , . .
, 9 } appears three times across the 15 pairs.
We then linearly interpolated between each pair using Eq.
(1) with 11 values of α equally spaced between 0 and 1 (inclusive).
This resulted in a total of 165 images (Appx. B).
For illustration we have provided the linear interpolation between a ( z i , z j ) pair in Fig.
= 0.
= 0. = 0. = 0.
= 0. = 0.
= 0. = 0.
= 0. = 0.
= 1.
Obtaining the neural network baseline.
We trained a multilayer perceptron (MLP), convolutional neural network (CNN) [4], and Vision Transformer (ViT)
[2] architecture using cross entropy loss and the Adam [3] optimizer with different hyperparameter choices (more details can be found in
Appendix A) on the normalized MNIST training data set.
We stopped training once a model reached a certain target accuracy ± 0.05% on the test data set.
We considered a maximum target test accuracy of 99.5% (for the CNN architecture) and 98.5% (for the MLP and ViT architectures).
The resulting models were then fed the DCGAN-generated images.
Obtaining the human baseline.
To obtain a human baseline for the DCGAN-generated images, we recruited 10 volunteers to manually label the images.
The images were shufﬂed and then distributed to each of the labelers.
The volunteers were not informed of the goal of the task, but were simply asked to label the images with numbers between 0 and 9 to the best of their ability, without consulting others.
We then anonymously processed the data and re-sorted the manual labels.
Results
We consider the classiﬁcation provided by the human volunteers to be the ground truth.
This means that if all human classiﬁers agree on a speciﬁc label for an image we consider this to be the true label.
On the other hand, if one or more of the human classiﬁers disagree on the label for a speciﬁc image, we consider this image to be a ‘boundary case’ example.
1 We will contrast the (dis-)agreement between the human classiﬁers with the (dis-)agreement between an equally large number of neural network (NN) models.
As summarized in Table 1 the resulting classiﬁcations by the neural network models on the DCGAN-generated images can then be grouped into three categories:
Type A: ‘Misclassiﬁcation’: Images on which neural network models disagree, whereas human classiﬁers agree.
Type B: Matching ‘ground truth’:
Images on which neural network models (dis)agree and human classiﬁers also (dis)agree.
Type C: Images on which neural network models agree, whereas human classiﬁers disagree.
is the
) images both network
[Table]
To compare the disagreement amongst a group of neural network models we need to decide which models to group together.
We focus on the two extreme cases: the exact same model trained with different random seeds and a diverse set of models (MLP, CNN, and ViT) trained with different optimization schemes (full details in Appx.
A).
The former experiment illustrates the differences between model architectures, whereas the latter is particularly useful for comparing neural networks with humans.
Each human is unique and hence we want to contrast the disagreements within the human classiﬁers with the disagreements amongst an equally diverse set of neural network models.
Do models with the same test accuracy misclassify the same images?
We consider the diverse set of neural network models with equivalent test accuracies.
Table 2 shows that the average % of images on which 2 models disagree is much lower than the % of images on which all 9 models disagree.
Further, when focusing on the images on which the human classiﬁers agree, we see that the actual % of misclassiﬁed images, i.e., on which the label predicted by the neural network is different from the unanimous label assigned by human classiﬁers, is also much smaller per model than when measured across all 9 models.
This illustrates that models that were trained to the same test accuracy behave differently on the DCGAN-generated data set and misclassify (and disagree on) different images. out of the 9) disagree.
The “9 models disagree” class reﬂects the number of images for which at least one model of the 9 has a different label (a more nuanced view on the level of ‘disagreement’ is discussed in Appx.
C).
[Table]
Do neural networks disagree on the same images as human classiﬁers?
A human classiﬁer disagrees with another human classiﬁer on average on 9.3 ± 2.2% of the data set.
As shown in
Table 2 this falls within the same range as the number of disagreements between two neural network models (4th column), which raises the question of whether the neural networks disagree on the same images as the humans, namely the complicated ‘boundary-case’ examples.
In Table 3 we compare the disagreements among the 9 neural network models with the disagreements between 9 humans.
There exist a signiﬁcant number of images on which either humans agree but neural network models do not (type A) or on which humans disagree whereas neural network models agree (type C).
It is surprising that for such a diverse set of neural network models there is relatively little disagreement on the for humans difﬁcult to classify ‘boundary-case’ examples.
[Table]
How does the maximum softmax value across models vary with the class?
In Table 4 we show that the maximum softmax value across models is much lower for type A images than for type C images.
Further, the maximum softmax value for images on which both humans and models disagree is higher than for images on which models disagree but humans do not (type A). Instead we would expect the ‘boundary case’ examples on which the humans also disagree to have similar maximum softmax values as the other examples on which the models disagree.
[Table]
How does the image type vary with respect to test accuracy of the considered models?
In Figure 2 we consider the (dis)agreements between the predictions of a network across 10 random seeds that was trained each time to the same test accuracy.
Standard deviations are obtained by repeating thistimes (i.e., over 100 different seeds in total).
Type A images form a clear representation of a speciﬁc handwritten digit as all the human volunteers agreed upon their label.
We consider type A images to be misclassiﬁed by the neural network models and hence we expect the number of type A images to decrease with test accuracy of the considered models.
For all three model architectures considered (MLPs, CNNs, and ViTs) we indeed observe that the percentage of type A images decreases with test accuracy (Figure 2, left).
Although our expectation for images on which human classiﬁers agree is clear, it is debatable what the desired behavior of a set of neural networks is for images on which human classiﬁers disagree.
However, if a set of models is initially split across multiple labels for a (for humans) difﬁcult to classify image, but then transitions to predict a unanimous label, we consider this to be an undesirable change as there does not exist a single true answer in this setting.
In Figure 2 (right) we study how the number of type C images (on which humans disagree, but neural networks agree) changes with the test accuracy of the considered models.
We ﬁnd that the number of difﬁcult to classify images that neural networks unanimously agree on increases with test accuracy.
Finally, Figure 2 clearly shows the variations across different network architectures, which were trained using the same optimizer until the same test accuracy was reached.
An MLP architecture (blue) misclassiﬁes the lowest number of images (type A) for any given test accuracy, but also predicts ‘boundary-case’ images with the largest ‘certainty’ (type C).
Test accuracy is therefore not an accurate indicator of the performance of these models on the considered data set.
[Figure]
Conclusion
To show the limitations of using test accuracy as the only metric for model evaluation we created a set of DCGAN-generated images that capture the transition from one MNIST class to another.
We found that models with the same test accuracy misclassify (and disagree on) different sets of images.
Further, we ﬁnd that although a higher test accuracy corresponds with neural network models becoming more unanimous in their label prediction, this may be undesirable for those images on which humans disagree as they do not have a clear class label associated with them.
Our experiments also highlight the differences between human and model predictions, as neural networks disagree amongst themselves on a different set of examples than where human classiﬁers disagree and when in disagreement have higher maximum softmax values on those images on which humans disagree.  Directions for future research
The study performed in this work was restricted to 10 human volunteers, who each labeled a total of 165 images.
It would be valuable to recreate this experiment with a larger amount of human classiﬁers and images, and to consider other datasets.
This would result in a more accurate measure of human certainty and allow richer comparison between which examples humans and networks ﬁnd ‘difﬁcult’.
As suggested by a reviewer it would be interesting to then take the label distribution provided by the human labelers to train a model and compare this with knowledge distillation. To further expand the scope of the experiment, a meticulous study of how the architecture and optimizer choices affect the behavior of the network could potentially lead to insights into inductive biases. For example, in Figure 2 the chosen MLP architecture misclassiﬁes the lowest number of images for any given test accuracy, but is also more unanimous in its label prediction on images on which humans disagree.
Establishing that this observation holds more generally and if so, understanding why it holds, would be valuable.
Furthermore, in this paper, we investigated models trained exclusively on MNIST data.
It would be interesting to perform the same experiments with CLIP [7], an image-text bi-encoder pretrained on a large corpus of image-text pairs.
CLIP is capable of classifying images without ﬁnetuning, and has been shown to possess remarkable generalization capabilities.
As such, we might ﬁnd that the behaviour of CLIP meaningfully differs from the behavior of the models we investigated here and more closely aligns with human behavior.
Finally, we believe there is value in exploring other metrics for model evaluation that could complement test accuracy.
For the study performed in this work, a ﬁrst step towards this goal would be to establish what the desired behavior of an ensemble of networks would be on images that could plausibly belong to multiple classes.
More generally, it would be valuable to seek out additional settings in which complementing test accuracy with different metrics might be beneﬁcial.
Acknowledgements
The author wishes to thank the human volunteers for their time and the reviewers for helpful comments and for suggesting interesting directions for future work.
The author is supported by the Maxwell
Institute Graduate School in Analysis and its Applications, a Centre for Doctoral Training funded by the UK Engineering and Physical Sciences Research Council (grant EP/L016508/01), the Scottish
Funding Council, Heriot-Watt University, and the University of Edinburgh
A Experimental details
Generating the data.
As the generator of the DCGAN architecture [6] we use ﬁve convolutional transpose layers with feature map size 64 with ReLU activation and batch normalization.
For the discriminator we use ﬁve convolutional layers with Leaky ReLU activation and batch normalization.
The weights are randomly initialized from N (0,0.02) and both models are trained using Adam [3] with h = 2e-4 and β 1 = 0 .
5 as suggested by Radford et al.
All other details on the data generation are provided in Section 2.
Model architectures.
We consider three different neural network architectures: a multilayer perceptron (MLP), a convolutional neural network (CNN) [4], and a Vision Transformer (ViT) [2].
As
MLP architecture we used a hidden layer with 500 input and output nodes and ReLU activation.
Our
CNN architecture has 4 convolutional layers and 3 fully connected layers, ReLU activation, batch normalization, max pooling, and dropout ( p = 0 . 4 ) on the middle fully connected layer.
Finally, the Vision Transformer (ViT) architecture is adapted from Dosovitskiy et al.
[2], where we usetransformer blocks, 8 heads in the multi-head attention layer, and a MLP size of 128.
Training.
We train the models using PyTorch [5] on the normalized MNIST [4] training data set with the Adam optimizer.
Training is stopped when a model has reached the target test accuracy ± 0.05% on the MNIST test data set.
To create the diverse set of models we train three models of each architecture type: for MLP1, CNN1, and ViT1 we use Adam with step size h
= 5e-4, batch size ( BS ) =
128 and no weight decay (WD).
For MLP2, CNN2, and ViT2 we use AdamW with h = 5e-4, BS = 128, and WD = 1e-05.
To introduce additional variation we train MLP3 using h = 1e-3,
BS = 256, WD = 1e-05; CNN3 without weight decay but using additional data augmentation; and
ViT3 using WD = 5e-05.
To create Figure 2 we train the same network across 10 random seeds to the same test accuracy using a batch size of 128 and the Adam optimizer with h = 5e-4.
We repeat this process 10 times (so using a total of 100 random seeds) to obtain standard deviations.
B Dataset
In Figure 3 we provide the full 165 DCGAN-generated images considered in this work.
These images were shufﬂed before being given to the human volunteers.
On top of each image we provide a (cid:88) if all 10 human classiﬁers agreed on the label for this image.
If for an image there was disagreement between the human classiﬁers, we state the different labels proposed.
[Figure]
C Further results
In the paper we focused on showing the limitations of test accuracy as the sole metric for model evaluation.
The results in this section further support this statement and try to answer some follow-up questions the reader may have.
Is an increased disagreement among human classiﬁers reﬂected by the behavior across neural network models?
Throughout this work we have categorized the human classiﬁcations into two groups: images on which the humans agree and images on which they disagree.
We now want to take a closer look at the latter category.
We consider the number of humans disagreeing with the majority vote as a measure of human certainty, i.e., if 2 humans disagree with the majority vote the humans are 80% certain.
Correspondingly, we can measure the level of ‘certainty’ of neural networks by studying how many networks disagree with the most prevalent label assigned to a speciﬁc image across all 10 networks.
The results are summarized in Table 5.
It is clear that there is a lot of variation across models that have the same test accuracy.
Further, there is not a clear correlation between the human certainty on speciﬁc images and the model certainty.
CNNs seem to have the lowest overall certainty (but still much higher than humans) on images where humans are ≤ 80% certain on.
Since we only have 10 human volunteers in this study, it may be worth repeating this experiment with a much larger sample size to give a more accurate representation of human certainty.
Further, the number of images in each "human certainty" class is rather limited and so scaling up along this axis is also worth exploring in future experiments.
Model certainty on images that humans are hc % certain on
Test acc
Model hc = 90% hc = 80% hc = 70% hc
[Table]
Can we visualize how the human and network (dis-)agreements are distributed across the dataset?
In Figure 4 we visualize using a heatmap on which images humans disagree amongst themselves, but CNNs do not (in red) and on which images CNNs disagree amongst themselves, but humans do not (in blue).
The images on which both networks and humans agree are indicated in gray, while images on which both networks and humans disagree are indicated in white.
The signiﬁcant number of red and blue squares clearly shows that CNNs disagree on a different set of images than the human classiﬁers.
Further, we observe that both the number of blue squares (on which CNNs disagree, but humans do not) and the number of white squares (on which both CNNs and humans disagree) decrease with test accuracy of the considered models, which corroborates the results shown in Figure 2.
[Figure]