Paper: How many views does your deep neural network use for prediction? (ICLR-2024)

Reviewer comment (Experiments):
Different SPLIT method will influence the final result? I think so. Hence, the proposed method indeed depends on the SPLIT method. If not, please conduct experiments for verification.

Author response:
To investigate the impact of the splitting method in the proposed method, we first conducted experiments to evaluate the prediction accuracy in terms of the number of estimated MSVs using ResNet-101 and the ImageNet validation set. We used the Voronoi partition as an additional splitting method. The results are shown in the table below.
Note that we used $\beta=8$ for the Voronoi partition, mainly because we could not finish our additional experiments with $\beta=16$ with the Voronoi partition during the rebuttal period (we will add results of Voronoi with $\beta=16$ in the camera ready if accepted).
From the table, we can see that there is a clear relationship between the accuracy and the number of estimated MSVs, despite the splitting method used in Algorithm 1.

| Alg.1 Settings              | \#MSVs=1 |   2   |   3   |   4   |   5   |   6   |   7   |   8   |   9   |   10  |
| :-------------------------- | :------: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| SLIC, $\beta=16$   |   0.538  | 0.808 | 0.865 | 0.891 | 0.905 | 0.919 | 0.919 | 0.922 | 0.923 | 0.937 |
| Voronoi, $\beta=8$ |   0.586  | 0.848 | 0.892 | 0.909 | 0.921 | 0.925 | 0.931 | 0.929 | 0.931 | 0.948 |

In Figure 16 of our updated manuscript, we compare the MSVs obtained with SLIC or Voronoi partition using the cat images in Figures 2 and 4.
Note that the estimated MSVs are shown in a single image, with each colored region representing one MSV. For example, the MSVs in Figure 2 are summarized as the upper left image in Figure 16.
In Figure 16, we report results with $\beta=8$ or $\beta=16$.
The first observation from Figure 16 is that the MSVs changes depending on the splitting method as well as the value of $\beta$.
This is mainly due to the fact that the MSVs satisfying Definition 2 is not uniquely determined, as discussed in Section 3.2.
The second but important observation from Figure 16 is that the number of estimated MSVs has a common tendency in its order given a splitting method and $\beta$ (the number of MSVs is shown in the upper part of each image).
These results suggest that the average number of MSVs has a robust relationship with prediction accuracy, while the MSVs for each image should be interpreted with caution due to its dependence on several hyperparameters.
We think that MSVs has much room for improvement as an XAI method, but we believe that our findings on its relationship to generalization make a definite contribution.
We have clarified these points in Appendix G of our updated manuscript.

Reviewer comment (Evaluation):
Will the size of view affect the final result? since in Figure 4, some msvs contain only few image region, while other contain a larger image region. Considering a msv containing more image regions often encodes more information than a msv containing few image regions, I think msv of different numbers of image regions cannot compare fairly.

Author response:
First, we would like to clarify that in the proposed algorithm we do not specify the view size in advance, and the view size is determined automatically to satisfy the minimum sufficiency.
Our results, such as Figure 1 or Table 1, suggest that the number of MSVs has a clear relationship with accuracy, even though the size of each view is not considered in these experiments.
However, there is a possibility that we can define a better measure to predict the generalization ability of DNNs by considering not only the number of MSVs but also the size of each MSV, which we would like to leave as future work.

Reviewer comment (Novelty):
This paper lacks a clear motivation for the proposed concept of MSVs. It is not adequately explained why MSVs are necessary or how they contribute to the understanding of generalization ability.

Author response:
As mentioned in Section 1, one of our main motivations for introducing MSVs is to investigate whether the multi-view concept, introduced by Allen-Zhu \& Li (2023) and shown to be related to the generalization ability of modern DNNs in terms of ensemble or distilled models, is also related to the generalization of (non-ensemble or distilled) DNNs.
Our results, such as Figure 1, provide empirical evidence that the number of views, in terms of the proposed MSVs, used in the prediction is related to the generalization performance of (non-ensemble or distilled) DNNs.

Reviewer comment (Evaluation):
How to use MSVs in real-world applications? MSVs need testing samples to predict the generalization ability of DNNs. However, if we can obtain testing samples, why do we need to predict, not measure, the generalization ability of DNN?

Author response:
We would like to emphasize that *labeled* test (or validation) samples are *NOT* required to calculate MSVs, and it is sufficient to prepare unlabeled test (or validation) samples.
Furthermore, the results of our experiments, shown in Figure 13, suggest that MSVs computed using the training dataset have a high correlation with those computed using unlabeled test (or validation) samples, assuming that these datasets follow the same distribution. This suggests the possibility of selecting the best performing model using only the training dataset.

Reviewer comment (Experiments):
Experimental results are not enough. I suggest the authors conduct experiments on NLP datasets.

Author response:
Thank you for your suggestion. As described in Section 5, we think it is possible to extend the proposed MSVs to be applied to language models. However, in this paper, we focus on computer vision tasks, including classification and object detection, and leave the NLP datasets to the future work.

Reviewer comment (Theory):
Lack of theoretical analysis. The authors do not theoretically explain the relationship between MSVs and the generalization ability of DNNs. Some XAI methods [cite1-4] have rigorous theoretical analysis to guarantee its faithfulness. I suggest the authors theoretically prove the faithfulness of MSVs.

Author response:
We agree that it is important to mathematically prove XAI methods to satisfy some axioms, such as faithfullness or sensitivity (Sundararajan et al., 2017).
In our method, the key property (i.e., axiom) is to satisfy Minimality and Sufficiency. By satisfying these properties, all the MSVs obtained can explain the behavior of the DNN in the sense that they are the smallest regions that output a certain label by the DNN. The proposed algorithm can guarantee that each view obtained satisfies Sufficiency and $\beta$-Split-Minimality. Also $\beta$-Split Minimality becomes equivalent to Minimality of our definition when $\beta = |\mathbb{V}|$, as discussed in Section 3.2. We will add proofs of these properties in our camera-ready version if accepted.
Regarding the relationship between MSVs and generalization performance, unfortunately only experimental evidence is available so far. However, we would like to emphasize that the relationship between the two is clear and highly correlated, as shown in Figure 1 and Table 1.

Reviewer comment (Novelty):
The notion of MSVs in this paper is quite similar to the Sufficient Input Subsets (SIS) proposed in [cite 1, cite 2], which is expected to be discussed in the Related Work section. SIS characterizes the minimal subset of input pixels (pixels outside of this subset is masked) for the model to achieve a certain level of confidence score. In this way, the proposed Minimal Sufficient Views seem to be a simple extension of the Sufficient Input Subsets, so the authors are encouraged to clarify the differences between the two methods.

Author response:
Thank you for pointing us to the SIS references.
We agree that the definition of SIS is similar to our MSVs, but we think there are several critical differences between the two, as follows:
* SIS papers [cite 1, cite 2] do not discuss the relationship between the number of the estimated SIS and the prediction accuracy.
In contrast, we are the first to empirically demonstrate the relationship between the number of views, in terms of MSVs, and accuracy, which we believe is the main contribution of our paper.
* As the reviewer pointed out, the definition of sufficiency is slightly different between SIS and MSVs (SIS is based on the confidence score and MSVs is based on the prediction class).
We note that the definition of minimality is also different in SIS and MSVs, which is more essential because this difference leads to the difference in the algorithmic design of the two, resulting in the efficient computation of MSVs.
Minimality in SIS is defined as "no subset of $x$ satisfies the sufficiency", while minimality of MSVs is given by "$x$ does not satisfy the sufficiency when an element is removed from $x$", meaning that the minimality condition in SIS is stronger than that in MSVs.
As stated in [cite 1], in order to satisfy minimality in SIS, the BackSelect procedure in SIS does not terminate immediately when sufficiency is violated.
In contrast, the proposed algorithm terminates the EstimateMSV procedure as soon as sufficiency is violated, resulting in a more efficient computation of MSVs.
We also propose $\beta$-split minimality to further accelerate the computation of MSVs. The table below shows the computation time required to compute MSVs with respect to $\beta$, which is the average time to obtain MSVs for a single image.

| $\beta$                      |   4  |   8  |  16  |   32  |
| ---------------------------- | :--: | :--: | :--: | :---: |
| Average Time per Image (sec) | 0.46 | 1.97 | 8.48 | 41.56 |

* Batched Gradient SIS is proposed in [cite 2] to improve the computational efficiency of the original SIS.
The advantage of the proposed algorithm over Batched Gradient SIS is that the proposed algorithm does not require gradient computation, allowing it to be applied to black-box models.

We have added a discussion of these points in Section 2 of the updated manuscript.

Reviewer comment (Evaluation):
The previous work [cite 2] has noted an interesting phenomenon: for many images in CIFAR-10 and ImageNet, the size of the Sufficient Input Subsets (SIS) is quite small (e.g., only 5% to 10% of total number of pixels) and pixels in SIS are sometimes located outside of the target object. This means that the model might learn shortcut solutions, such as using blue pixels within the sky region to predict the bird class. Since the definition of MSV is similar to SIS, I wonder if a similar phenomenon occurs in this paper. From the current figures presented in this paper, most MSVs are located on the target object and seem to have clear semantic meanings, but I’m not sure if there are some “failure cases” in which the MSVs correspond to patterns that are not related to the target object (e.g., pixels within the sky region or the grass region).

Author response:
We examined the estimated MSVs with ResNet-101 and images from OpenImages, and found an example where the regions outside the class object are specified as MSVs.
The results are shown in Figure 17 of our updated manuscript, where the image is classified as class 304 (*Leaf Beetle*), while some MSVs contain only background leaf regions.
From the definition of the MSVs, we can see in Figure 17 that the ResNet-101 model has multiple views to predict the image as a leaf beetle, some of which include only the background leaf region.

Reviewer comment (Experiments):
About the baseline value for masking the image. Although using the average value of the pixels in the training data as the baseline value is a common practice in literature, it is encouraged to test if the derived MSVs and the relationship to the generalization ability are robust under different choices of baseline values. This is because in most views (a masked image), the size of the mask is quite large, thus greatly influencing the output of the model. It is not clear if the current conclusions still hold under a different baseline value.

Author response:
We conducted additional experiments where the baseline value for masking the image was white, black, or random (with a normal distribution), using the SLIC splitting and $\beta=16$.
The rest of the experimental setup is the same as in Table 1, i.e., we computed MSVs with each baseline for images in the ImageNet validation set using ResNet-101.
We grouped the validation images according to the number of estimated MSVs and evaluated the
prediction accuracy in each group; the results are shown in the table below.

| Alg.1 Settings              | \#MSVs=1 |   2   |   3   |   4   |   5   |   6   |   7   |   8   |   9   |   10  |
| :-------------------------- | :------: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| SLIC, $\beta=16$, Average   |   0.538  | 0.808 | 0.865 | 0.891 | 0.905 | 0.919 | 0.919 | 0.922 | 0.923 | 0.937 |
| SLIC, $\beta=16$, White     |   0.595  | 0.857 | 0.897 | 0.912 | 0.915 | 0.927 | 0.926 | 0.917 | 0.931 | 0.919 |
| SLIC, $\beta=16$, Black     |   0.62   | 0.854 | 0.885 | 0.897 | 0.898 | 0.911 | 0.913 | 0.903 | 0.919 | 0.914 |
| SLIC, $\beta=16$, Random    |   0.604  |  0.86 | 0.898 | 0.916 | 0.923 | 0.914 | 0.922 | 0.942 | 0.900 | 0.919 |

We see that the accuracy of images with few MSVs (1 or 2) is relatively low, despite the baseline for masking.
The accuracy consistently increases as the number of MSVs increases when we use the average value for the baseline compared to other baseline values, especially for images with a large number of MSVs.
We have included these results in Appendix H of our updated manuscript.

Reviewer comment (Theory):
I do not quite agree with the claim that “MSVs with common features were obtained for multiple images” in the same class on Page 6. The notion of “left eye”, “right eye” are based on human perception, but it is not clear whether the model also encodes these features for inference. Moreover, the MSVs are defined in the pixel space instead of the feature space. It is not appropriate to simply claim that feature “a left eye with circular shape on a black cat” is equivalent to the feature “a left eye with an almond shape on a white cat”.

Author response:
We agree that the MSVs are defined in input (pixel) space, not feature space.
We also agree that it is not clear from our results whether the model encodes features such as "left eye" or "right eye".
What we can clearly say from Figure 4 is that the masked images are all classified as Class 285 (*Egyptian Cat*) by the prediction model (here ResNet-101), as well as the original images.
As the reviewer pointed out, the definition of "common feature" here is not mathematically clear, and we used the term based on our (authors') perception.
We think that a mechanical way to infer (well-defined) common features from MSVs of multiple images is a future research direction. An example of such an extension is reported in the SIS paper [cite 1], where clustering methods are used to estimate common patterns.
We have added a discussion of these points in Appendix G of the updated manuscript.

Reviewer comment (Experiments):
I wonder how will different superpixel methods, such as SLIC and the Voronoi partition, influence the resulted MSVs for the same input image. Will the result be similar or totally different?

Author response:
To investigate the impact of the splitting method in the proposed method, we first conducted experiments to evaluate the prediction accuracy in terms of the number of estimated MSVs using ResNet-101 and the ImageNet validation set. We used the Voronoi partition as an additional splitting method. The results are shown in the table below.
Note that we used the average as the baseline and $\beta=8$ for the Voronoi partition, mainly because we could not finish our additional experiments with $\beta=16$ with the Voronoi partition during the rebuttal period (we will add results of Voronoi with $\beta=16$ in the camera ready if accepted).
From the table, we can see that there is a clear relationship between the accuracy and the number of estimated MSVs, despite the splitting method used in Algorithm 1.

| Alg.1 Settings              | \#MSVs=1 |   2   |   3   |   4   |   5   |   6   |   7   |   8   |   9   |   10  |
| :-------------------------- | :------: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| SLIC, $\beta=16$, Average   |   0.538  | 0.808 | 0.865 | 0.891 | 0.905 | 0.919 | 0.919 | 0.922 | 0.923 | 0.937 |
| Voronoi, $\beta=8$, Average |   0.586  | 0.848 | 0.892 | 0.909 | 0.921 | 0.925 | 0.931 | 0.929 | 0.931 | 0.948 |

In Figure 16 of our updated manuscript, we compare the MSVs obtained with SLIC or Voronoi partition using the cat images in Figures 2 and 4.
Note that the estimated MSVs are shown in a single image, with each colored region representing one MSV. For example, the MSVs in Figure 2 are summarized as the upper left image in Figure 16.
In Figure 16, we report results with $\beta=8$ or $\beta=16$.
The first observation from Figure 16 is that the MSVs changes depending on the splitting method as well as the value of $\beta$.
This is mainly due to the fact that the MSVs satisfying Definition 2 is not uniquely determined, as discussed in Section 3.2.
The second but important observation from Figure 16 is that the number of estimated MSVs has a common tendency in its order given a splitting method and $\beta$ (the number of MSVs is shown in the upper part of each image).
These results suggest that the average number of MSVs has a robust relationship with prediction accuracy, while the MSVs for each image should be interpreted with caution due to its dependence on several hyperparameters.
We think that MSVs has much room for improvement as an XAI method, but we believe that our findings on its relationship to generalization make a definite contribution.
We have clarified these points in Appendix G of our updated manuscript.

Reviewer comment (Presentation):
Minor. The visualization result of GradCAM in Figure 7 is a bit weird. It is suggested to check the original GradCAM paper and compare this result with that of the original paper.

Author response:
We replaced the GradCAM result with the one applied to the **last** layer of the 4th residual layer of ResNet-101, instead of the **first** layer of the 4th residual layer. See Figure 7 of our updated manuscript and footnote 6.

Reviewer comment (Experiments):
The method sounds computationally expensive. Given that the author pitch this as a model selection/XAI method, an analysis on runtime will help.

Author response:
We evaluated the computation time of Algorithm 1 with ResNet-101 using 1000 images sampled from the ImageNet validation set used in the evaluation of Figure 1.
The experiment was performed on a single machine with a single GPU (Nvidia A100).
The results are summarized in the table below, in which the computation time is averaged over 1000 images.
We can see that the computation time decreases as we use smaller $\beta$.
From the table, it will take about less than 8 minutes to compute the MSVs for these 1000 images when $\beta=4$.
Note that the average number of MSVs still has a high correlation with the generalization ability when $\beta=4$, as shown in Figure 12.
Since MSVs can be computed in parallel with respect to each image, we can further accelerate the computation of MSVs by applying parallel computing, although the results in the table below are computed on a single thread in a single machine.
We have included these results in Appendix B.1 of our updated manuscript.

| $\beta$                      |   4  |   8  |  16  |   32  |
| ---------------------------- | :--: | :--: | :--: | :---: |
| Average Time per Image (sec) | 0.46 | 1.97 | 8.48 | 41.56 |

Reviewer comment (Writing):
Although I find the multi view idea and its relation to accuracy interesting, I find the method lack coherence. Is it an XAI method, or a model selection method?

Author response:
We believe that the main contribution of our paper is to first show the relationship between the number of views used in the prediction of DNN models and their generalization ability, where views in a prediction are defined in a simple way, which we call MSVs, and an algorithm is proposed to (approximately) compute them efficiently.
In this sense, the main application of the proposed algorithm is the selection of DNN models without the need of a labeled holdout set.

Under the observation that the multi-view perspective is related to the performance of DNN models, our secondary claim is that XAI methods, especially attribution visualization, should take multi-view into account.
The difference between MSVs as an XAI method and many existing XAI methods is that MSVs provides multiple sets of sufficient features. DNNs determine their output by combining multiple features, and there may be multiple combinations of these features.
Existing methods that present a single heat map cannot represent such a combinatorial aspect of DNNs.
In contrast, each of the MSVs presented in the proposed method corresponds to a region that is sufficient and minimal to maintain the output of the DNN. This allows for a more accurate understanding of the nonlinear behavior of the DNN.
However, we think that MSVs has a lot of room for improvement to be used as an XAI method.
For example, the dependence of the estimated MSVs on the splitting method used should be reduced to be used for XAI.

Reviewer comment (Evaluation):
In either case, the evaluation is lacking. Not sufficient comparison to existing XAI/model selection methods.

Author response:
For evaluation as a model selection method, we compared the proposed method with baseline methods, including average confidence, average entropy, and average margin, as shown in Figure 6.
These methods are relatively primitive, but we believe that these baselines are reasonable choices because, for example, average confidence has a high correlation with the test accuracy when considering only CNN models.
We agree that more sophisticated methods should be better included in the comparison, but we have had difficulty finding methods that can be applied to DNN model selection using only an unlabeled validation set or the training set itself.
We would appreciate having other candidates for comparison.

Reviewer comment (Experiments):
What is the difference between single view based method (like gradcam) with combining all MSV into a single image?

Author response:
We have added Figure 16 in the updated manuscript, which shows the estimated MSVs in a single image.
For example, the MSVs in Figure 2 are summarized as the upper-left image in Figure 16.
Single-view methods such as GradCAM show attribute importance by the intensity of a single color. In contrast, MSVs use multiple colors to represent different views, each of which is sufficient to maintain the prediction.

Reviewer comment (Writing):
Minor, but in Definition 1, c(f(x)): c has not been defined before.

Author response:
The definition of $c_f(x)$ is given in the first paragraph of Section 3.2.