Paper: A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis (ICLR-2024)

Reviewer comment (Experiments):
Lack of interpretability analysis. As mentioned by the author this paper is not about accuracy, hence I shall expect more quantitative/qualitative experiments to fully understand inner works. Since the author says INTR can detect the "attributes", the author can measure how precise the model can detect. What is the "accuracy" in detecting the attributes (e.g., Sec 4.2 of [1])? What are the captured attributes? Are these attributes shared or different among the different classes (e.g., it can detect the stripe of a bird, are they the same stripes or different in colors/patterns)? Why is that? I think all these results can be shown in the main paper.

Author response:
**Lack of interpretability analysis.**
Thank you for the suggestion and reference. We checked the experiment in Sec 4.2 of [1] and found that it may not be feasible for our INTR. First, INTR does not impose regularizers to segment an object into parts. Second, the experiment will require training a regressor to localize part locations, which needs detailed part annotation. 

With that being said, we have conducted additional quantitative studies and discussions. First, the attributes that INTR captures are local patterns (specific shape, color, or texture) useful to characterize a species or differentiate between species. These attributes can be shared across species if the species are visually similar. These can be seen in Figure 23 and Figure 24. In Figure 23, we applied the query of Heermann Gull to the image of Heermann Gull (first row) and Ring Billed Gull (second row). Since these two species are visually similar, several attention heads identify similar attributes from both images. However, at Head 8, the attention maps clearly identify the unique attribute of Heermann Gull that is not shown in Ring Billed Gull. Please see the caption for details. 

In Figure 24, we present the cross-attention map activated by the ground-truth queries for two closely related species, the Baltimore Oriole and the Orchard Oriole. Additionally, we manually document some of the attributes by checking whether the attention maps align with those human-annotated attributes in the CUB dataset. We found that INTR can identify shared and discriminative attributes in similar classes.

When the species are visually dissimilar (cf. Figure 9), we see that the attributes specifically to each species (every column) cannot be found in other species (the off-diagonal elements of every row). 

We will provide more discussions in the final version and move important messages into the main paper.

Reviewer comment (Evaluation):
Comparison with other interpretable models. The only comparison I noticed is Fig 3. Are other models not able to identify attributes like INTR does? How do you quantitatively measure it? From what I see in this figure, ProtoPFormer is also able to locate local attributes, but there are not enough samples here.

Author response:
**Comparison with other interpretable models.**
Besides Figure 3, we include more comparisons in Figure 10 and Figure 25. As mentioned on page 7 (the paragraph before Sect. 4.2), we do not claim that these compared methods cannot detect attributes. We include them mainly as references to show what our approach can achieve with a simple and extensible model design. One particular advantage of INTR is that it can easily detect attributes of different sizes/shapes. In contrast, ProtoPNet, ProtoTree, and ProtoPFormer often need to pre-define the prototype sizes/shapes. 

To quantify the difference between our INTR and ProtoPFormer, we conducted a human study. We randomly selected five images from each of the four species selected randomly from the CUB dataset.  To verify the attributes detected by models, we provide the models’ interpretation attention maps and image-level attribute information (from the CUB metadata) to seven individuals who do not have any knowledge about the project. We ask them to list all the attributes that are captured by the attention maps. One attribute is considered detected by a model if more than half of the individuals identify it from the attention maps. Our quantitative analysis reveals that INTR outperforms ProtoPformer, achieving an attribute identification accuracy of 74.7% compared to ProtoPformer's 42.2%. 

We will expand this study to contain more individual observers and bird species in the final version.

Reviewer comment (Evaluation):
Image manipulation experiments seem not systematic enough, how do you measure the changes? E.g., after manipulating the "part", how many images changed the classification results? Are they still the correct or the wrong results?

Author response:
**Image manipulation experiments.**
We initially provided Figure 5 as an analysis, not an experiment. Thanks to your suggestion, and we expanded the study to be more systematic. We considered images from three species of the CUB dataset namely Red-winged Blackbird, Scarlet Tanager, and Orchard Oriole. We removed the red spot of the Red-winged Blackbird (Figure 5), edited Scarlet Tanager by changing the black wings and tail to red to make it look like Summer Tanager (Figure 26), and replaced the color of Orchard Oriole’s belly from brown to orange to make it look like Baltimore Oriole (Figure 5). All of these manipulations follow the attribute guidance described in https://www.birds.cornell.edu/home/. We randomly selected 10 images from each of these species for manipulation and observed that 29 cases changed their prediction after manipulation (out of 30), which shows a 96.7% success rate. This holds the property that INTR can identify tiny image manipulations that distinguish between classes.

Reviewer comment (Evaluation):
The attention map now looks a bit not "clean" enough, when some attributes cannot be detected from the image, I suppose the attention should be empty instead of looking at the background, if it is looking at the background, does it mean it is a feature? or it is overfitting? The author did not further analyze it.

Author response:
**Attention maps.**
Thank you for the comments. It is worth noting that in our original submissions, Figures 12-17 are indeed hard classes, as we want to ensure that every picked class has misclassified examples (the last row) to match the presentation style of Figure 1. Given the high performance of the Butterfly (BF) dataset, selecting a challenging case (the one pointed out by the reviewer in Fig. 13) was necessary to maintain this pattern. 

In the rebuttal, we further present Figures 27-28, which are classes without misclassified cases. These attention maps are much clearer.

Furthermore, it is worth highlighting that the attention heads presented are normalized based on attention scores, which may, in some instances, result in the background being highlighted. However, when we sort them based on the un-normalized scores, we find that only the essential attributes are highlighted, as illustrated in Fig. 4.

We also note that, according to https://www.birds.cornell.edu/home/, habitats are also considered important messages to differentiate species. Thus, we do expect for some species, the attention maps may focus on the background.

Reviewer comment (Evaluation):
What is the "accuracy" in detecting the attributes (e.g., Sec 4.2 of [1])? What are the captured attributes? Are these attributes shared or different among the different classes (e.g., it can detect the stripe of a bird, are they the same stripes or different in colors/patterns)? Why is that?

Author response:
**Lack of interpretability analysis.**
Thank you for the suggestion and reference. We checked the experiment in Sec 4.2 of [1] and found that it may not be feasible for our INTR. First, INTR does not impose regularizers to segment an object into parts. Second, the experiment will require training a regressor to localize part locations, which needs detailed part annotation. 

With that being said, we have conducted additional quantitative studies and discussions. First, the attributes that INTR captures are local patterns (specific shape, color, or texture) useful to characterize a species or differentiate between species. These attributes can be shared across species if the species are visually similar. These can be seen in Figure 23 and Figure 24. In Figure 23, we applied the query of Heermann Gull to the image of Heermann Gull (first row) and Ring Billed Gull (second row). Since these two species are visually similar, several attention heads identify similar attributes from both images. However, at Head 8, the attention maps clearly identify the unique attribute of Heermann Gull that is not shown in Ring Billed Gull. Please see the caption for details. 

In Figure 24, we present the cross-attention map activated by the ground-truth queries for two closely related species, the Baltimore Oriole and the Orchard Oriole. Additionally, we manually document some of the attributes by checking whether the attention maps align with those human-annotated attributes in the CUB dataset. We found that INTR can identify shared and discriminative attributes in similar classes.

When the species are visually dissimilar (cf. Figure 9), we see that the attributes specifically to each species (every column) cannot be found in other species (the off-diagonal elements of every row). 

We will provide more discussions in the final version and move important messages into the main paper.

Reviewer comment (Evaluation):
Are other models not able to identify attributes like INTR does? How do you quantitatively measure it?

Author response:
**Comparison with other interpretable models.**
Besides Figure 3, we include more comparisons in Figure 10 and Figure 25. As mentioned on page 7 (the paragraph before Sect. 4.2), we do not claim that these compared methods cannot detect attributes. We include them mainly as references to show what our approach can achieve with a simple and extensible model design. One particular advantage of INTR is that it can easily detect attributes of different sizes/shapes. In contrast, ProtoPNet, ProtoTree, and ProtoPFormer often need to pre-define the prototype sizes/shapes. 

To quantify the difference between our INTR and ProtoPFormer, we conducted a human study. We randomly selected five images from each of the four species selected randomly from the CUB dataset.  To verify the attributes detected by models, we provide the models’ interpretation attention maps and image-level attribute information (from the CUB metadata) to seven individuals who do not have any knowledge about the project. We ask them to list all the attributes that are captured by the attention maps. One attribute is considered detected by a model if more than half of the individuals identify it from the attention maps. Our quantitative analysis reveals that INTR outperforms ProtoPformer, achieving an attribute identification accuracy of 74.7% compared to ProtoPformer's 42.2%. 

We will expand this study to contain more individual observers and bird species in the final version.

Reviewer comment (Evaluation):
I understand that the attention map is not regularized by any loss functions so the attention maps may look weird. But how do you explain the attention focus on the background, but sometimes focus on the "attributes" (e.g. Fig 13)? How can we know what attributes will fire up which heads?

Author response:
**Attention maps.**
Thank you for the comments. It is worth noting that in our original submissions, Figures 12-17 are indeed hard classes, as we want to ensure that every picked class has misclassified examples (the last row) to match the presentation style of Figure 1. Given the high performance of the Butterfly (BF) dataset, selecting a challenging case (the one pointed out by the reviewer in Fig. 13) was necessary to maintain this pattern. 

In the rebuttal, we further present Figures 27-28, which are classes without misclassified cases. These attention maps are much clearer.

Furthermore, it is worth highlighting that the attention heads presented are normalized based on attention scores, which may, in some instances, result in the background being highlighted. However, when we sort them based on the un-normalized scores, we find that only the essential attributes are highlighted, as illustrated in Fig. 4.

We also note that, according to https://www.birds.cornell.edu/home/, habitats are also considered important messages to differentiate species. Thus, we do expect for some species, the attention maps may focus on the background.

Reviewer comment (Evaluation):
The (potential) influence of the encoder is not really discussed in this work; Standard DETR already employs a quite large CNN to ‘encode’ the information into embeddings, and one could argue that certain ‘decisions’ as to what is important are already made there. Note that the receptive field of each element of the feature map emitted by the encoder is substantial (potentially even global, especially when ViT is used) and therefore could incorporate information of various locations throughout the actual input image. Please correct me if I’m wrong, but the CA essentially shows us to which locations of the feature map the classifier is paying attention to – which does not necessarily have to directly correspond to the input image? I’d like to see some discussion/clarification of this point.

Author response:
**Influence of Encoder.**
Thank you for the insightful question. The large receptive field of a deep CNN or a plain Transformer indeed indicates that every feature map location has been affected by many pixel locations in the input image. On the one hand, this helps capture the context in the image (to differentiate visually similar pixels or local patterns). On the other hand, it may lose the localizability — each feature map location does not necessarily correspond to its associated pixel/patch locations. 

That said, we think several design choices in ResNets and the Transformers used in vision (e.g., ViT or the encoder in DETR) help improve localization. First, both of them have “skip” links. That is, there is a shortcut from the input pixel/patch to the corresponding feature map location. Such a skip link has been shown to improve localization in semantic segmentation (e.g., U-Net) and object detection (e.g., spatial pyramid pooling) as well. In [A], the authors empirically showed that a significant amount of feature information is passed from the previous layers to the later ones through the skip links. Second, in the encoder of DETR, fixed positional encodings are added to all the encoder layers. We surmise that this would further strengthen the location correspondence between the input image and the feature maps. Finally, as evidenced by recent work of ViTs like DINO-v1 and DINO-v2, we do see a fairly accurate correspondence of the object/part boundaries in the feature map and in the image.

We hope that these discussions address your concern about the interpretability of INTR.

[A] Maithra Raghu et al., Do Vision Transformers See Like Convolutional Neural Networks? NeurIPS 2021

Reviewer comment (Experiments):
Self-attention in the Transformer Decoder quickly becomes prohibitive – as the authors correctly state. It is still worth noting that the complexity in memory and time grows quadratically with the number of classes due to the self-attention operation performed across the class-specific query tokens. I’d be curious to know whether the authors might have some suggestion whether all classes are required to be included as decoder input ‘all the time’, or whether some pre-selection could be performed to reduce the computational complexity, at least at inference time.

Author response:
**Efficiency.**
Thank you for the insight on the quadratic growth of computation. We have indeed been investigating methods to reduce it. The first method is to perform hierarchical classification following the available taxonomy (e.g., the organism taxonomy). We can learn queries for different taxonomy levels (e.g., for different families, genera, and species). During inference, we then apply these sets of queries in sequence. In other words, we do not need to input all the species-level queries to the decoder at once; only those belonging to the predicted genus are needed. Consider a two-level hierarchy with C^2 classes, where every C of them is grouped. Hierarchical inference takes 2*O(C^2) computation, largely saving it from O(C^4) by the plain inference.

Even without hierarchical training, one may take a pre-trained INTR, perform hierarchical clustering to group the classes (e.g., based on the validation confusion matrix), and select one representative class to represent each cluster. During inference, we then use those representative classes, from the root of the hierarchy down, to perform hierarchical classification. In Fig. 21, we show an example that the top predicted classes by INTR indeed share similar appearances and genera; one can thus use one query to represent the others. In Fig. 6, we also show that, by taking only the most visually similar queries as input, INTR can identify finer-grained attributes.

Reviewer comment (Experiments):
The authors state that they employ the DETR model pretrained on ImageNet and MSCOCO – which is a lot of data that importantly provides various different aspects of images (classification, as well as (multi-object) detection data). While the authors acknowledge this fact in the manuscript, it is still worth pointing out since it is a potential point of concern that introduces some uncertainty as to how backbones trained on less data or ‘only ImageNet’ might perform when paired with INTR.

Author response:
**Pre-training influence.** 
Thanks for carefully checking our paper, especially the footnote on page 6 that discusses our usage of an MSCOCO-pre-trained model. 

During the rebuttal period, we tried to pre-train the INTR model from scratch, i.e., randomly initializing the Transformer encoder and decoder in DETR, using the ImageNet data. Unfortunately, we found the hyper-parameter setting of DETR not directly applicable to this pre-training. While we have tried to perform hyper-parameter selection, we were short of time to find a good configuration. We will try to continue the study. 

As an alternative, we take the pre-trained DETR model and replace its encoder (i.e., ResNet-50 + Transformer encoder) with a separately pre-trained DeiT-S (using ImageNet-1K) and ViT-H (using ImageNet-21K). Details of this study are in Appendix Section F. We found that using a DeiT-S or ViT-H encoder trained without MSCOCO can sometimes outperform the original ResNet-50 + Transformer encoder in terms of **accuracy**, as shown in Table 4. However, in terms of the  **interpretation/visualization**, using the pre-trained DETR model is still the best. Figure 22 shows an example. We attribute this to two reasons: 1) the pre-trained DeiT-S and ViT-H are not well-aligned with the pre-trained decoder, and 2) pre-training with detection data does help with focusing on details.

With that being said, we respectfully think that these results should not undermine our main contributions and claims. First, DETR is a model architecture designed initially for detection. Training it from scratch may need some careful investigation. Second, with the accessibility to large-scale detection datasets and pre-trained models, building a model from the most suitable starting point should be allowed and encouraged. In our case, the MSCOCO dataset does not contain fine-grained class labels and part annotations, quite different from our downstream tasks. Besides, what we found interesting is that while existing literature mostly uses classification pre-trained models to facilitate other downstream tasks, we show that the opposite—using detection pre-trained models to facilitate classification—is also worth exploring.

Reviewer comment (Experiments):
There are remaining challenges in the presented insights regarding reduced interpretability when using a different backbone: While reuse of pretrained backbones is legitimate (and should be encouraged), the results show that using another (non-DETR) pretrained backbone (e.g. the presented ViTs in Fig. 22) significantly hinders the efficacy of the presented INTR. This might be due to misalignment, but if the method should be useful, it must be applicable to a variety of backbones of interest, not just one specific combination optimized for DETR. Whether the method transfers well onto other architectures still remains somewhat open and not entirely satisfyingly resolved.

Author response:
**Pre-training influence.** 
Thanks for carefully checking our paper, especially the footnote on page 6 that discusses our usage of an MSCOCO-pre-trained model. 

During the rebuttal period, we tried to pre-train the INTR model from scratch, i.e., randomly initializing the Transformer encoder and decoder in DETR, using the ImageNet data. Unfortunately, we found the hyper-parameter setting of DETR not directly applicable to this pre-training. While we have tried to perform hyper-parameter selection, we were short of time to find a good configuration. We will try to continue the study. 

As an alternative, we take the pre-trained DETR model and replace its encoder (i.e., ResNet-50 + Transformer encoder) with a separately pre-trained DeiT-S (using ImageNet-1K) and ViT-H (using ImageNet-21K). Details of this study are in Appendix Section F. We found that using a DeiT-S or ViT-H encoder trained without MSCOCO can sometimes outperform the original ResNet-50 + Transformer encoder in terms of **accuracy**, as shown in Table 4. However, in terms of the  **interpretation/visualization**, using the pre-trained DETR model is still the best. Figure 22 shows an example. We attribute this to two reasons: 1) the pre-trained DeiT-S and ViT-H are not well-aligned with the pre-trained decoder, and 2) pre-training with detection data does help with focusing on details.

With that being said, we respectfully think that these results should not undermine our main contributions and claims. First, DETR is a model architecture designed initially for detection. Training it from scratch may need some careful investigation. Second, with the accessibility to large-scale detection datasets and pre-trained models, building a model from the most suitable starting point should be allowed and encouraged. In our case, the MSCOCO dataset does not contain fine-grained class labels and part annotations, quite different from our downstream tasks. Besides, what we found interesting is that while existing literature mostly uses classification pre-trained models to facilitate other downstream tasks, we show that the opposite—using detection pre-trained models to facilitate classification—is also worth exploring.

Reviewer comment (Writing):
It might be worth changing the way how HxW is described: The authors consistently describe ‘H’ and ‘W’ as the ‘number of grids’ – however, it is strongly suggested to instead refer to the elements as “grid cells”, since the entire image/matrix HxW can be seen as the grid, with H and W indexing the ‘grid cells/elements of the grid’.

Author response:
**HxW.** Thank you for the suggestion, and we will use grid cells in our final version.

Reviewer comment (Evaluation):
As this paper’s evaluation contains mostly qualitative results (e.g. showing the visualizations and explanations of INTR), it would be great if they can show the real utility of the explanations for a human-related task [a]. In this case, the decreases in accuracy shown in Table. 2 can be completely negligible. Otherwise, I still see ResNet clearly better than INTR (although this is not a major point).

Author response:
**Real utility of the explanations.**
We appreciate your feedback. To our knowledge, in many scientific or safety-critical domains, such as biodiversity and medical applications, domain experts often care more about whether the model is trustworthy or can discover new knowledge (e.g., traits for animals or biomarkers for diseases) rather than purely achieving higher accuracy. In our current paper, the targeted real utility is indeed automatic attribute/trait identification and discovery for organisms. This is of great impact because the analysis of traits, the integrated products of genes and environment, is critical for biologists to predict the effects of environmental change or genetic manipulation and to understand the significance of patterns in the evolutionary history of life. Specifically for fine-grained species, our approach can aid biologists in rapid identification and differentiation between species, refining and updating taxonomic classifications, and gaining insights into relationships between species. 

To demonstrate this capability, we conduct additional analyses focusing on closely related pairs of butterfly species within the Cambridge Butterfly dataset. Our objective was to pinpoint the distinctions discernible by our model and compare them with the differences identified by experts. Specifically, we selected three finer species (Heliconius melpomene and Heliconius elevatus; Heliconius erato and Heliconius melpomene; Heliconius melpomene and Heliconius erato) from the Cambridge Butterfly website and confirmed that our method effectively detects these differences as shown in Figure 18 and 19 (and Figure 6). This implies that our approach can assist experts in accurately discerning (and potentially discovering) the unique characteristics of fine-grained species. 

The traits identified by human experts can be found here.

Website: https://www.cliniquevetodax.com/Heliconius/index.html 

Figure 6: https://www.cliniquevetodax.com/Heliconius/pages/melpo%20vs%20elevatus.html

Figure 18: https://www.cliniquevetodax.com/Heliconius/pages/melpo%2520vs%2520erato.html

Figure 19: https://www.cliniquevetodax.com/Heliconius/pages/erato%20favorinus.html

Finally, as discussed in the first response to reviewer dzqa, improving the classification accuracy may hurt the model’s interpretability. We thus think it is fine to lose the classification accuracy a bit for trading better interpretability.

Reviewer comment (Evaluation):
The authors may also want to evaluate the explanations using proxy metrics in XAI.

Author response:
**Evaluate the explanations using proxy metrics in XAI.**
Thank you for the suggestion. In our original submission, we indeed included results using proxy metrics in XAI (cf. Sect. F and Table 6 in the Appendix). We apologize if we did not make it clear.

Specifically, we applied the *deletion* and *insertion* metrics. Given a post-hoc explanation method that generates a saliency map for the input image (e.g., Grad-CAM, cf. Sect. 2.2 and Sect. A), these metrics first sort the image patches based on the saliency scores (descending order). The deletion metric then iteratively removes patches from the image, inputs the manipulated image to the classifier, and measures the confidence of the correct class. The more sharply the confidence drops (i.e., the lower the AUC is), the better the explanation is, as it highlights the most discriminative patches, which, once removed, would drastically reduce the classifier’s confidence. In contrast, the insertion metric blurs the original image and iteratively discloses the patches. The more sharply the confidence increases (i.e., the higher the AUC is), the better the explanation is, as it highlights the most discriminative patches, which, once inserted, would drastically improve the classifier’s confidence. Please see more details in Sect. F, paragraph: Comparisons to post-hoc explanation methods.

The results are shown in Table 6 of the Appendix (during the rebuttal, we further include the results of RISE). INTR notably outperforms Grad-CAM and RISE, achieving a higher insertion AUC and a lower detection AUC. 

That said, XAI metrics are designed mainly for post-hoc explanation methods like Grad-CAM and RISE, which aim to explain a black-boxed classifier like ResNet. For self-interpretable methods like ProtoPNet, ProtoPFormer, ProtoTree, and our INTR, which build specific classifiers, XAI metrics are seldom reported as they may not be fair across different methods.

Reviewer comment (Experiments):
It has been unclear to me whether the bad performance of INTR stems from the capability of query vectors or not? It would be great if the authors have ablation studies for the query vectors (e.g. for classes with high inter-class variations, a simple vector could not be representative for the whole class). Also, how do they affect the explanations.

Author response:
**Performance of INTR.**
Thank you for the insightful comment. In analyzing INTR’s prediction, we did find that some errors result from intra-class variations. Specifically, for many bird species, different genders and life stages show different appearances and attributes (https://www.allaboutbirds.org/guide/Painted_Bunting). For example, in Fig. 1, the last row (misclassified) is indeed a female Painted Bunting; others, male Painted Bunting. These intra-class variations can be considered *sub-categories*, and we surmise that learning one query vector per class may only capture the major sub-categories in the dataset but not the entire variations. 

During the rebuttal period, we investigated the idea of learning *M* queries per class. This idea results in M logits per class (cf. Figure 2), and we averagely pool them to represent each class in both training (for the cross-entropy loss) and in testing. (We also investigated max pooling but found it hard to converge.) We see a notable gain on the CUB dataset: using $M=2$ queries per class improves INTR from **71.8** to **75.7**. We also include a qualitative result in Fig. 20. The image is an immature male Hooded Oriole, and INTR misclassifies it. Using $M=2$ queries, the model can correctly classify the image and highlight the black throat attribute. We will include a comprehensive evaluation of this multi-query version of INTR in the final version. 

In our humble opinion, the improvement by using multiple queries per class does not change our main paper's main contributions and messages. In contrast, it further showcases INTR’s extendability and its potential to discover sub-categories within a class.

Reviewer comment (Experiments):
The number of heads is equivalent to the number of concepts in ProtoPNet and its variants. For CUB, I believe the number of distinctive features is less than 8 (to my experience). Then I believe this number of heads should be adjusted according to the dataset.

Author response:
**The number of heads.**
Thank you for the comment. In our humble opinion, the number of heads in INTR can be different from the number of concepts per class in ProtoPNet. In ProtoPNet, each class is allowed to learn a separate set of prototypes; in INTR, the model parameters of the cross-attention heads are shared by all classes. About the number of discriminative features, according to the CUB dataset, there are 312 human-annotated attributes across more than 20 part-{pattern, shape, size} combinations. Thus, we respectfully think it is reasonable for INTR to use eight or more heads. 

That said, the number of heads is adjustable as it is essentially a hyper-parameter. In our original submission, we mainly followed the DETR paper to use eight heads. We did conduct an ablation study on the CUB dataset in Table 5 and found that eight heads slightly outperform other numbers. During the rebuttal, we further compare different numbers of heads for the Butterfly and Pet datasets and see similar trends. We will include a more comprehensive ablation study in the final version.

Reviewer comment (Evaluation):
Regarding the real utility of the explanations, although the authors stated that the targeted real utility is automatic attribute/trait identification and discovery for organisms and provided context in Sect. G about four categories of traits, it is unclear from Fig.1 how INTR extracts size and morphology information. Simply highlighting background pixels provides only loose support for INTR's ability to extract habitat-related traits. There is a need for a systematic evaluation of INTR's efficiency in trait discovery, as most of the claims about its utility are currently subjective and unconvincing.

Author response:
**Real utility of the explanations.**
We appreciate your feedback. To our knowledge, in many scientific or safety-critical domains, such as biodiversity and medical applications, domain experts often care more about whether the model is trustworthy or can discover new knowledge (e.g., traits for animals or biomarkers for diseases) rather than purely achieving higher accuracy. In our current paper, the targeted real utility is indeed automatic attribute/trait identification and discovery for organisms. This is of great impact because the analysis of traits, the integrated products of genes and environment, is critical for biologists to predict the effects of environmental change or genetic manipulation and to understand the significance of patterns in the evolutionary history of life. Specifically for fine-grained species, our approach can aid biologists in rapid identification and differentiation between species, refining and updating taxonomic classifications, and gaining insights into relationships between species. 

To demonstrate this capability, we conduct additional analyses focusing on closely related pairs of butterfly species within the Cambridge Butterfly dataset. Our objective was to pinpoint the distinctions discernible by our model and compare them with the differences identified by experts. Specifically, we selected three finer species (Heliconius melpomene and Heliconius elevatus; Heliconius erato and Heliconius melpomene; Heliconius melpomene and Heliconius erato) from the Cambridge Butterfly website and confirmed that our method effectively detects these differences as shown in Figure 18 and 19 (and Figure 6). This implies that our approach can assist experts in accurately discerning (and potentially discovering) the unique characteristics of fine-grained species. 

The traits identified by human experts can be found here.

Website: https://www.cliniquevetodax.com/Heliconius/index.html 

Figure 6: https://www.cliniquevetodax.com/Heliconius/pages/melpo%20vs%20elevatus.html

Figure 18: https://www.cliniquevetodax.com/Heliconius/pages/melpo%2520vs%2520erato.html

Figure 19: https://www.cliniquevetodax.com/Heliconius/pages/erato%20favorinus.html

Finally, as discussed in the first response to reviewer dzqa, improving the classification accuracy may hurt the model’s interpretability. We thus think it is fine to lose the classification accuracy a bit for trading better interpretability.

Reviewer comment (Evaluation):
Regarding evaluating the explanations using proxy metrics in XAI, although the authors mentioned that such metrics are seldom reported for self-interpretable classifiers like ProtoPNet, the current evaluation using deletion and insertion as proxy metrics appears to be of limited significance. A more meaningful benchmark would involve comparing the explanations with other concept-based, self-interpretable classifiers, rather than relying on post-hoc explainers.

Author response:
**Evaluate the explanations using proxy metrics in XAI.**
Thank you for the suggestion. In our original submission, we indeed included results using proxy metrics in XAI (cf. Sect. F and Table 6 in the Appendix). We apologize if we did not make it clear.

Specifically, we applied the *deletion* and *insertion* metrics. Given a post-hoc explanation method that generates a saliency map for the input image (e.g., Grad-CAM, cf. Sect. 2.2 and Sect. A), these metrics first sort the image patches based on the saliency scores (descending order). The deletion metric then iteratively removes patches from the image, inputs the manipulated image to the classifier, and measures the confidence of the correct class. The more sharply the confidence drops (i.e., the lower the AUC is), the better the explanation is, as it highlights the most discriminative patches, which, once removed, would drastically reduce the classifier’s confidence. In contrast, the insertion metric blurs the original image and iteratively discloses the patches. The more sharply the confidence increases (i.e., the higher the AUC is), the better the explanation is, as it highlights the most discriminative patches, which, once inserted, would drastically improve the classifier’s confidence. Please see more details in Sect. F, paragraph: Comparisons to post-hoc explanation methods.

The results are shown in Table 6 of the Appendix (during the rebuttal, we further include the results of RISE). INTR notably outperforms Grad-CAM and RISE, achieving a higher insertion AUC and a lower detection AUC. 

That said, XAI metrics are designed mainly for post-hoc explanation methods like Grad-CAM and RISE, which aim to explain a black-boxed classifier like ResNet. For self-interpretable methods like ProtoPNet, ProtoPFormer, ProtoTree, and our INTR, which build specific classifiers, XAI metrics are seldom reported as they may not be fair across different methods.

Reviewer comment (Experiments):
Regarding the performance of INTR, does increasing M to 2 equate to doubling the number of heads, in terms of the concepts to be learned? In Fig.20's bottom row, while increasing M can rectify misclassifications, the resultant explanations tend to be noisier. This should be revised for the next version.

Author response:
**Performance of INTR.**
Thank you for the insightful comment. In analyzing INTR’s prediction, we did find that some errors result from intra-class variations. Specifically, for many bird species, different genders and life stages show different appearances and attributes (https://www.allaboutbirds.org/guide/Painted_Bunting). For example, in Fig. 1, the last row (misclassified) is indeed a female Painted Bunting; others, male Painted Bunting. These intra-class variations can be considered *sub-categories*, and we surmise that learning one query vector per class may only capture the major sub-categories in the dataset but not the entire variations. 

During the rebuttal period, we investigated the idea of learning *M* queries per class. This idea results in M logits per class (cf. Figure 2), and we averagely pool them to represent each class in both training (for the cross-entropy loss) and in testing. (We also investigated max pooling but found it hard to converge.) We see a notable gain on the CUB dataset: using $M=2$ queries per class improves INTR from **71.8** to **75.7**. We also include a qualitative result in Fig. 20. The image is an immature male Hooded Oriole, and INTR misclassifies it. Using $M=2$ queries, the model can correctly classify the image and highlight the black throat attribute. We will include a comprehensive evaluation of this multi-query version of INTR in the final version. 

In our humble opinion, the improvement by using multiple queries per class does not change our main paper's main contributions and messages. In contrast, it further showcases INTR’s extendability and its potential to discover sub-categories within a class.