From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper proposes a new model intepretability technique. The main idea is creating a final sparse layer composed of interpretable features using a loss function formulation that promotes diversity and sparsity. The technique is tested on different architectures and datasets, with a particular focus on fine grained classification.

The reviewers remarked the good accuracies obtained with the sparse features. The reviewers also pointed out some aspects on the paper presentation that needed improvement (e.g. order of the sections was confusing; the descriptions of the diversity loss function and the Glm-saga need more details), which were addressed by the authors in their feedback and in the reviewed version of the paper. However, there are concerns about this paper that still hold after the authors' feedback: (1) limited novelty with respect Glm-saga; (2) limited interpretability capactiy of the proposed approach; (3) need for a deeper understanding on why the proposed method is better with respect to previous approaches. Overall, the reviewers think the work has an interesting direction but it's not mature enough for being published yet. We recommend the authors to follow the recommendations to extend the analysis of the results.

As a minor comment, the current version of the paper is more than 9 pages in length, which exceeds the page limit for ICLR papers (this has not been taken into account for the final decision).
justification_for_why_not_higher_score: The direction of the paper is interesting. However, all the concerns raised by the reviewers suggest that the paper is not mature enough to be published.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper1851/Authors
title: Response to Reviewer 3(iicb)
comment: We thank Reviewer 3(iicb) for the valuable feedback. 

Regarding the weaknesses: 

> The paper is not easy to follow with missing details and an unorganized structure.
...
>  $ L_{final} $ is defined as diversity loss plus cross-entropy, while in Section 3.3 $$ L_{target}$  is cross-entropy alone.
* Glm-saga is moved to related work, to reduce the confusion with the loss-terms and improve the structure.

> For example,$\lambda_1, \lambda_2$  is not explained in Section 3.2.2
* Thanks for pointing that out. We added their definition. 

> a). Feature diversity loss: Such loss is widely explored in the existing literature like MCL, FRL, CCMP. It is unclear why the proposed  is better.
We discuss the mentioned competitors and compare our loss to MCL and FRL in Tables 2 and 6. Our method outperforms them in all metrics and does not require an additional hyperparameter in its formulation. 

* For MCL specifically, we found that only using one fully-connected layer on the features, as opposed to their multilayer-setup, harms the performance substantially, as mentioned in Section COMPARISON WITH OTHER LOSS FUNCTIONS. Additionally, we saw that the sparse solution does usually not primarily contain the features that were assigned to it in the loss formulation (In initial experiments only 16% of the assigned features in the sparse layer were assigned to it in the dense training). 

*  We added theoretical benefits of our loss over MCL in the method section. 

   * “s.t. different to MCL absent features do not have to be localized in small background patches.” 

> b). Feature sparsification loss: It is a direct follow-up of glm-saga. And the sparsification in the final layer leading to better interpretability was also investigated.
* We thank you for agreeing on their conclusion of increased interpretability via sparsification. 
  * As mentioned in the general comment, while building on glm-saga we improved accuracy and interpretability and clarified this. 

> c). Measure of interpretability: The measurement in this paper is coarse by counting the dimension of the feature. A low dimensionality does not necessarily lead to better interpretability. For example, one can inject an identity matrix after the final FC layer. The identity matrix is fully sparsed, but meaningless. Therefore, a better measurement would be the one-to-one correspondence between a concept and a single feature vector. Such measurement is already well-explored in Net-dissection (Bau et al., 2017).
* We argue that sparseness and low dimensionality together lead to interpretability as the identitiy matrix would be sparse, but would only work for number of classes = number of features, but by restricting number of features to be smaller than number of classes that is not possible. Additionally, the feature diversity loss encourages features to be localized differently, which reduces the chance of features already detecting an entire class. Indeed, one-to-one correspondence would be a convincing measurement. This idea was followed with a different formula to Bau et al. in the paper with the attributes in CUB, as they are not densely annotated. However, this would require all kinds of annotated attributes, which is not feasible. Therefore, measuring just one subset of concepts biases against models that learn other unannotated concepts. In Future work we might have humans try and align features with a certain workflow to measure “alignability”, similar to glm-saga. 

  * As mentioned in the general comment, we added visualizations in Figures 9 to 17 to give examples for increased interpretability.

From: ICLR.cc/2023/Conference/Paper1851/Authors
title: Response to Reviewer 2(Vk6D)
comment: We thank Reviewer 2(Vk6D) for the review, encouragement and interesting papers and first respond to the summary.

 

> While I would empathize if the authors feel that they may not be necessary because the model is "self-explaining", I would push back with [3], which argues to the contrary.” 

* We completely agree that the model is not fully self-explaining. But we argue that the low number of features and sparse structure allow to more easily analyze the features post-hoc leading to a more interpretable model.  Additionally, just the sparsity, as shown by the glm-saga paper, makes the model more interpretable.  

  * We adressed this in the paper and specify our claim towards a “more interpretable model” 

We have shown that some features can be interpreted automatically, using annotations and add an exemplary way of interpreting a feature without annotations.  For  the raised weaknesses you see with our method, we want to mention a few counter arguments: 

> My key concern is the analysis of the learned features. First, the global analysis seems to require dense attribute-level annotations, such as what's found in the CUB dataset. However, these attributes are expensive to collect and not often provided.
* As mentioned in the general comment, dense annotations are just one way to align the features. Generally, the low number of features allows for a thorough analysis of the learned features. 

> A hidden assumption behind the above analysis is that the model does use these semantic attributes for decision making. This seems to be a strong assumption, as various works [e.g. 1, 2] found that neural networks can use various spurious correlations and shortcuts, and furthermore many popular explanation methods are not well-equipped to detect such usage. I would appreciate any discussion on this aspect.
* It is correct that we assume that our model uses the semantic attributes for decision making for this attribute-based analysis, which seems supported by the high correlation of some of the features with the attributes and their localization. By manually aligning the features one can recognize all kinds of concepts with no assumptions, also spurious correlations or shortcuts. Our method is generally more suited than a model with a higher dimensional feature space, as it is more feasible to actually analyze the entire final decision. Glm-saga showed examples of detected spurious correlations via humans and a lower portion of the concepts being spurious correlations. 


  * We now discuss detecting spurious correlations in the paper. 

  * The added Figures 9 to 17 also indicate the connection to semantic attributes for decision making. 

> The manual feature alignment on the four-engine attribute seems a bit too ad hoc. There could be many different explanations, such as the color or style of the plane, that result in the heatmap focus. Such statements should be made more quantitative, and even better, falsifiable (e.g. does removing two engines affect the heatmap?).
* You are correct, that this image does not show the process that led to the alignment of the attribute and feature.  

  * We added an explanation for the made statement in Appendix E. It is however hard to quantify this alignment without any additional data. Figure 7 shows that the absence of the 4-engines leads to a reduced feature activation.  Additionally, the current feature visualization, except for the new Figures 9 to 17, is based upon removing patches in the input, see Appendix D.  

> The description on the diversity loss function moves quite fast, especially for readers not familiar with MCL and FRL. The same goes with the feature selection, which assumes prior knowledge of glm-saga.
* We motivated and described the diversity loss more clearly and introduced glm-saga before the feature selection.

From: ICLR.cc/2023/Conference/Paper1851/Authors
title: Response to Reviewer 1(9CdM)
comment: We thank Reviewer 1 (9CdM) for the review and questioning our statements.  

We argue that the model is more interpretable, since one can analyze every single feature thoroughly, as clarified in the general comment. Many thanks for the proposed optional points. However, they do not fully match to our contributions, as we will briefly discuss below. 
> Possibility 1: Since there are few features, they can be visualized easier: I'm not satisfied because now we are talking about the efficacy of visualization techniques, many of which can map complex decisions to simple-to-understand visualizations (Grad-CAM, etc..)
* Other work like glm-saga and MCL has also visualized the localization of specific features.  
* Grad-Cam only allows to localize what aspect of the image was important for this class, but not what was recognized in this region. For CUB, the model generally looks at the bird for a dense model, but it is not clear what it sees there. Additionally, saliency maps show various weaknesses, see [Sanity Checks for Saliency Maps](https://arxiv.org/abs/1810.03292). 
We dissect the decision process into first detecting interpretable features and then combining them in an interpretable sparse way, s.t. it is easier to follow the decision. Compare the localization of meaningful features in Figures 1 and 9 to 17 to a general saliceny map. 
   * We added visualizations showing the  difference between conventional dense models and our SLDD-model in Figures 9 to 17 

> Possibility 2: Fewer features can be easily aligned with the human-nameable concept: I do not understand why the analysis in Fig 3b and Fig 4 can also not be done for traditional networks. In the end, we would only be left with features that correlated with the concepts anyways; Why does it matter whether we run them on 5 or 50 features to start with? If more than one feature correlates with a single concept, they can then simply be visualized as a "group".
* See general comment: While generally feasible, the features might correspond to classes instead of abstract human concepts already. We added the data into the paper:  
  * Finally, the total number of features that is used by the unrestricted $(n^∗_f = n_f)$ models 
in Table 3 is reduced from 912 to 719 for the models with Ldiv. This supports our motivation that 
different features capture different concepts and therefore need to be shared. 
6 

> Possibility 3: Fewer features can show us what was really important: There are a lot of ways to rank the "importance" of features in neural networks. Why isn't ranking the features by feature importance enough? Why do we need to train a network to only make decisions using these features?
* Having an unrestricted number of features leads to more class-specific features, just as above.  
* Feature importance is not easy to measure. Glm-saga has shown in their paper that ranking the features by weight magnitude and only using those with the highest, harms the performance a lot. They additionally showed, that humans could not emulate the behavior of a dense model after inspecting some features and weights, whereas this was possible for the sparse mode.

From: ICLR.cc/2023/Conference/Paper1851/Authors
title: General Response
comment: We thank the reviewers for their thorough review, encouraging feedback and valuable criticism. In the following, we address shared concerns in the next paragraph and discuss individual remarks in separate subsections. 

General remarks: 
* We did apparently not present the novelty of our approach clear enough, as noted by both R1, R2 and R3. Please let us clarify.  Glm-saga was proposed for more debuggable but less accurate neural networks by computing a sparse linear layer to the features of a deep neural network. In their work, they used ~20 features per class and show increased interpretability for humans. As correctly pointed out, we made use of it. However, we increase sparsity, interpretability and accuracy. This has been achieved by our proposed feature diversity loss, feature selection and the follow-up finetuning step. The interpretability is discussed below. The increased accuracy and the sparsity properties have been shown in table 2.  
    * We extended the chapter on glm-saga to clarify our contributions and improved the presentation of these claims in the paper. 
* R1-R3 raise concerns regarding the interpretability of the proposed model.  
    * Please note, that the possibilities for increased interpretability and easier feature alignment have already been mentioned in the glm-saga paper. However, since the method optimizes for a huge number of features, this is not feasible in practice. Thus, manual inspection to understand the meaning and relevance of features is getting very labor-intensive and spurious correlations can happen. For this reason, we present an approach to heavily reduce the number of features. This enables a manageable alignment using techniques from feature visualization or localization. Thus, a lower number of features makes inspection more feasible (as pointed out in section “Alignment without Additional Data” now “Manual Alignment”). 
    * The demonstrated alignment with the semantic attributes on the CUB dataset is just an example that our learned sparse features align with these semantic attributes, also humans use for decision making. This is in general also possible for non-reduced feature vectors, as R1 pointed out. Additionally, not reducing the number of features leads to class-specific features, which would then naturally correlate with class-attributes, while in fact recognizing an entire class.  
    * As consequence, we will clarify this part and modified the paper as follows 
         * We add the manual alignment of the four-engine aircraft in the appendix to give an example of how one can approach the feature alignment without additional attribute information. 
         * We changed from “interpretable model” to the term “more interpretable model” as proper feature alignment is required to be fully interpretable, as mentioned in Section 4.3. 
         * We also put more emphasis on the optionality of attributes in the paper by renaming the chapter “Alignment without Additional Data” to “Manual Alignment” and clarifying it there. Additionally, we discuss the potential recognition of spurious correlations. 
         * We added example images in Figures 9 to 17 showcasing the increased interpretability through differently localizing features 
* Both R2 and R3 noted that the order of the sections was slightly confusing, as an idea of glm-saga is required to understand the feature selection. Additionally, we named the loss inaccurately. 
    * To address this issue, we reordered the structure of the paper and placed the description of glm-saga with additional details in the related work.

From: ICLR.cc/2023/Conference/Paper1851/Reviewer_9CdM
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes to train a sparse neural network that only utilizes a small portion of all possible input features per class. The paper argues that due to a small number of input features, they are more interpretable. Another main highlight is that despite using only 10% of the features, the accuracy of the tested network remains relatively stable (97-100%). The paper shows that the small number of features that the model uses to make decisions can also be easily aligned with human-nameable concepts for additional explainability. 
strength_and_weaknesses: **Strengths:**

*[S1]:* Surprisingly effective results: The main highlight of the paper for me is the fact that despite using very few features, the proposed method can achieve such good accuracy results. 

*[S2]:* Clearly presented: The paper is well-written and easy to follow. The experiments are described clearly and the main techniques used are adequately clear.


**Weaknesses:**

*[W1]:* Unclear why the proposed method is better for anything: The paper does a poor job of motivating exactly why a model using few features is more interpretable. I attempt to understand it from several angles, but do not find a satisfactory answer. 

- Possibility 1: Since there are few features, they can be visualized easier: I'm not satisfied because now we are talking about the efficacy of visualization techniques, many of which can map complex decisions to simple-to-understand visualizations (Grad-CAM, etc..)
- Possibility 2: Fewer features can be easily aligned with the human-nameable concept: I do not understand why the analysis in Fig 3b and Fig 4 can also not be done for traditional networks. In the end, we would only be left with features that correlated with the concepts anyways; Why does it matter whether we run them on 5 or 50 features to start with? If more than one feature correlates with a single concept, they can then simply be visualized as a "group". 
- Possibility 3: Fewer features can show us what was really important: There are a lot of ways to rank the "importance" of features in neural networks. Why isn't ranking the features by feature importance enough? Why do we need to train a network to only make decisions using these features?

For all of those, there is a direct actionable item for the authors to show; i.e., show us the results of the alternatives and how the proposed method fares better. 

*[W2]:* Limited Novelty: The proposed work is a fairly straightforward adoption of glm-saga; While this would absolutely not be a problem on its own, the fact that there is also not sufficient analysis (W1 above) makes it harder to overlook the fact that the paper does not present "previously unknown" technique that others might use differently. 
clarity,_quality,_novelty_and_reproducibility: **Clarity:** To me, it is quite clear, See S2 above.

**Quality:** See strengths and weaknesses above.

**Novelty:** There is a limited novelty in both the techniques used and the empirical evidence (and discussion).

**Reproducibility:** The method is built on existing models and a code is also promised, I think reproducibility is excellent. 
summary_of_the_review: While it was interesting to read, I found myself having critical questions about the main motivations of the paper by the end. The paper hinges on two main things: A network that is "inherently interpretable" is better than interpretability tools built on top of traditional networks. And the second is that fewer features lead to better interpretability. As mentioned above in W1, I am not sure about either of those claims and the paper does not show enough proof that it is true. So, despite interesting results (only using a few features gives good accuracy), this is a sub-par paper for me looking at it primarily from an interpretability angle. The paper isn't really written from a feature-compressive perspective, So, I am also unable to comment on that. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1851/Reviewer_Vk6D
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposed model that makes predictions from a sparse set of features. To do so, it employs a novel diversity-based loss and a feature selection process. The model is shown to achieve competitive performance from only very few features (e.g. 5). Empirical analysis on the learned features is able to map them to human-understandable semantic concepts. 
strength_and_weaknesses: Strengths: 

1. The paper differs from most vision interpretability papers in proposing a self-explaining model rather than focusing on post hoc explanations. 
2. This paper proposed a very sparse model that achieves very good performance, using a novel loss formulation. 

Weaknesses: 

1. My key concern is the analysis of the learned features. First, the global analysis seems to require dense attribute-level annotations, such as what's found in the CUB dataset. However, these attributes are expensive to collect and not often provided.
2. A hidden assumption behind the above analysis is that the model does use these semantic attributes for decision making. This seems to be a strong assumption, as various works [e.g. 1, 2] found that neural networks can use various spurious correlations and shortcuts, and furthermore many popular explanation methods are not well-equipped to detect such usage. I would appreciate any discussion on this aspect. 
3. The manual feature alignment on the four-engine attribute seems a bit too ad hoc. There could be many different explanations, such as the color or style of the plane, that result in the heatmap focus. Such statements should be made more quantitative, and even better, falsifiable (e.g. does removing two engines affect the heatmap?). 
4. The description on the diversity loss function moves quite fast, especially for readers not familiar with MCL and FRL. The same goes with the feature selection, which assumes prior knowledge of glm-saga. 

[1] https://arxiv.org/abs/2004.07780

[2] https://arxiv.org/abs/2104.14403
clarity,_quality,_novelty_and_reproducibility: See above. 
summary_of_the_review: Overall, I think this paper is on track for making a good contribution, but the interpretability analyses feel a bit thin. While I would empathize if the authors feel that they may not be necessary because the model is "self-explaining", I would push back with [3], which argues to the contrary. 

[3] https://aclanthology.org/2020.acl-main.386/
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1851/Reviewer_iicb
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper provides a training strategy by applying sparsification to obtain a more interpretable network. The training loss combines the feature diversity loss and sparsification loss used in Glm-saga. The measure of feature diversity is proposed and the paper shows that classes with the diverse feature can improve the accuracy. 
strength_and_weaknesses: 
Strength:

Regularizing the sparse feature for a more interpretable network is a doable method. 

Weakness:

1. The paper is not easy to follow with missing details and an unorganized structure. For example, $\lambda_1, \lambda_2$ is not explained in Section 3.2.2. In Section 3.2.1, $L_{final}$  is defined as diversity loss plus cross-entropy, while in Section 3.3 $\mathcal{L}_{target}$ is cross-entropy alone. 

2. The main contribution of the paper is not clear. For example, the claimed contribution includes:

a). Feature diversity loss: Such loss is widely explored in the existing literature like MCL, FRL, CCMP. It is unclear why the proposed $\mathcal{L}_{div}$ is better. 

b). Feature sparsification loss: It is a direct follow-up of glm-saga. And the sparsification in the final layer leading to better interpretability was also investigated. 

c). Measure of interpretability: The measurement in this paper is coarse by counting the dimension of the feature. A low dimensionality does not necessarily lead to better interpretability. For example, one can inject an identity matrix after the final FC layer. The identity matrix is fully sparsed, but meaningless. Therefore, a better measurement would be the one-to-one correspondence between a concept and a single feature vector. Such measurement is already well-explored in Net-dissection (Bau et al., 2017).
clarity,_quality,_novelty_and_reproducibility: See Strength And Weaknesses. 
summary_of_the_review: This paper targets an interesting problem. However, the contribution of the paper is not clear. Before having a better understanding of the challenges that this paper resolves, I tend to vote for a reject recommendation. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold