Paper: Learning to Intervene on Concept Bottlenecks (ICLR-2024)

Reviewer comment (Evaluation):
I am confused as to why the performance in Table 1 is lower for CB2Ms in the Full versus the Identified sets. Are examples in the Identified set those in which the memory module predicts that the example is misclassified? If so, why is the baseline CBM task performance so high (this would imply a high false positive rate?) It would be good for the authors to expand on possible False Positive and False Negative rates of the memory module, for each domain.

Author response:
- **Why is the Performance of CB2M in Tab 1 worse on full set compared to the identified instances?** With identified instances we consider all instances of the validation set to which the CB2M reapplies an intervention. Thus, the identified instances row shows that reapplying an intervention with CB2M does in general result in a good concept and class accuracy on these examples (after reapplying). Thus, the full set includes the samples of this identified set, but also all remaining (mistake) samples. This is the reason for the perceived lower accuracy on the full vs identified set. We have restructured Tab. 1 to make this more clear. Indeed, the concept accuracy of the base CBM on the identified instances seems comparably large. A few concept errors can, however, already result in a large drop in task performance, as observed in class performance of CBM on the identified set. Nevertheless, we have provided explicit FPR and FNR for the generalization task in the appendix (App. A.6). These suggest quite low FPR for CB2Ms, in contrast to a high FNR. However we assume this to be due to hyperparameter tuning for high precision and the fact that negative samples consist of all base model errors, including those that are in principle not correctable via generalizing interventions.

Reviewer comment (Experiments):
I am not convinced that the authors’ selection of baselines is adequate. As the authors detail in the Related Work, there have been several efforts to learn intervention policies (e.g., CooP) which select which next example to query people over. Yet, the authors never compare to any of these policies. None of these prior policies, to my understanding, leverage reuse of past interventions – as such, the authors could make the case that their method is complementary to these approaches; i.e., could be combined with methods which learn to intervene, which could justify not including such a baseline. However, the authors do not make such claims. I would be keen for the authors in the rebuttal to expand on how their work relates to other intervention policies and why they did not compare against them as baselines.

Author response:
- **Comparison to baselines (CooP und Shin et al.)**: 
In fact, the goal of our work is not to propose a method that stands in contrast to the mentioned works (CooP and Shin et al.), but rather CB2M represents a method that can be considered orthogonal to these. Specifically, Shin et al. propose several heuristics to estimate which concepts are the most important to intervene on for a given sample. Thus, they do not provide a method to estimate which samples should receive interventions, but instead “only” provide a concept ordering for given samples. Similarly, CooP can provide a subset of concepts to intervene on (even over multiple samples). However, CooP cannot be used directly to identify whether a sample in isolation should receive an intervention. Moreover, neither of these methods tackle the problem of the inefficiency of one-time interventions. In contrast, CB2M can be used to detect samples which potentially require interventions and can then even generalize given interventions to new samples. Thus, overall the mentioned approaches tackle different problem settings such that CB2Ms do not stand in opposition to these, but can in principle be used in conjunction with them (as e.g., Fig. 3 suggests).

Reviewer comment (Experiments):
The authors emphasize that their approach is model-agnostic. However, all experiments are with a CBM backbone. In the absence of augmenting other concept-based systems with their modules, I do not think the authors should claim their method is model-agnostic. If the authors would like to emphasize this claim, I believe experiments are needed with at least one other concept-based system. Otherwise, I think it is fine to leave for future work, but the text should be couched as such.

Author response:
- **CB2M is not model-agnostic**:
Indeed, our approach is not in general model-agnostic, but rather agnostic in the context of CBM-like architectures, as long as they have an accessible concept layer and encodings. We have clarified this in the paper and updated the wording where necessary

Reviewer comment (Evaluation):
I am confused and concerned by Table 9 in the Appendix. The standard deviation is massive for CUB in particular. The authors note that the wide variance could be due to the threshold selection. However, it’s not clear to me why the threshold selected across the 5 seeds would vary so much that it leads to this level of variance in the number of generalized interventions? What is the variance in the selected threshold (can you please provide example threshold values?) and/or further explanation of what is happening here?

Author response:
- **High standard deviation in Tab 9 in the appendix**:
This happens due to two facts: We optimized $t_d$ only once for all augmentations, which is not necessarily optimal given the individual nature of each augmentation. Additionally, the augmentations salt&pepper and speckles have a more disruptive effect on the model encodings, causing fewer samples to be recognized for interventions. We have added a more detailed discussion on this in App. A.6.

Reviewer comment (Evaluation):
It’s not clear to me that CB2M is better than softmax at detecting (with the exception of the Parity C-MNIST domain). The performance of CB2M and softmax are within error bounds for CUB. Can the authors expand on why this may be further?

Author response:
- **Detection of mistakes for softmax and CB2M is not different (except for C-MNIST)**:
Indeed, our results do suggest the baseline softmax approach to be en par with CB2M. However a major difference between softmax and CB2M is that the softmax approach looks at the model in total. Thus it can, in principle, not distinguish between bottleneck and predictor mistakes. In our investigated data set settings prediction errors are solely caused by bottleneck errors. In more natural and complex settings in which both types of errors occur (bottleneck and predictor error) using the softmax approach will be insufficient. Moreover, softmax and CB2M are not exclusive, as we show in App. A.7 where we combine both approaches into one.

Reviewer comment (Evaluation):
The authors’ assumption of perfect humans is sensible for this work. However, I would encourage the authors to think about how their module(s) may be challenged if human interventions are incorrect (or uncertain – e.g., Collins, Espinosa-Zarlenga et al, “Human Uncertainty in Concept Based Systems” AIES 2023). Such challenges would be worth expanding on in a Limitations section (of which the authors do not suitably have here).

Author response:
- **Assumption of perfect human interventions**:
Yes, indeed this is an assumption of our work. We have added a discussion on this in a new limitations section (at the end of the  evaluations section), linking the mentioned reference as well as to other relevant works.

Reviewer comment (Writing):
As a minor semantic note (which does not affect my score): the authors caption Fig 3 as “Less is more” — but really, this shows that the less is enough / sufficient to achieve high task performance; not that less is more than having further interventions. I’d encourage the authors to change that caption :)

Author response:
- **Less is more, Fig. 3**:
Thanks for pointing that out, we have modified the caption.

Reviewer comment (Evaluation):
The evaluation of the paper is a weakness. The authors set up a few situations on a few common datasets to show the utility of their method, but none of these experimental setups are particularly compelling and seem somewhat contrived. It is unclear whether the goal of the method is to increase model performance, improve HCI, or something else.

Author response:
**What is the concrete goal and potential use-cases of your method?** Overall, in this work, we propose to extend CBMs with an inspectable and interactable memory module. This, among over things allows to increase the effectiveness of user interactions on CBMs in
two major ways: First, a CB2M is able to propose a preselection of potential errors, which can then be presented to a human user. By effectively guiding the users attention to model mistakes, this preselection potentially reduces the amount of required human inspection
and interactions. Second, given the generalization abilities of the CB2M’s memory module, interventional feedback on few samples can effectively be transferred to novel and “unintervened” samples. Thus, overall the goal of CB2Ms is to more efficiently and effectively integrate interventional feedback from expert human users into concept bottleneck models. In fact, as the reviewer hints, the medical domain is a very valid use-case of CB2Ms. Particularly, such high-stake decision domains require extensive and detailed human expert inspection, cf. [1]. However, as high-quality feedback from medical doctors is generally expensive and sparse in nature, it is specifically necessary to make the most of those interactions that can be provided. This is particularly, where CB2Ms can shine. We have demonstrated these abilities on several benchmark data sets, among other things confounded data sets. Specifically, in our updated version we have added a second confounded data set (CUB (conf.)) to make these points more clear.

Reviewer comment (Evaluation):
From a performance perspective, on the CUB dataset, the state-of-the-art performance is around 92%+, but the method in the paper achieves only 88.7%. While raw performance may not be the main goal, using accuracy as a measure of usefulness is unconvincing if it does not surpass current performance. The method does not appear to offer a way to improve model accuracy beyond what is already possible, such as breaking the 92% ceiling with human feedback and interventions.

Author response:
**CB2M does not achieve SOTA results on CUB**: From a performance perspective, the objective of CB2M is not necessarily to achieve new SOTA results, but CB2Ms are rather specifically designed as a flexible extension to increase the effectiveness of human inter-
actions (see above) on any CBM-like base model ($g \circ f$). In this work we introduce them in the context of vanilla CBMs based on Koh et al. (2020)’s work. Nevertheless, they can in principle also be combined with more recent architectures which provide a higher baseline performance as the ones which the reviewer suggest. Moreover, the results presented in Tab. 1 in fact depict model performances based on the augmented CUB version, thus explaining the differences which the reviewer hints at this point. We have now marked this as (Aug.) in the paper to make this more clear.

Reviewer comment (Experiments):
The human interventions used in the experiments are simulated, which raises the question of whether humans could actually interact with the method in the way the authors propose.

Author response:
Thanks so much for taking the time to go through our changes and answers as well as appreciating the work with acceptance. We share the reviewers concerns related to XAI research and claims often made therein. Indeed, whether CBMs represent more understandable models is an important open topic in the community, yet orthogonal to our work. More importantly, we agree that the utility of interventions as introduced in the original CBM paper (Koh et al. (2020)) could benefit from more user evaluations. While this is a pressing issue, it is beyond the scope of our work. Specifically, given the assumption that interventions in general are useful for users, CB2Ms improve the efficiency of these interventions once they are provided. Again we very much appreciate the fruitful discussion and are happy if you reconsider your score.

Reviewer comment (Experiments):
The paper does not demonstrate the method applied in realistic, high-impact scenarios, such as a doctor using it to improve team performance or accuracy beyond a standard black-box model. This leaves uncertainty about the method’s potential applications.

Author response:
**What is the concrete goal and potential use-cases of your method?** Overall, in this work, we propose to extend CBMs with an inspectable and interactable memory module. This, among over things allows to increase the effectiveness of user interactions on CBMs in
two major ways: First, a CB2M is able to propose a preselection of potential errors, which can then be presented to a human user. By effectively guiding the users attention to model mistakes, this preselection potentially reduces the amount of required human inspection
and interactions. Second, given the generalization abilities of the CB2M’s memory module, interventional feedback on few samples can effectively be transferred to novel and “unintervened” samples. Thus, overall the goal of CB2Ms is to more efficiently and effectively integrate interventional feedback from expert human users into concept bottleneck models. In fact, as the reviewer hints, the medical domain is a very valid use-case of CB2Ms. Particularly, such high-stake decision domains require extensive and detailed human expert inspection, cf. [1]. However, as high-quality feedback from medical doctors is generally expensive and sparse in nature, it is specifically necessary to make the most of those interactions that can be provided. This is particularly, where CB2Ms can shine. We have demonstrated these abilities on several benchmark data sets, among other things confounded data sets. Specifically, in our updated version we have added a second confounded data set (CUB (conf.)) to make these points more clear.

Reviewer comment (Writing):
It is unclear where \( x_e \) is taken from in the method description.

Author response:
**It is not clear where $x_e$ is taken**:
In fact, the encoding $x_e$ for a sample $x$ is the input of the last layer of the bottleneck network $g$ (Sec. 2.2, first paragraph).

Reviewer comment (Writing):
In Equation 2, using “val” as a variable is potentially confusing, as it is commonly associated with “validation” in machine learning.

Author response:
**Confusion concerning val notation**: Indeed, we have updated this to valid(·).

Reviewer comment (Novelty):
The reviewer questions what a real-world application of this method would be that could make the ML community genuinely excited—specifically, a case where the method is both understandable and useful to intended practitioners.

Author response:
**What is the concrete goal and potential use-cases of your method?** Overall, in this work, we propose to extend CBMs with an inspectable and interactable memory module. This, among over things allows to increase the effectiveness of user interactions on CBMs in
two major ways: First, a CB2M is able to propose a preselection of potential errors, which can then be presented to a human user. By effectively guiding the users attention to model mistakes, this preselection potentially reduces the amount of required human inspection
and interactions. Second, given the generalization abilities of the CB2M’s memory module, interventional feedback on few samples can effectively be transferred to novel and “unintervened” samples. Thus, overall the goal of CB2Ms is to more efficiently and effectively integrate interventional feedback from expert human users into concept bottleneck models. In fact, as the reviewer hints, the medical domain is a very valid use-case of CB2Ms. Particularly, such high-stake decision domains require extensive and detailed human expert inspection, cf. [1]. However, as high-quality feedback from medical doctors is generally expensive and sparse in nature, it is specifically necessary to make the most of those interactions that can be provided. This is particularly, where CB2Ms can shine. We have demonstrated these abilities on several benchmark data sets, among other things confounded data sets. Specifically, in our updated version we have added a second confounded data set (CUB (conf.)) to make these points more clear.

Reviewer comment (Experiments):
[Critical] My biggest concern with this paper is its evaluation. In my opinion, it is not fair to evaluate a method like CB2M, which can take and store extra samples at test time to improve performance on future unseen samples, against methods that completely ignore the same feedback even if in theory it could be used to improve their performance as well. For example, given that the validation set is used to construct the initial mistake memory, at the very least I would expect to see as a fair baseline a CBM that was able to update its weights using feedback from the same validation set during training (as otherwise CB2M is unfairly being trained on more data than the CBM baseline!). Similarly, when considering how the intervention memory is used, it would be fair to have as a baseline a CBM whose weights are updated at test time so that feedback from a new intervention is considered in the model. This can be done, say, via a variety of online learning algorithms that aim to minimize the cross entropy loss of the corrected concept prediction given the provided ground-truth (intervened) label (or even a baseline that runs a few gradient decent steps on this new intervention's label and corresponding sample). Having such a baseline and showing the CB2M still beats that baseline would provide very strong evidence that the memory mechanism introduced in this paper is worthwhile and novel compared to methods that already exist there to address similar problems.

Author response:
- **Question 1 & 2: On the fairness of evaluation between CBM and CB2M**: Indeed, CB2M has access to more information than CBM in the evaluation. We provide results for CBM finetuned on the respective validation sets (or the test set in case of CUB (Aug.)) in our
updated version (App. A.6) where we differentiate between different amounts of finetuning steps (i.e. finetuning epochs). We observe that in all cases, CB2M performs better than doing few finetuning steps, however, for CUB and Parity CMNIST, finetuning for an extended amount of time outperforms CB2M. On the other side, finetuning on Parity-MNIST (unbalanced) does not provide any real benefit, even for an extended number of steps.

  Though we agree to a certain degree with the reviewer on comparing the models with the same amount of data, these comparisons are also unfair towards CB2Ms in the context that CB2M does not perform parameter optimization based on the validation set, whereas the CBM’s parameters in Tab. 10 were specifically finetuned on this. Additionally, a particular benefit of CB2M (in comparison to CBM  inetuning) is the potential use-case of dynamic, online learning settings during model deployment. In this case, finetuning on each novel set of data is infeasible and might even lead to issues such as catastrophic forgetting. 
  
  Moreover, this form of finetuning the CBM’s parameters on the interventional data has even more drawbacks in comparison to CB2M in the context of interpretablity and interactability. Specifically, it is very difficult to remove already applied interventions from the finetuned model, if it turns out these interventions were bad/incorrect. On the other hand it is difficult to inspect the representation of the finetuned model, where in CB2M a user can simply inspect the model’s memory.

Reviewer comment (Experiments):
[Critical] Regarding the evaluation of the method on unseen data points (e.g., Table 1), it is unclear whether it is fair for ECB2Ms to be able to use the entire (unmodified) test set as part of their memory loading while none of the baselines can have the same benefit at training time. Evaluating CB2M and other baselines on a modified version of this test set seems like a very unfair comparison if none of the other methods were able to obtain any feedback from simulated interventions on the unmodified version of these samples. It may be fairer to avoid any sort of leakage from the test set into the training set of CB2M by avoiding loading into its initial memory samples that are highly related to the ones it will be evaluated on (this can be done by splitting the test set into two). I understand this is how this method may be used in practice (with the memory taking advantage of test-time samples to build a database of interventions and mistakes); comparing it against a CBM's results on the same table makes it seem like this is an apples-with-apples comparison when in reality it may not be such. Making this distinction clearer, and if possible the evaluation fairer, would help a lot to indicate how this method works and how it is expected to be used.

Author response:
- **Train/Test leakage:** On CUB, we specifically want to evaluate how well CB2M can generalize interventions when similar versions of intervened data points reoccur (one of the motivations for CB2M). Therefore, this "leakage" between the unmodified and augmented versions of the test set is deliberate. We agree that the comparison to the base CBM (without) finetuning is unfair, similar to the other datasets.
 As we have added the results for the finetuned CBM (which also has access to the unmodified test set), we believe that this comparison is then again comparable to the other two datasets.

Reviewer comment (Evaluation):
[Critical] With the exception of C-MNIST, the results shown in the identification of mistaken samples do not appear to be statistically significant (see the standard deviations). Because of this, it is unclear whether the proposed method identifies errors better than, say, the naive softmax baseline. Similarly, the effect on generalized interventions based on CB2M's detection on the full dataset (Table 3) does not appear to be statistically significantly better than what one observed when using the Softmax detection for all datasets (see standard deviations). This casts some doubt on the usability of this method.

Author response:
- **Stastical significance of Tab. 2 and Tab. 3:** Sorry, we simply missed to answer this point in our initial response (also see discussion with reviewer XnGw). 
 Indeed, our results do suggest the baseline softmax approach to be on par with CB2M. However a major difference between softmax and CB2M is that the softmax approach looks at the model in total. Thus it can, in principle, not distinguish between bottleneck and predictor mistakes. In our investigated data set settings prediction errors are solely caused by bottleneck errors. In more natural and complex settings in which both types of errors occur (bottleneck and predictor error) using the softmax approach will be insufficient. Moreover, softmax and CB2M are not exclusive, as we show in App. A.7 where we combine both approaches into one.
 Consequently, we further adapted the discussion of Tab. 3 to place the results in better context.