Paper: Explanation Shift: How Did the Distribution Shift Impact the Model? (ICLR-2024)

Reviewer comment (Evaluation):
I am not convinced that the results demonstrate the empirical superiority of the proposed method relative to the baselines. The authors only compare to the baselines in Figure 2. Here, there are several other methods that are competitive with explanation shift. In addition, the authors do not show confidence intervals in this figure. I also contest that "good indicators should follow a progressive steady positive slope", as if the goal is distribution shift detection, the only thing that should matter is the outcome of the hypothesis test.

Author response:
Answer A.1: The reviewer may note that we also compare baselines B4, B5, and B6 in Figure 3. Wrt our evaluation, we fully agree with the reviewer that the approach's effectiveness is key. This raises the question of how best to measure effectiveness. We have decided to construct experiments in Figures 2 and 3 in a way such that we can control the level of distribution shift by continuously increasing $\rho$, the co-variance (Fig 2), and the level of OOD data (Fig 3). A good metric for distribution shift should be able to mirror this control parameter. By the design of our experiment, step changes or fluctuations point towards less reliable metrics for recognising distribution shifts. We will make this experimental design clearer in the text. Also, see the extra evaluation above (2. quantifiable metrics).

Reviewer comment (Experiments):
All of the datasets considered are tabular datasets. How would the authors adapt their method to images (where X could be pixels) and text (where X is a sequence of tokens and f is an LLM)? It seems like Shapley values would be less meaningful and harder to compute in these settings.

Author response:
Our method is limited to tabular data. While this constitutes a limitation to our approach, we posit that understanding the distribution shift of tabular data is of problem hugely relevant for both research and practice.

Reviewer comment (Evaluation):
In the real-world datasets, it is unclear what the ground truth should be, and so it is hard to say whether the proposed method is behaving as intended. For example, how do we know in Section 5.2.2 that the distribution in CA18 is actually different than CA14, in a way that affects model performance?

Author response:
A.3 As described in section 2.2 (Impossibility of model monitoring), (Garg 2021)’s theorem shows that it is not possible to relate distribution shift to a change in model performance, particularly for tabular data where concept shift cannot be detected by properties of the visual domain. Therefore, our approach focuses on the change in how a given model works on data.

Reviewer comment (Theory):
One important aspect of distribution shift detection is to isolate the shift to particular distributions (i.e. particular shifts in Definitions 2.1-2.5). This is underexplored in most of paper. The authors do explore this by examining the feature importances in Section 5.2.2, but this seems quite ad-hoc and should be characterized further. For example, how would this behave theoretically in the covariate shift case in Example 4.1?

Author response:
A1.4 We fully agree with the reviewer, in the last paragraph of section 2.1 we have stated: "_In practice, multiple types of shifts co-occur together and their disentangling may constitute a significant challenge that we do not address here._"". We have only analyzed particular shifts on synthetic data where we can fully characterize the type of shift.

Reviewer comment (Theory):
The authors should consider examining the scenario where the number of samples on the target domain is limited, both empirically and theoretically. How does the power of the two-sample test scale with the number of target domain samples?

Author response:
A1.5: It is correct that our recognition procedure is a binary classifier and the reliability of the classifier depends on the number of samples on which it is trained. Previous work has studied the limitations of Classifier Two Sample Tests in the particle physics domain, in which they suggest using bootstrap permutation tests to get a better handle on the test's accuracy.

We've taken this methodology. In our own experiments (check out Section 5.2.2), we have included bootstrap permutation testing between the in-distribution (CA14) bootstraps and other states bootstraps.
Furthermore, in our experiment in section 5.2.1 where an increasing fraction of data is from OOD, we that from a small fraction onwards our metrics picks up the changes in the model due to distribution shift. 

[3] Model-independent detection of new physics signals using interpretable SemiSupervised classifier tests, Chakravarti, Purvasha and Kuusela, Mikael and Lei, Jing and Wasserman, Larry, The Annals of Applied Statistics, 2023

Reviewer comment (Theory):
The authors provide some interesting case studies and examples of the utility of Shapley based explanations in identifying distribution shifts. One aspect of this discussion that could be done better is trying to emphasize what additional assumptions could be required so that explanations such as Shapley can indeed be used for detecting, say, concept shift. This is not an impossible task for other methods, see for example: Liu et al [1]. See also experiments in this paper.

Author response:
A.1: Paper [1] can deal with concept shift because the authors assume that labels $D^{new}_Y$ are given. In our problem scenario, we make the realistic assumption that $D^{new}_Y$ are unknown. Thus, we address a different problem. Also cf. our Main Point 1 at the beginning of our rebuttal.

We will contrast our work against [1] in our discussion. This will further clarify and strengthen our work.

Reviewer comment (Experiments):
The overall empirical evaluation, in my view is weak. May be at least discuss how general the approach will be (can it be used on other data-modalities, beyond tabular data?)

Author response:
A.2: ​​We have systematically varied models, model parametrizations, and input data distributions. As indicated in Main Point 2 (at the beginning of the rebuttal), we have designed carefully controlled experiments (which are novel by themselves). Additionally, the appendix contains supplementary experimental results, providing details on synthetic data experiments, extending to further natural datasets, showcasing a broader range of modelling choices, and different explanation mechanisms. Thus, we believe that one of the paper's strengths is an extensive empirical evaluation.

As the reviewer correctly notes, the Classifier Two Sample Test on the explanations distributions is limited to tabular data and is not able to detect concept shift (we have stated this in the discussion). For non-tabular data, it needs further modifications that are out-of-scope for the paper.   Characterizing distribution shift on tabular data constitutes an economically hugely important problem that still lacks appropriate treatment.

Reviewer comment (Writing):
What is $\alpha_i$ in Example 4.3? I am not sure this example is valid based on what I think $\alpha_i$ will be in this example. Can authors clarify?

Author response:
A.3: The Shapley values are a product of a weighting (here: alpha_i for feature i) multiplied by the actual value of the feature (here: x_i). Thus,Shapley values explain a model f by a linear model with coefficients $\alpha_i$. We will further clarify that the Shapley values can be directly calculated for a linear model and IID data by this formula (Aas, 2021)

The specific example shows that there is a feature that is not used by the model and is shifted. In this case, we have a distribution shift, that does not affect the model f nor the explanation shift, and any “distribution shift” methods based on input data will flag it.

Reviewer comment (Experiments):
Why aren't methods that don't actually need a causal graph such as [2] compared to in the paper?

Author response:
A.4: We do not compare to this paper for two reasons:
This paper works in a different problem setup. It requires labels $D^new_Y$, whereas we assume that these are not available (cf. answer A.1; Main Point 1)
We will be happy to acknowledge the recent work that is reported in the two arxiv papers. However, given that they have not been published at a conference yet, we think that their absence in our discussion should not be held against our submission.

Reviewer comment (Theory):
I see a discussion on using LIME as a possible explanation for addressing the same, however, I am not sure whether crucial properties of Shapley are necessary for this method to succeed. I think additional discussion on which explanations have the desirable properties for use in detecting explanation shifts will make the paper stronger.

Author response:
A.6: The theoretical properties that we require from a feature attribution method are efficiency and uninformativeness (cf section 2.3). We use and compare Shapley values and Lime because these approaches (and their implementations) fulfil or at least approximately fulfil these properties.

Reviewer comment (Writing):
Motivation for considering shifts in explanation is not convincing. This, I believe, is because the goal is not stated concretely to then motivate the solution. The stated goal of investigating interaction between distribution shifts and learned model needs to be further characterized in quantifiable forms. I agree that not every type of distribution shifts are important to detect. The ones that change model behaviour in some meaningful way are important. However, it is unclear to me why explanations, that too Shapley values, is a meaningful or useful summary of model behavior to consider. Say, we care about AUC or any metric of choice to summarize model behavior, we can directly detect shifts in this metric using the same classification pipeline as proposed. If the reason for using explanations is for the interpretability of the shift in model behavior, the problem and solution needs to be stated differently. I think that some of this is addressed in the Appendix by pointing to related work that uses explanations for model debugging, drift detection, or other tasks. This could be used in Introduction to more clearly motivate the approach.

Author response:
A.1: Main Points 1 and 2 at the beginning of our rebuttal address these concerns. To summarize: AUC or likewise metrics cannot be applied in our problem setting, which is frequent in the real world, where new data $D_x^{new}$ comes without labels $D_y^{new}$.

As (Garg 2021) have shown, correctly estimating the performance $\text{perf}(D^{new})=\sum_{(x,y)\in D} \ell_\text{eval}(f_\theta(x),y)$ is impossible in the presence of distribution shift. Their results imply that AUC and likewise methods are inapplicable for discovering distribution shift when $D_y^{new}$ is not given.

Given the model predictions $f_\theta(x)$, we use the Shapley values, as they attribute the contribution of input features to the prediction (more information in section 2.3).

We will use your question and our response to make the introduction clearer. Thank you.

Reviewer comment (Evaluation):
Evaluation setup can be improved in terms of quantifying evaluation metrics like sensitivity. The magnitude of AUC does not matter that much I believe. Authors are right in asking for sensitivity in the method but it is not directly evaluated. For instance in Figure 2, it is unclear between (ours) and (B7) methods in the left figure and between (ours) and (B2) which one is better. All seem sensitive in the sense that different correlation coefficients result in different AUCs.

Author response:
A.2 In order to clarify and provide a quantifiable metric, we have extended the experimental evaluation in the main rebuttal. We say baselines are underperformant because they achieve a lower correlation with $\rho$ or are independent of the model used.

Reviewer comment (Presentation):
Presentation of experiment can be improved. The case study on ACS data in Sec 5.2.2 is a good place to showcase how the question in the paper title (how the shift impacted the model) is addressed by the method. The findings are stated in a matter-of-fact way. These could be contextualized in the data making the significance of the method and its results more apparent. The questions asked through the experiments could be described at the start of Sec 5 and what we learned from the results could be described more directly at the end of the section.

Author response:
Many thanks for the suggestion; we fully agree with the reviewer. These minor edits will improve the paper's clarity. We will take them into account.

Reviewer comment (Writing):
Please clarify the problem statement more formally. What aspects of the model behavior under distribution shifts (interaction between distribution shift and learned model) is of interest?

Author response:
A.4 Section 4 covers all the different types of shifts: Uninformative shifts vs Informative Shifts. Among the latter, problems in practice have to deal with Covariate shifts, Prediction shifts, and Concept shifts. Discovering concept shift (if not domain assumptions can be made, nor labeled data can be obtain), which is why we are interested in covariate and prediction shifts. These are all defined in section 4 and related to explanation shifts.

Reviewer comment (Theory):
Please motivate the approach (focus on explanations) in terms of the problem statement. How does Shapley values capture the desired aspects of the model behavior?

Author response:
The Shapley values, weigh the influence of feature values towards a prediction, 
$\sum_{j \in N} S_j(f, x^{\star}) = f(x^{\star}) - E[f(X)])$(Efficiency Property - cf. Section 2). 

The strength lies in the decomposition of predictions, offering insight into how each feature contributes to the overall outcome. This attribute is valuable when dealing with high-dimensional distributions, providing a more informative distribution than prediction distributions.

By employing feature attribution explanation techniques, specifically Shapley and LIME, we exploit this efficiency property. These techniques facilitate a granular understanding of how shifts in specific feature distributions impact the model. 
he decomposition of the prediction across the contribution of each features  allows for a higher dimensionality distribution than the predictions distributions. Therefore the explanation distribution compasses always more information.