Paper: Trust Regions for Explanations via Black-Box Probabilistic Certification (ICLR-2024)

Reviewer comment (Evaluation):
It is unclear the pros and cons of the three sampling strategies. The authors did not provide a deeper discussion of the three sampling strategies.

Author response:
As mentioned in the Experiments section, the pro of unif is that it is the simplest and also sometimes the fastest strategy. unifI and adaptI are more complicated but are preferable when you have $1000$s or $10000$s of features respectively. Our advice would be to use unif for up to $100$s of features, unifI for $1000$s of features, and adaptI for $10000$s of features or greater.

Reviewer comment (Experiments):
The experiments are confusing. What is the purpose of ZO and why is it used as a baseline? Why does the convergence of w to a similar value indicate the accuracy of the explanation?

Author response:
As mentioned in the related work, ZO is a standard approach used for black-box adversarial attacks. We adapted this method to our setup since ours is a novel problem in XAI with no prevalent baselines, where we thought an adapted version of ZO would be the closest and fairest baseline.  

First note that the convergence of $w$ relates to the accuracy of the estimated *trust region width* for a given threshold $\theta$ and has nothing to do with the quality or accuracy of an *explanation*. For the synthetic experiments, we know the true $w$ and hence we can validate the different methods with certainty. For the real datasets, the true $w$ are unknown. Thus the different methods converging to similar values gives some confidence that the methods are returning an accurate enough $w$, as the mechanics behind the methods are quite different (especially between ZO and the others) and hence their converging to a similar value just by chance is unlikely.

Reviewer comment (Presentation):
What is the intuition of method design? At the least, the author should provide one figure to illustrate why the method is designed in this way and how the method works.

Author response:
We have now added Figure 1 to the paper which tries to convey the main ideas behind the strategies. More detailed description is provided in Section 4. Also note that we had uploaded videos showing the working of the three strategies in our original submission.

Reviewer comment (Experiments):
Could the authors provide a real-world application to demonstrate the significance of the research?

Author response:
One real-world application is *explanation reuse*. As mentioned in (Dhurandhar et al., 2019) (and also in our experience), given today's cloud-driven world where a model could exist on one cloud platform and explainability is offered as a service by a different corporation, each query to the black box model has an associated cost and hence finding explanations for many examples in a dataset can be prohibitive, not just in terms of time and network traffic but also monetarily. Finding regions where an existing explanation is valid can produce significant savings in all these three aspects, since one now does not have to find explanations for every example which typically takes $1000$s of queries (e.g. $5000$ is the default for LIME).  

Another example is in finance where banks want to find cohorts where a certain action (related to credit cards, loans, etc.) might have similar effect on all members of the cohort. Our approach will provide such cohorts where a certain explanation is valid for the cohort and hence recourse may be possible. More applications can be found in  (Liao et al., 2022), where they surmise that stability of explanations, which will lead to such trust regions, is important for stakeholders performing tasks such as model improvement, domain learning, adapting control and capability assessment.

Reviewer comment (Evaluation):
What are the pros and cons of the three sampling strategies? What kind of scenario are they suitable for and what insight do they give practitioners?

Author response:
As mentioned in the Experiments section, the pro of unif is that it is the simplest and also sometimes the fastest strategy. unifI and adaptI are more complicated but are preferable when you have $1000$s or $10000$s of features respectively. Our advice would be to use unif for up to $100$s of features, unifI for $1000$s of features, and adaptI for $10000$s of features or greater.

Reviewer comment (Experiments):
The authors claimed the method is model agnostic and quite general. So can the method generalize to graph data?

Author response:
Yes, the method can work for graph data if the graphs are embedded in some compact space. For example by vectorizing adjacency matrices or using embeddings from Graph Neural Networks (GNNs).

Reviewer comment (Theory):
To what extent is the found w stable? i.e., the bound of P (f (x);w) >=theta.

Author response:
As seen in the experiments (see Tables 2-4 in the appendix), the found $w$ is quite stable from $Q=1000$. Even $Q=100$ is sufficient for low dimensions. The bounds are also quite tight from $Q=1000$ (see Tables 5, 7 in the appendix).

Reviewer comment (Theory):
Despite that the authors mentioned in Section 8 Discussion that the l_\infinty ball can potentially be generalized to hyper-rectangles or l_p balls, all these hyper-balls may not carry semantic meanings of the samples. Therefore even if we can certificate that the pseudo-samples within the hyper-balls have high fidelity, those pseudo-samples may not carry meaningful information, since human interpretation on images is very different from Euclidean distances. It is encouraged that the authors address this concern with a discussion.

Author response:
Yes you are right that all examples in a region may not be realistic input examples in the domain lying on the data manifold. In such cases one could after the region is found consider only realistic examples in the region for their downstream task. In a sense we would provide a superset of examples for which the explanation is valid. 

Another option as discussed in Appendix G is one could simply apply our methods to the latent space
(e.g. as learned by an auto-encoder) rather than the input space. Thus, although the regions will
be hypercubes in the latent space they will be more free-form in the input space which might be interesting. As such, our approaches should still apply.

Reviewer comment (Theory):
The connection between the Algorithm 1 and Algorithm 2 proposed in Section 4 seems to be detached from the original optimization formulated in equation 1. It is encouraged that the authors strengthen the connection.

Author response:
Algorithm 1 decides on a width $w$ and Algorithm 2 tries to certify that width (i.e. check if $f_{x_0}(x) \ge \theta \quad \forall x \in B_\infty(x_0, w)$). If certified, Algorithm 1 chooses a bigger width which again Algorithm 2 tries to certify. Thus, the two algorithms are tightly linked to the optimization formulated in eq. 1, where we are trying to find the maximum possible (certified) width.

Reviewer comment (Experiments):
The authors mention several advantages of the novel explainability certificate, including stability of the explanations and explanation reuse, but did not provide numerical results or analysis on these benefits over existing explainable AI tools. It is encouraged that the authors

Author response:
Our approach compliments the current work in XAI where XAI toolkits (viz. AIX360, Captum, InterpretML, etc.) contain predominantly explanation methods and/or (quality) metrics. One can take these methods and metrics and apply our approach on top to obtain trust regions, which could be used for any of the applications outlined in the introduction. As such in the experiments we do provide a comparison between LIME and SHAP using our methodology that is of a different flavor than previous works. This type of analysis can be used to compare and contrast XAI methods
on individual examples, on regions, as well as on entire datasets, and across different models. Also note that our methods perform quite well around $Q=1000$ where we obtain a region (i.e. a set of examples) in which an explanation is valid. This is (potentially) much fewer queries than say LIME-like methods which by default query a model $5000$ times per example it wants to find an explanation for.

Reviewer comment (Novelty):
This certificate is closely related to adversarial examples, and in related literature there are also studies on the maximal hyper-balls that will not suffer from a decision change. Their algorithm is also similar to the searching algorithm proposed in Algorithm 1. The novelty here is the quality metric and the explanation function. It is encouraged that the authors include more discussion on different explanation functions, or give concrete examples in Section 2.

Author response:
Example explanation functions would be linear like in LIME, logistic regression, small decision trees, rule lists, interpretable neural architectures such as CoFrNet. We have now mentioned some of these in Section 2.

Reviewer comment (Experiments):
In XAI, the explainability of features is sometimes more important than individual samples. Is it possible to extend the techniques to features?

Author response:
A thing to note here is that when we find trust regions we are finding instances in the input space where an explanation, which are typically feature importances, is valid
for all of them. In other words, the feature importances hold for all examples in that region. Hence, we are highlighting important features in that region.

However, if your goal is to find features that are similar one could potentially transpose the data matrix and find regions of similar features, where the explanations would now point to important instances.

Reviewer comment (Experiments):
One of the most important aspects in XAI tools is visualization. It is encouraged that the authors add visualization of the regions, provide some intuitive explanations, with simple (semi-)synthetic data.

Author response:
We have now added Figure 1 to the paper which tries to convey the main ideas behind the strategies. More detailed description is provided in Section 4. Also note that we had uploaded videos showing the working of the three strategies in our original submission.

Reviewer comment (Novelty):
I felt only the first of the three techniques is well motivated, while the other two don't really contribute to the main idea of the paper.

Author response:
> I felt only the first of the three techniques is well motivated

In Section 4 last paragraph, we motivate the three strategies. As mentioned there and further referenced in Appendix D, we previously shared videos to provide more intuition about the methods. Now in the introduction we have added a figure to further clarify the working of these methods.

As such, unifI ($2^{\text{nd}}$ technique) resembles a dynamic grid search where we randomly pick different locations in the current region and search around them. The $3^{\text{rd}}$ technique adaptI does a dynamic adaptive search where we again randomly pick different locations in the current region and search around them, but additionally focus our search more and more around the locations that we find more promising points (a.k.a. low fidelity examples).

Reviewer comment (Writing):
Since the main contribution is a formalization of the novel problem, it would be nice to have a simple example to illustrate what a certification looks like.

Author response:
> it would be nice to have a simple example to illustrate what a certification looks like.

As mentioned above we have now added Figure 1 to the paper which tries to convey the main ideas behind the strategies. More detailed description is provided in Section 4. Also note that we had uploaded videos showing the working of the three strategies in our original submission.

Certification in Figure 1 would be our algorithm outputting $w=0.5$ with the (certification) probability based on Theorem 1 being $1$ for all three strategies.

Reviewer comment (Writing):
What is fidelity? Perhaps the definition is not strictly necessary for understanding the paper, but since it is used so many times, I think it is critical to define it.

Author response:
> What is fidelity?

As you rightly mentioned, the exact definition of fidelity is not critical to the exposition but, to improve clarity, we now in Section 2 point to eq. 14 in the appendix, where we have defined fidelity consistent with previous works (Dhurandhar et al., 2022; 2023; Ramamurthy et al., 2020).

Reviewer comment (Writing):
The pseudo-code and the explanation in section 4 are not well written. I may be wrong, but the pseudo-code seems to be executing a binary search, in which case it is needlessly obscured.

Author response:
> ... but the pseudo-code seems to be executing a binary search, in which case it is needlessly obscured.

Algorithm 1 doubles or halves the width depending on if the current region was certified or not. It is similar to binary search but with subtle differences such as the doubling and also what upper bound width to set when a region is found to be invalid (i.e. the "else" statement). Algorithm 2 embodies the three strategies for certifying a fixed region and is very different than binary search.

Reviewer comment (Evaluation):
For w1>w2>w3, it seems to be possible that hypercubes of halfwidth w1 and w3 could be valid trust regions while w2 isn't. Since the problem statement involves finding the largest certificate, would the certification really be meaningful around the example? For instance, if the explanation passes the quality threshold over most of the domain but is bad around important points, it seems that the certificate could just be the entire domain, revealing little about the point of interest. Is my understanding correct?

Author response:
> For w1>w2>w3, it seems to be possible that hypercubes of halfwidth w1 and w3 could be valid trust regions while w2 isn't.

This is not possible. We are trying to find a connected (indeed, convex) region in the input space around a point where its explanation is valid for all examples within this region (with high probability). If w2 is found to be invalid our approach will return only w3 in this case.