Paper: ReX: A Framework for Incorporating Temporal Information in Model-Agnostic Local Explanation Techniques (ICLR-2024)

Reviewer comment (Experiments):
The experimental part is not very sufficient, and the comparison models used are not many and not the most advanced.

Author response:
We are conducting an experiment on *AcME—Accelerated model-agnostic explanations: Fast whitening of the machine-learning black box* and will report the results later.

Reviewer comment (Evaluation):
After adding the REX framework, the efficiency of the model has decreased significantly. Does this have practical application value?

Author response:
As we discussed in section 4.2, For LIME and KSHAP, ReX is not the main time-consuming part and does not need any more sampling to get explanations with more fidelity. 

For Anchors, the running time heavily depends if there are suitable predicates in the vocabulary. If ReX adds suitable predicates, the Anchors will run faster and otherwise slower.

And as we reported in Table 3, we didn't encounter notable issues with efficiency in experiments.

Reviewer comment (Presentation):
Which models correspond to the methods in Table 4?

Author response:
As we mentioned in Section 4.3, Table 4 is related to explaining the sentiment analysis LSTM with Anchors and Anchors. We are happy to also add such information in the caption of Table 4.

Reviewer comment (Evaluation):
It is unclear whether all 1-D and 2-D predicates positively impact explanations.

Author response:
1. Do all 1-D and 2-D predicates positively impact explanations?

We agree that there could be some redundant 1-D and 2-D predicates. 

But the essential point is that there exists some effective 1-D and 2-D predicates helping to construct explanations with higher fidelity. 
For rule-based methods like Anchors, if a predicate is not contributing to the explanation, it will simply be excluded. Moreover, for an instance that does not need 1-D and 2-D predicates, ReX generates explanation with the original feature predicates.
For attribution-based like LIME and KSHAP, 

This ensures that **ReX does not make an explanation worse**.

We appreciate your suggestion and will expand on this discussion.

Reviewer comment (Experiments):
Including more examples beyond those discussed in Section 3.1 could strengthen the paper's argument for the advantages of REX.

Author response:
We are conducting experiments on the Llama-2 model and the financial phrasebank datasets. We will report the results later.

Reviewer comment (Experiments):
Incorporating another textual dataset could bolster claims regarding REX's generalizability.

Author response:
We are conducting experiments on the Llama-2 model and the financial phrasebank datasets. We will report the results later.

Reviewer comment (Experiments):
The section on human evaluations lacks some details. For instance, is the experimental design a within-groups setup?

Author response:
We are happy to provide more details.

**The questionnaires are similar for all users with minor variations in the order of presentation.**. We presented the five questions, Q1, Q2,..., and Q5 in a random order. For each question, we presented the explanations generated without/with ReX in a random order. Also, we presented the ten sentences to be predicted in a random order.

Among all the 50 sentences to be predicted, on 19 sentences,  the prediction by LSTM model differs from groundtruth(the sentiment classified by humans). On these sentences ReX can still help people to predict better. Users make the same prediction as the LSTM model on 48.9% of these instances with vallina explanation methods, while ReX improves this to 66.7%.
In other words, ReX indeed helps humans make “wrong predictions” made by the model.

Reviewer comment (Experiments):
The precise procedure for the user study is unclear. For example, does each set consist of ten new sentences used specifically for that test block?

Author response:
We are happy to provide more details.

**The questionnaires are similar for all users with minor variations in the order of presentation.**. We presented the five questions, Q1, Q2,..., and Q5 in a random order. For each question, we presented the explanations generated without/with ReX in a random order. Also, we presented the ten sentences to be predicted in a random order.

Among all the 50 sentences to be predicted, on 19 sentences,  the prediction by LSTM model differs from groundtruth(the sentiment classified by humans). On these sentences ReX can still help people to predict better. Users make the same prediction as the LSTM model on 48.9% of these instances with vallina explanation methods, while ReX improves this to 66.7%.
In other words, ReX indeed helps humans make “wrong predictions” made by the model.

Reviewer comment (Evaluation):
Exploring challenging tasks, such as instances where the model makes incorrect predictions, would offer additional insights into human understanding of model reasoning.

Author response:
We are happy to provide more details.

**The questionnaires are similar for all users with minor variations in the order of presentation.**. We presented the five questions, Q1, Q2,..., and Q5 in a random order. For each question, we presented the explanations generated without/with ReX in a random order. Also, we presented the ten sentences to be predicted in a random order.

Among all the 50 sentences to be predicted, on 19 sentences,  the prediction by LSTM model differs from groundtruth(the sentiment classified by humans). On these sentences ReX can still help people to predict better. Users make the same prediction as the LSTM model on 48.9% of these instances with vallina explanation methods, while ReX improves this to 66.7%.
In other words, ReX indeed helps humans make “wrong predictions” made by the model.

Reviewer comment (Evaluation):
If humans can also make “wrong predictions” made by the model, it is very convincing that users understand the model's reasoning.

Author response:
We are happy to provide more details.

**The questionnaires are similar for all users with minor variations in the order of presentation.**. We presented the five questions, Q1, Q2,..., and Q5 in a random order. For each question, we presented the explanations generated without/with ReX in a random order. Also, we presented the ten sentences to be predicted in a random order.

Among all the 50 sentences to be predicted, on 19 sentences,  the prediction by LSTM model differs from groundtruth(the sentiment classified by humans). On these sentences ReX can still help people to predict better. Users make the same prediction as the LSTM model on 48.9% of these instances with vallina explanation methods, while ReX improves this to 66.7%.
In other words, ReX indeed helps humans make “wrong predictions” made by the model.

Reviewer comment (Experiments):
It is unclear whether REX could be generalized to image inputs, particularly simple image sequences with important temporal information for classification.

Author response:
Thanks for your suggestion. We would add such a discussion in our revised version. Briefly:

**ReX can be generalized to image sequence inputs.**

We use video classification models as an example.

To the best of our knowledge, there are few works especially no model-agnostic techniques focusing on explaining these models. One possible way to apply existing local model-agnostic methods is to consider each frame as a feature and use the presence of these frames as feature predicates in the explanation vocabularies. Formally, for an input $x$, as mentioned in Section 2, feature predicates $\{p_j\}_{j=1,2,...,|x|}$ are defined as $p_j := f_j\ op\ c_j$ where $f_j$ represents the j-th frame and $c_j$ is usually the original value of $f_j$. For perturbation models, existing local techniques change the feature values, i.e. change the image of some frames.

To incorporate temporal information, ReX extends the vocabularies and perturbation models in the following way.
1. Extending vocabularies:

In Section 3.1, we define 1-D predicates as
$$\exists i \in \mathbb{Z}^{+}\ such\ that\ f_i\ op_1\ c \wedge i\ op_3\ d  
     \quad\text{ where }c \in \mathbb{R}, d\in \mathbb{Z}^{+}, op_1,op_3 \in \{=, >, <,\geq,\leq\}$$
and 2-D predicates as
$$
\begin{array}{r@{}l}
    &\exists i, j \in \mathbb{Z}^{+}\ such\ that\ f_i\ op_1\ c_1 \wedge f_j\ op_2\ c_2 \wedge j - i\ op_3\ d \\
    &\text{ where }c_1, c_2 \in \mathbb{R}, d\in \mathbb{Z}\setminus\{0\}, op_1, op_2, op_3 \in \{=, >, <,\geq,\leq\}.
\end{array}
$$
We illustrate the effects of a single feature(i.e. frame)'s absolute position by 1-D temporal predicates and the effects of the relative position between two features by 2-D temporal predicates.
And we incorporate these predicates into the vocabularies.

2. Extending perturbation models:

We can still follow the description in section 3.2, using function $rf: \mathbb{I}\rightarrow 2^{\mathbb{I}}$ to remove features and $sf:\mathbb{I}\times\mathbb{Z}\times\mathbb{Z}\rightarrow 2^{\mathbb{I}}$ to switch features, and then construct a extended perturbation model in the same way.

Reviewer comment (Evaluation):
The interpretation of Table 4 in Section 4.3 lacks precision. Specifically, the claimed improvements in precision and coverage, cited as "80.9% and 56.7%" in the text, cannot be observed from the results in Table 4.

Author response:
As we mentioned in Section 4.3, Table 4 demonstrates that the average $precision_u$ of Anchors* is 87.3% while that of Anchors is 48.3%, so we claim that ReX improves $precision_u$ by 80.9%(**$48.3\times 180.9\\% = 87.3$**). Similarly, the average $coverage_u$ of Anchors* is 68.8\% while that of Anchors is 43.9%, so we claim that ReX improves $coverage_u$ by 68.8%($43.9*156.7\\% = 68.8$).

Reviewer comment (Experiments):
What is the average length of the textual input data?

Author response:
1. What is the average length of the textual input data?

Average length: 19.23

Reviewer comment (Evaluation):
If the text input data consists of long paragraphs, would that impose computational burdens on REX?

Author response:
2. If the text input data consists of long paragraphs, would that impose computational burdens on REX?

Such long inputs would cause the baseline explanation methods to run longer but REX doesn't make the situation worse. 

The model-agnostic explanation techniques require to do perturbations on input data. A very long input sentence will increase the time of obtaining the results of the target model's output. And to produce explanations with high fidelity, the number of sampled instances will increase a lot. 

For ReX, As we discussed in section 4.2, for LIME and KSHAP, ReX is not the main time-consuming part and achieves better fidelity with the same number of samples as the original techniques. For Anchors, the running time heavily depends if there are suitable predicates in the vocabulary. If ReX adds suitable predicates, the Anchors will run faster and otherwise slower.

Reviewer comment (Experiments):
Empirical evaluation can include more models and datasets.

Author response:
We are conducting experiments on the Llama-2 model and the financial phrasebank datasets. We will report the results later.

Reviewer comment (Theory):
In case of LIME-Text explainer (equivalently for SHAP), words of a sentence are randomly deleted to obtain different binary vectors (1= word present, 0=absent) in order to compute distance between original & perturbed samples and fit a linear model. If the words are shifted, then the binary vectors remain unchanged. It was unclear from text how temporal information is recovered in this setting using the existing ridge regression model used by LIME?

Author response:
The temporal information is recovered using the temporal predicates. 
After LIME is augmented by ReX, the 1-D and 2-D predicates represent the effect of absolute positions and relative positions respectively. And for the corresponding binary vectors, 1 means the predicates evaluate to true, while 0 means false. If a word is shifted, its absolute position and relative position to other words are also changed. Therefore, **the binary vectors corresponding to the predicates about the word's position are changed**, so the temporal information can be recovered using the existing regression model.

Reviewer comment (Experiments):
Compared to the base case of no temporal information, how do (a) number of perturbed samples and (b) number of parameters in the linear model fit by lime/shap, scale when temporal information is requested from the explainer?

Author response:
(a) We achieve higher fidelity by using **the same** number of perturbed samples.

(b) The number of parameters equals the number of predicates in the vocabulary. For sentiment analysis, we simply use all the feature, 1-D, and 2-D predicates. For anomaly detection, we limit the temporal predicates in a window of 20 steps.

Reviewer comment (Writing):
Section 3. Your notation is under-specified. It may be helpful in this section to clarify what the 'features' of the example inputs you presented are (e.g., what are the 'features' of the sentence input "Bob is not a bad boy")? Is $f_j$ in this case the token that appears at index $j$ in the sentence, or something else? Similarly, do your feature predicates $p_j$ compare the value of token at index $j$ to some threshold (e.g. "the token at timestep $t$ = 'fails'", or something else)? (EDIT: After reading Section 3.1, it seems like $f_j$ is the value of the token at index $j$? Can you clarify?)

Author response:
### b.Section 3. Your notation is under-specified. It may be helpful in this section to clarify what the 'features' of the example inputs you presented are (e.g., what are the 'features' of the sentence input "Bob is not a bad boy")? Is $f_j$ in this case the token that appears at index $j$ in the sentence, or something else? Similarly, do your feature predicates $p_j$ compare the value of token at index $j$ to some threshold (e.g. "the token at timestep $t$ = 'fails'", or something else? (EDIT: After reading Section 3.1, it seems like $f_j$ is the value of the token at index $j$? Can you clarify?)

**Answer:**

1. what the 'features' of the example inputs you presented are?

There is a general definition of features that defines them as **attributes of input data.** For example, in image data, features can be pixels, and in sentence data, features can be words. Since this definition of features is widely used and given the limited space, we did not provide a detailed explanation of what features are.

For the example in section 3.1, the "features" refer to the tokens (words) of the example inputs.


2. Is $f_j$ in this case the token that appears at index $j$ in the sentence?

Yes. We have defined $f_j$ in **the last paragraph of page 3**, where we stated that *$f_j$ represents the $j$th feature of input $x$*. In the example, $f_j$ represents the j-th token of the example inputs.


3. do your feature predicates $p_j$ compare the value of token at index $j$ to some threshold.

Yes. We have also defined feature predicates $p_j$ in **the last paragraph** of page 3, where we stated that *$p_j := f_j\ op\ c$*. Here, $p_j$ compares the value of the token at index $j$ to a threshold value denoted as $c$.


We are happy to incorporate additional explanations to provide further clarity in the paper.

Reviewer comment (Writing):
Section 3.2. "Extending vocabularies: I don't understand what is described in this paragraph. What do you mean by 'serve as a feature of an input', and a 'method to evaluate the predicate on a given input'? Do you mean that you construct a new set of indicators for each original datapoint where you evaluate whether the predicate holds for that datapoint, and then learn each "surrogate model" using the set of original features plus the predicates?

Author response:
### c.Section 3.2. "Extending vocabularies: I don't understand what is described in this paragraph. What do you mean by 'serve as a feature of an input', and a 'method to evaluate the predicate on a given input'? Do you mean that you construct a new set of indicators for each original datapoint where you evaluate whether the predicate holds for that datapoint, and then learn each "surrogate model" using the set of original features plus the predicates?

**Answer:**

We define the *vocabulary* in the first paragraph of page 4. The vocabularies are a set of predicates used to construct the explanations.

The original vocabularies only contain **feature predicates**. ReX **extends** the vocabularies by incorporating additional predicates such as the 1-D and 2-D temporal predicates mentioned in **Section 3.1**, so that can incorporate temporal information in the explanation(**Definition 3.3**). 

Since the vallina explanation methods use only the original features as predicates, we said we should make new predicates also "serve as a feature".

When utilizing the predicates in the explanation techniques, existing explanation methods need to know if the predicates are true or false, so we need to "evaluate the predicate on a given input".

Reviewer comment (Reproducibility):
Section 3.3. I am confused from the explanation provided about how the perturbation model that you've described is used for each of the individual explanation methods. It is unclear to me how this perturbation method evaluates the possible importance of all of the possible predicates (from the many possibilities that exist). Take LIME as an example. I am confused about how being able to generate new datapoints where features are "switched" (for example, say we switch $f_j = c_j$ and $f_i = c_i$), allows us to assign an "importance score" to, for example, the predicate $f_j = c$ AND $f_i = c_i$ AND $i - j >= 1$. Can you clarify exactly how you calculate the importance scores for each of the 1D and the 2D temporal predicates using such perturbation methods?

Author response:
### d.Section 3.3. I am confused from the explanation provided about how the perturbation model that you've described is used for each of the individual explanation methods. It is unclear to me how this perturbation method, evaluates the possible importance of all of the possible predicates (from the many possibilities that exist). Take LIME as an example. I am confused about how being able to generate new datapoints where features are "switched" (for example, say we switch $f_j = c_j$ and $f_i = c_i$, allows us to assign an "importance score" to, for example, the predicate $f_j = c$ AND $f_i = c_i$ AND $i - j >= 1$. Can you clarify exactly how you calculate the importance scores for each of the 1D and the 2D temporal predicates using such perturbation methods?

**Answer:**

Let's take ReX+LIME(LIME*) and sentiment analysis as an example to clarify how the perturbation methods calculate the importance scores for each predicates.

For an input sequence $x = w_1w_2w_3$, where $w_i$ represents the $i$-th word in the sequence, we can perform a perturbation by switching the 2nd and 3rd words, resulting in $x' = w_1w_3w_2$.

The feature predicates $f_2 = w_2$ and $f_3=w_3$ will evaluate to **false** on $x'$ since the 2nd feature of $x'$  is $w_3$ while the 3rd one is $w_2$. 

For 2-D predicates like 
\begin{array}{r@{}l}
    &\exists i, j \in \mathbb{Z}^{+}\ such\ that\ f_i\ =w_2  \wedge f_j\ =\ w_3 \wedge j - i\ \ge \ 1 \\
\end{array}
will evaluate to false. since there exists $f_3 = w_2$ and $f_2 = w_3$ but $2-3<1$.

LIME samples multiple datapoints using the perturbation model. For each sampled datapoint, LIME* calculates the results of all predicates in the explanation vocabularies, and stores the results as a binary vector, along with the corresponding model output for that datapoint.

This process generates a matrix, as shown below:


|     | Predicate 1 | Predicate 2 | ... | Predicate m | Model Output |
| --- | ----------- | ----------- | --- | ----------- | ------------ |
| data1 | 1 | 1 | ... | 0 | 1 |
| data2 | 1 | 1 | ... | 1 | 0 |
| ... | ... | ... | ... | ... | ... |
| datak | 1 | 0 | ... | 1 | 1 |

LIME uses a regression algorithm to assign importance scores to each predicate based on this matrix.

Reviewer comment (Experiments):
Section 4.3. Can you clarify how you assigned participants to explanation conditions in the user study? Did a single participant only see explanations from 1 of {Anchor, Anchor*}? Did you present a single "test", and then ask them to simulate the model on 10 sentences, several different times (so elicited 50 total predictions), or show all 5 "tests" right away and then collect 10 predictions per user?

Author response:
### e. Section 4.3. Can you clarify how you assigned participant to explanation conditions in the user study? Did a single participant only see explanations from 1 of {Anchor, Anchor*}? Did you present a single "test", and then ask them to simulate the model on 10 sentences, several different times (so elicited 50 total predictions), or show all 5 "tests" right away and then collect 10 predictions per user?

**Answer**

We are happy to provide more details about the user study.

1. Can you clarify how you assigned participant to explanation conditions in the user study?

We showed all users the five questions and 2 explanations. The questionnaires are similar for all users with minor variations in the order of presentation. We presented the five questions, Q1, Q2,..., and Q5 in a random order. For each question, we presented the explanations generated without/with ReX in a random order. Also, we presented the ten sentences to be predicted in a random order.

2. Did a single participant only see explanations from 1 of {Anchor, Anchor*}?

No, each participant saw both explanations.


3. Did you present a single "test", and then ask them to simulate the model on 10 sentences, several different times (so elicited 50 total predictions), or show all 5 "tests" right away and then collect 10 predictions per user?

We presented one explanation at a time and asked participants to simulate the model on 10 sentences for each explanation. Since there were 10 explanations in total, we repeated this process 10 times.