Paper: TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documents (ICLR-2024)

Reviewer comment (Presentation):
No diagram or algorithm besides Figure 2, which is poorly captioned and ambiguous in places. The paper includes many details, but compelling and concise high level summaries are lacking.

Author response:
We have updated the diagram for Figure 2 to focus on detailing the speculative decoding methodology we employ in our sampling algorithm.  We hope that this can give much greater insights into how the speculative decoding works, but also give some basic insights into how our sampling algorithm proceeds.  In particular, the algorithm proceeds by continuing to sample random subsets, compute the model output, and update the Shapley values.  This can be continued until convergence, but for our work we stop after a fixed number of samples.

Reviewer comment (Presentation):
The statement *"In the traditional Shapley, log-probabilities are needed for every candidate output"* should come at the start of section 3.1 and not the end (the start of the paragraph is less motivated and more confusing as a result).

Author response:
We would like to thank the reviewer for their specific and useful feedback which improved the draft, especially motivating the largely improved Figure 2.

Regarding the placement of the algorithm, we toyed around with some ideas, mainly: reproducing the standard Shapley permutation sampling algorithm in the main text (but this felt too rudimentary for a Shapley audience), and including a condensed version of our algorithm into the main text (but this ended up feeling even more confusing than having nothing).  We understand that the current solution is not perfect, and are happy to consider further suggestions on how to resolve this.

Thank you for your reevaluation, and we are glad to have appropriately addressed your concerns.

Reviewer comment (Evaluation):
The improvements are less pronounced on the MIRACL dataset and performance appears varied.

Author response:
Here, we actually believe that this does not represent a shortcoming of our method as discussed briefly in the results section.  As mentioned in the response to everyone, we have updated the appendix with specific MIRACL examples showing case scenarios where our method clearly answers correctly, but receives no AUC points because of the methodology used to construct the MIRACL dataset.  In particular, the MIRACL dataset human annotators only annotate 10 labels, specifically the top ten retrieved by an existing retrieval system. Accordingly, our method’s ability to find a more diverse set of passages amongst extremely long contexts is not effectively captured by the “MIRACL (Original)” AUC metrics.  To address this, we explore using psuedolabels for the MIRACL dataset, and in this scenario our method consistently outperforms the baseline, emphasizing the effectiveness of our approach.

Reviewer comment (Writing):
Notations are not fully explained and are hard to parse. For example, in Notation paragraph, why is the input space of a language model defined as $\mathbb{R}^d \times \mathcal{P}([d])$. What do these notations correspond to in plain English? In the same paragraph, $[.]_+$ is introduced but not used. It would be helpful if the authors could update the notation and check the consistency throughout the paper.

Author response:
First, we hope that the updated Section 3 will be easier to parse than the previous version, but we still answer your specific questions directly here.
The input space of a masked text generation model defined as $\mathbb{R}^d \times \mathcal{P}([d])$, is now written as $[V]^d \times \\{0,1\\}^d$ which hopefully makes it clearer that ${x}\in \mathcal{X}=[V]^d$ represents the input sequence and ${s}\in\mathcal{M} = \\{ 0,1 \\}^d$ represents the binary input masks.  $[\cdot]_+$  is now moved to be introduced right after it is used for the first and only time.

Reviewer comment (Theory):
The motivation for technical details is not fully explained. For example, in the current version, I do not understand why Eq (2) is better than Eq (1) in explaining text generation.

Author response:
We have now further delineated Eq (1) and Eq (2) by forcing the different choice of $v_\ell(\cdot)$ and $v_p(\cdot)$ respectively. The major difference is as follows. It is computationally infeasible to compute the full probability vector over the exponentially large space $[0,1]^{[V]^m}$, so we will instead turn to greedy decoding or K-beam decoding.  But using these algorithms will make the probability of generating certain outputs exactly zero, meaning we can no longer consider their log-probabilities and the entire Eq (1) becomes an ill-defined quantity. In contrast, Eq (2) can set some probabilities to zero without taking the logarithm, and the defined quantity is not only well-defined but should additionally be close to the Shapley-Shubik value as if we were to have the full exponentially-sized vector.

Reviewer comment (Evaluation):
Figure 4 shows that the proposed method is not always better than an extremely simple baseline (i.e., similarity score), especially on MIRACL.

Author response:
In the discussion of the results, we actually mention why we think this demonstrates the potential usefulness of our method, rather than a downside. Because the MIRACL dataset was constructed using the same similarity-score based approaches, there is a higher likelihood that a human annotator will give a positive label to the passages preferred by existing similarity-score approaches.  We have further added Section E to show exactly what we mean by this on specific examples from the MIRACL corpus and give further motivation for the pseudo-label approach as a fair comparison of the different methods.

Reviewer comment (Experiments):
Lack of baselines. Although previous works on text generation explanation would take a long time to run, it would still be interesting to see their performances and time consumption. Currently, the paper does not include any baseline for explanation.

Author response:
To the best of our knowledge, there are no existing feature attribution methods which can be directly applied to seq-to-seq text generation.  The main category of explanations available for text-to-text generation are natural language explanations designed for end users.  We are now trying to adapt existing feature attribution methods to enable comparison and are currently running experiments to try to compare AUC results.  We hope to update you once they are completed.

Reviewer comment (Evaluation):
On the token-wise explanations, Shapley-derived insights offer only superficial clarity on text generation tasks. The significant scores of tokens give minimal information to the audience. For instance, the tokens are sometimes not even readable words. Moreover, identical tokens in the starting prompt and following question context could have varying importance ratings, which can potentially confuse the audience.

Author response:
> identical tokens in the starting prompt … could have varying importance ratings

We apologize for the confusion regarding this.  Indeed, our method can easily be applied to the instructions or prompt of the LLM.  In our application, we are focusing on the explanations for the documents because it is meaningful for the QA task.  For future work, as you mention, an important aspect of LLMs to focus on is understanding the prompt.  For example, imagine an application where you are trying to design a good set of instructions.  Maybe you have five different prompts and they are all giving the same performance.  Using our method, you can see that prompt #1 is important for certain data samples while prompt #2 is important for other data samples.  Here, we believe it is a feature and not a bug that the importance rating will change for different data samples.  One can imagine that after seeing TextGenSHAP’s explanation, you will be able to design a new prompt which has the important parts of prompt #1 and the important parts of prompt #2, or perhaps a pipeline which automatically selects the correct prompt to be used for each specific example.  (similar to how we automatically select the relevant documents in our application)

> token-wise explanations … offer only superficial clarity

Although we agree that tokens are sometimes not readable words, we believe that the tokens are a critical ingredient to providing an accurate explanation.  For example, the same English word can be tokenized in multiple different ways, and it is possible that the model prediction will change with different tokens, meaning our explanations should also change.  If your major issue is with tokens being too small and unreadable words, we think you will appreciate our hierarchy we used going from paragraphs → sentences → words, which puts less emphasis on small unreadable tokens.  We encourage you to check out our linked visualization.

Reviewer comment (Evaluation):
The Shapley attributions on input tokens reveal unreliable explanations on text generation tasks, as the outputs of LLMs are usually uncertain. The output quality of LLMs usually highly depends on the instructions given to LLMs, which means the prediction changes may related to the changes in instructions rather than the input questions and other stuff.

Author response:
> output quality of LLMs usually highly depends on the instructions given to LLMs

> identical tokens in the starting prompt … could have varying importance ratings

We apologize for the confusion regarding this.  Indeed, our method can easily be applied to the instructions or prompt of the LLM.  In our application, we are focusing on the explanations for the documents because it is meaningful for the QA task.  For future work, as you mention, an important aspect of LLMs to focus on is understanding the prompt.  For example, imagine an application where you are trying to design a good set of instructions.  Maybe you have five different prompts and they are all giving the same performance.  Using our method, you can see that prompt #1 is important for certain data samples while prompt #2 is important for other data samples.  Here, we believe it is a feature and not a bug that the importance rating will change for different data samples.  One can imagine that after seeing TextGenSHAP’s explanation, you will be able to design a new prompt which has the important parts of prompt #1 and the important parts of prompt #2, or perhaps a pipeline which automatically selects the correct prompt to be used for each specific example.  (similar to how we automatically select the relevant documents in our application)

Reviewer comment (Evaluation):
TextGenSHAP claims to estimate Shapley Values. However, the proposed evaluation metrics are insufficient to support the claims. It is highly recommended to provide either theoretical analysis on the axiom of Shapley Value on the generated values from TextGenSHAP or empirical evidence with commonly used evaluation metrics (e.g., absolute error metrics or l2 error) that can directly reveal the values are similar to Shapley Values.

Author response:
> TextGenSHAP claims to estimate Shapley Values. 

We add some more detailed discussion on the Shapley value and typical approaches to the appendix, mentioning why this approximation is standard.  For greater details on this approximation and for L1 and L2 errors on toy datasets one can refer to [1,2].  We emphasize that such L2 metrics can only be calculated on toy datasets since for real datasets it is computationally infeasible to get the exact value.  Hopefully this can help explain why these sampling algorithms are standard in the first place.

Reviewer comment (Writing):
Some annotations are not stated clearly in the methodology section. For instance, $v(\cdot)$ is only illustrated as a value function, but this is still vague under the settings of LLMs. According to the paper, the value function maps features to text outputs, which means the output of the value function is only the tokens but not the probability of each token. If this is the case, could the authors explain how the estimation of Shapley values in Eq. (2) has been processed?

Author response:
> Some annotations are not stated clearly

Hopefully our revamped notation and background section can alleviate these concerns.  The ‘value function’ terminology is standard for the Shapley value, but now we have changed from the one-hot output generation to writing $v(\cdot)$ as a probability vector.  We have also delineated the log-likelihood value function and the probability value function to make the distinction clearer.

Reviewer comment (Evaluation):
The outputs produced by LLMs can be considerably influenced by various sampling strategies (e.g., temperature, search methods, etc.), which means the output probability can be very different even the LLMs receive the same input text. Are there specific approaches within your framework designed to address these challenges? Please correct any misinterpretation on my part. From my perspective, the mechanism proposed in this paper does not equip to tackle or circumvent these inherent difficulties when explaining the generative results of LLMs.

Author response:
> influenced by various samplings

Indeed various search strategies can affect the output generations. Although temperature scaling will have a trivial linear effect on the output generations, other methods like K-beam and top-P sampling will affect the support of the generated output, consequently influencing the final output distribution.  We mention this briefly in Section 4.1, reiterating that our approach can easily be extended to handle any of these decoding methods, but we stick with greedy decoding which is standard for ODQA.

Reviewer comment (Reproducibility):
Some essential details are omitted in the paper. For example, how to do speculative decoding is not clear to me.

Author response:
We first emphasize that speculative decoding is an exact computation of the originally decoded likelihoods, but just decodes in a more efficient way.  We have now clarified this in the text and updated Figure 2 to focus on explaining how the speculative decoding method gradually constructs a tree of possible outputs which can then all be computed in a single pass of the decoder (instead of needing multiple autoregressive passes when decoding one-by-one).  We are happy to answer further details which are still unclear after our new figure.