Paper: XplainLLM: A QA Explanation Dataset for Understanding LLM Decision-Making (ICLR-2024)

Reviewer comment (Writing):
The writing requires improvement in two key areas: (1) The technical details are unclear, particularly regarding the training of the QA model M and graph model G, as well as the evaluation process for the reason-elements. (2) Numerous typos need correction. E.g., in the subsection of Decision Interpretation on Page 4, "consider any node..." needs to be "for any node".

Author response:
**1. (W1.1) Clarification on Technical Details:**

1. Training of Model M and Model G: Model M is our pre-trained LLM, and Model G is GAT to extract the reason-elements. We use the pre-trained RoBERTa-large as Model M, and the GAT is trained with the Model M (embedding concatenation). They trained with the loss function of cross-entropy loss. The optimization involves fine-tuning both the knowledge representation (via model G) and the predictive components (via model M). The training details can be found in Appendix E.5.1-2. We will revise Section 3.1 to clarify the training details.
2. Evaluation Reason-elements: The reason-elements are 
the key elements that contribute to the decision-making process of the model. They are extracted from the graph modular, mainly based on the attention weights of the GAT. Therefore, they are evaluated by the accuracy of the decision model. We will revise Section 3.1 to clarify the evaluation of reason-elements.

**2. (W1.2) Correction of Typos:**
We appreciate the reviewer pointing out the typos in our manuscript. We will thoroughly review the document to correct these errors, and ensure that our revised version is free of typos.

Reviewer comment (Experiments):
The task 2 setting is a bit redundant. Since one of the input E_why already contains the predicted answer, why we need to feed it to another QA model for prediction? How often does the QA model's predicted answer differ from E_why?

Author response:
**3. (W2) Clarification on Model Selection:**

The design of Task 2 is primarily focused on exploring the transferability of learned knowledge of one LLM to enhance the performance of another LM. The key points are as follows:
- **Purpose of E_why:** The E_why serves as a knowledge transfer medium. It encapsulates the reasoning used by the first LLM to arrive at its prediction. 
- **Transferability via E_why:** Feeding E_why to another LLM is not about re-predicting the answer but rather about examining how the explanation influences the understanding and performance of the second LLM. This process tests the effectiveness of the explanation in conveying the reasoning knowledge of the first LLM to another. We investigate the potential of explanations to serve as a knowledge bridge between different models here.
- **Impact on Performance:** Our experiments demonstrate that by utilizing the explanations generated by one LLM, another LLM can enhance its performance (one example is shown below). With the help of explanations, the second LLM increases its accuracy by 2.7%. The different ratio of predicted answers after transferring is around 2%-4%.
The results indicate LLMs can benefit from the reasoning and knowledge encapsulated in the explanations, leading to improved decision-making and understanding.


    | Model                      | Method    | Accuracy |
    |----------------------------------|----------|----------|
    | GPT-3.5                          | Vanilla  | 72.3%    |
    |GPT-3.5 +explanations| Vanilla  | 75.0%    |

Reviewer comment (Experiments):
It is not clear why the RoBERTa-Large model is used as the decision model. Since the final explanations are generated by the GPT-3.5-turbo model, the performance gain may benefit from the external knowledge provided by GPT-3.5-turbo. It is worth considering different types of decision and generator models.

Author response:
**a. We selected RoBERTa-Large for several reasons:**
   - RoBERTa-Large has been widely used in the NLP community and is well-known for its robust performance. Its widespread use in the NLP community makes it a reliable base model for evaluating new methods and approaches. Our explanation quality can be influenced by the performance of the decision model, so we choose a model with good performance as the decision model.
   - Our dataset focuses on explaining the reasoning process of LLMs. There is no human-understandable explanation to demystify the reasoning process of LLMs. RoBERTa is a fundamental model in the LLM family, we believe that it is important to start with such a model to establish a baseline for future research.

**b. Explanation Generation:**
- We chose GPT-3.5 as the generator model because of its advanced language capabilities. While it's true that GPT-3.5-turbo's internal knowledge might contribute to the performance gain, our design tries to minimize its influence. The main advantage of our approach is that it transforms the 'black-box' nature of LLMs into a more transparent and interpretable structure. With our graph modular design, we narrow down the knowledge search space of the LLM, providing a clearer understanding of the model's decision-making process. By using strong constrained instruction and the reason-elements learned from the decision model as our prompt to generator model, we can control the generation content. The generator model is more like a "translator" that translates the model's reasoning into natural, human-understandable language. 

As a pioneer study, we believe that our dataset can be further improved and expanded. We will continue to expand our dataset to include a variety of LLMs in our future work. We hope that our work can inspire more research in this area and contribute to the development of XAI, making it more interpretable, trustworthy and human-centered.

Reviewer comment (Novelty):
The proposed framework seems promising in explaining a model’s decision-making in a human-understandable way. However, it is not clear how much it benefits the dataset construction. Solely using GPT-3.5-turbo or GPT-4 for prompting can facilitate the creation of a dataset with question-answer-explanation triples.

Author response:
While GPT-3.5 or GPT-4 are powerful in generating human-like text, they are not designed to explain the reasoning process of curtain LLMs. The reviewer mentioned question-answer-explanation triples, such task is different from our task. Our task is to explain the reasoning process of curtain LLMs, which is different from the generation task. Without the knowledge of the decision model, the generator model cannot generate explanations that reflect the reasoning process of the decision model. 

Our approach can interpret the reasoning process, and link the reasoning process to a generator model. As mentioned before, the generator model is more like a "translator" that translates the decision model's reasoning into human-understandable language.

Reviewer comment (Experiments):
To what extent the reason-elements extraction with KGs and GAT can enhance the explanation generated by GPT-3.5-turbo? How about solely using GPT-3.5-turbo to generate explanations?

Author response:
First, please refer to the response to **W2** (*Clarification on Our Task*).

Since our task is to explain the reasoning process of curtain LLMs, we need to extract the learning knowledge from the decision model to generate explanations. The use of KGs and GAT for extracting reason-elements ensures that the explanations can reflect the reasoning process of the decision model. KGs provide structured and domain-specific knowledge, which when combined with the GAT, allows us to extract the key reason-elements that contribute to the decision-making process. There are two main advantages of our approach: (1) enhancing the explainability of the decision model's outputs, and (2) reducing the irrelevant knowledge and improving the performance of the decision model.

We want to emphasize that our methodology is universally applicable across a diverse range of LLMs. The foundational elements of our approach, such as the integration of KGs and the interpreting component, are designed with flexibility in mind and are not specific to RoBERTa. These components can be adaptable to the architectures and functionalities of various LLMs.

Reviewer comment (Evaluation):
The biggest confusion I had while reading the paper is whether the paper introduces a model for explainable AI or introduces a dataset for XAI. The paper claims that it does the latter, however the dataset is automatically derived from the outputs of a model itself. For example, the reason set is the output of a graph neural network model which itself is a black-box and can be inaccurate. What is the guarantee that the reason-set faithfully reflects the working of the GAT model and therefore how can we be convinced that a dataset derived from a model can be used to evaluate other XAI models in an unbiased way?

Author response:
**1. (W1) Clarification on Dataset and Faithfulness:**

Our paper primarily introduces a dataset for XAI, and here's how we ensure its relevance and utility:

- **Dataset Construction:** The core contribution of our work is the development of a novel dataset for XAI. The dataset is designed to introduce a way to dive deep into understanding the decision-making of LLMs, transforming a black box into human-understandable explanations. Moreover, our dataset can be helpful for future research in XAI, for example, debugging a model, evaluating its performance, designing human-AI interactions, etc.
- **Clarification on Reason-elements:** The dataset is derived from certain outputs of models. The reason-elements generated by the GAT are not used directly as the ground truth for explainability. Instead, it serves as a starting point for generating more refined and human-centered explanations.
- **Clarification on Faithfulness:** 
  - While GATs can be complex, they are not entirely black-box models. They offer a level of interpretability by explicitly modeling the relationships and relative importance between nodes in a graph. This allows us to trace back and understand why certain nodes are given more prominence in the reasoning process. 
  - The main advantage of our approach is that it transforms the 'black-box' nature of LLMs into a more transparent and interpretable structure. With our graph modular design, we narrow down the knowledge search space of the LLM, providing a clearer understanding of the model's decision-making process. 
  - We conducted comprehensive experiments to validate the faithfulness of our explanations. We included expert evaluation, human evaluation and automated evaluation to assess the quality of our explanations. Our results show that our explanations are with high quality and accuracy. To reduce the bias, we followed the methodology outlined by Hoffman et al. (2018) [1] for cross-human alignment and evaluation questions design. This approach helps to minimize bias and ensure the faithfulness of our explanations. 
- **Broader Applicability Beyond XAI Model Evaluation:** Our dataset is not limited to evaluating XAI models. It has broader applications, such as:
  - **Debugging the model:** Our dataset can be used to debug a model by identifying the reason-elements that are not aligned with human understanding and expectations. 
  - **Training and fine-tuning the model:** Our dataset can be used to train and fine-tune the model to generate more accurate and interpretable explanations. 
  - **As a benchmark for future research:** Our dataset can be used as a benchmark for future research in XAI. It can be used to evaluate the performance of different XAI models and compare their effectiveness.

As a pioneer study, we believe that our dataset can be further improved and expanded. We hope that our work can inspire more research in this area and contribute to the development of XAI, making it more interpretable, trustworthy and human-centered.

---
[1] Hoffman, Robert R., et al. "Metrics for explainable AI: Challenges and prospects." arXiv preprint arXiv:1812.04608 (2018).

Reviewer comment (Evaluation):
A related point is what is the guarantee that the dataset will not favor models that are from the same family as the model used to create the dataset? What is the guarantee that it won’t penalize a model which in reality is a better XAI model but produces explanations which are different from the current dataset.

Author response:
**2. (W2) Clarification on Faithfulness:**
[Similar comment to W1]
This concern is similar to W1. As we have mentioned in W1, our design transforms the 'black-box' nature of LLMs into a more transparent and interpretable structure. With our graph modular design, we narrow down the knowledge search space of the LLM, providing a clearer understanding of the model's decision-making process. The attention mechanism is used to identify the most relevant nodes (reason-elements). The attention weights are not arbitrary, but are the result of a learned process. It reflects the decision-making process of the model.
- **(W2.1) Faithfulness of Reason-elements:** The top-k reason-elements are selected based on the attention weights. These attentions are the most influential in the decision-making process of the model. The faithfulness of these reason-elements is grounded in the GAT's ability to accurately capture and identify the most relevant knowledge in the graph. We acknowledge that the GAT is not perfect, and there is room for improvement. However, we believe that our approach is a step in the right direction towards making LLMs more interpretable and trustworthy. 
- **(W2.2) Contextual Relevance and Model Limitations:** In the examples you mentioned, such as the inclusion of "play_mozart" for a question about music, it's important to recognize that models may sometimes establish connections that, while potentially relevant, might not align with every human interpretation. This is a limitation inherent in current AI models, stemming from their reliance on patterns found in training data. The model's decision to include such elements is based on its learned associations and not on arbitrary selection. While this may not always align with human expectations, it is a reflection of the model's reasoning process. We have added a related analysis in the limitation section. 

    We also want to emphasize that our dataset is not limited to be a benchmark for XAI models. It doesn't necessarily mean we want to penalize a model. Rather, the evaluation should focus on the overall coherence, relevance, and completeness of the explanations in relation to the query and reasoning. The goal is to evaluate whether the model can generate explanations that are meaningful and contextually appropriate, not whether it can replicate specific elements from certain datasets.
    We hope our research can inspire more research in this area and contribute to the development of XAI, making it more interpretable, trustworthy and human-centered.

Reviewer comment (Evaluation):
The reason elements are selected as the top-k elements based on attention scores. However the attention scores are themselves computed by a black-box graph neural network model. What is the guarantee that the reason-elements are faithful and reflect the actual decision making process of the model?

Author response:
**3. (W3) Clarification on Explanation:** 
We understand your concern about the quality of our explanations. As we mentioned before, our reason-elements are derived from KGs and highlighted by the attention mechanism. It serves as a starting point for generating more refined and human-centered explanations. These reason-element can not represent the complete explanation, but are cues to guide the generation process of formulating the complete explanation. The main idea is that we utilize these reason-elements through the constrained generation instruction, we can "translate" the model's reasoning into natural, human-understandable language. 

As for the hallucination problem, it is a very common problem and a big topic in LLMs. We are not designing to remove all the hallucination, but using strong constrained instructions to narrow down the knowledge search space of LLMs, to ensure the trustworthy. Our instruction is to guide the LLMs to generate more relevant and focused explanations, thereby reducing the hallucinated content.

Reviewer comment (Evaluation):
A similar observation applies to the example in Appendix D.4. For the given question: “The people danced to the music, what was the music like for them?” — why would a top-reason element be “play_mozart”? Introducing Mozart for this question is unnecessary and might make the explanation meaningless. Also, if used as a benchmark, if a model does not generate “Mozart” as part of its explanation, why should it be penalized?

Author response:
**(W2.2) Contextual Relevance and Model Limitations:** In the examples you mentioned, such as the inclusion of "play_mozart" for a question about music, it's important to recognize that models may sometimes establish connections that, while potentially relevant, might not align with every human interpretation. This is a limitation inherent in current AI models, stemming from their reliance on patterns found in training data. The model's decision to include such elements is based on its learned associations and not on arbitrary selection. While this may not always align with human expectations, it is a reflection of the model's reasoning process. We have added a related analysis in the limitation section. 

    We also want to emphasize that our dataset is not limited to be a benchmark for XAI models. It doesn't necessarily mean we want to penalize a model. Rather, the evaluation should focus on the overall coherence, relevance, and completeness of the explanations in relation to the query and reasoning. The goal is to evaluate whether the model can generate explanations that are meaningful and contextually appropriate, not whether it can replicate specific elements from certain datasets.
    We hope our research can inspire more research in this area and contribute to the development of XAI, making it more interpretable, trustworthy and human-centered.

Reviewer comment (Evaluation):
My biggest question is the effectiveness of the reason-set as an output of the GAT process. How accurate and effective are they? Because from the examples given in Fig. 1 and the appendix, they are unfortunately not very effective.

Author response:
**1. (Q1) Method Effectiveness:**
We will clarify the effectiveness of our method in the following aspects:
- **Working Process:** The reasoning process of LLMs is complex and difficult to interpret. Our approach integrates KGs to narrow down the knowledge search space of LLMs. Through a more transparent and interpretable graph structure, we can better understand the decision-making process of LLMs. GAT is a typical method to learn the importance of nodes in a graph, and influence the reasoning process of LLMs. However, we acknowledge our limitations in the current design. The quality of KG (which is ConceptNet in our case) and more advanced GNN are important factors that can affect the effectiveness of our method. We believe that our approach and dataset are a step in the right direction towards making LLMs more interpretable and trustworthy.
- **Human-centered Evaluation:** We included expert evaluation, human evaluation and automated evaluation to assess the quality of our explanations. Our evaluation includes eight dimensions, and is evaluated by 50 general users, 3 experts and gpt4/3.5. The results confirm the high quality and accuracy of our explanations. One expert commented that our explanations are "In comparison to prior explanations, these explanations provide a more intuitive understanding of the model's decision-making process. The explanations are cogent, and even in instances of erroneous predictions, the underlying reasoning remains transparent and comprehensible." 
- **Clarification on Accuracy:** Our final explanation is not used to affect the answer selection. The explanations are generated to reflect the decision-making process of the LLM in a human-understandable way. The accuracy of the explanations is also influenced by the LLM's performance. If the LLM's performance is low, the explanations will also be less accurate. However, it doesn't mean that our explanations are not accurate or not helpful. Our explanations can be used to debug the model.

Reviewer comment (Novelty):
Instead of presenting this paper as a dataset generation paper, why not present this model + explanation from GPT-3.5 as a model for XAI? I believe the model itself has merits and analyzing its results could be interesting.

Author response:
**2. (Q2) Emphasize Our Dataset:**
Thank you for recognizing the value of our approach. We agree extending our work as a model for XAI is a promising direction. We appreciate the perspective you've offered. However, there are several key reasons why we emphasize the dataset in our paper:
- **Filling a Crucial Gap in XAI:** Our primary motivation is to address a significant gap in XAI, specifically the lack of comprehensive datasets that can explain the decision-making process of LLMs in a human-understandable way. By focusing on the dataset, we aim to provide a valuable resource that not only demystifies the "black-box" nature of LLMs, but more importantly, can catalyze further research and development in XAI.
- **Urgency and Importance of Our Dataset:** In the rapidly evolving field of AI, the urgency for robust and reliable datasets for XAI is paramount. Our dataset offers a unique and timely contribution that can immediately benefit researchers and practitioners in the field, helping them to develop more transparent, interpretable, and trustworthy AI systems.

Reviewer comment (Experiments):
As per my understanding, the why-not choose part requires the dataset to have multiple-choice question answers and is not generalisable to other scenarios (please correct me if I am wrong).

Author response:
We appreciate the opportunity to clarify the generalizability of the 'why-not choose' of our model.

The 'why-not choose' component of our model is designed to provide explanations for not selecting certain options in a multiple-choice question setting. The intuition behind this design is to provide a more comprehensive explanation for the model's decision-making process. In a multiple-choice question (MCQ) setting, the model's decision is not only based on the correct answer but also on the incorrect options. Therefore, it is important to provide explanations for both the correct and incorrect options. 

While it is true that our current implementation focuses on MCQ scenarios, our approach can be adaptable to other formats. One of the key aspects of our methodology is the comparative analysis of different possible answers or outcomes. This can be applied to scenarios beyond MCQs, such as:

- **Binary or Open-Ended Questions:** By slightly modifying the approach to compare the selected answer with potential alternatives or reasoning paths. For example, in a binary question, the model's decision can be compared with the alternative decision. In an open-ended question, the model's answer can be compared with the counterfactual answer.
- **Decision-Making Scenarios:** Where the model can explain why certain options were not chosen based on a set of criteria or data points.

We acknowledge that extending our 'why-not choose' approach to scenarios beyond MCQs is an interesting area for future research and development. We are actively exploring ways to adapt and apply this methodology to a broader range of question types and decision-making processes. 

It is important to highlight that the primary objective of our dataset is to explain the reasoning behind the LM's decision-making process. Our dataset is designed to enable 
general users to easily understand this process, which can be helpful for future human-centered applications. We believe that our paper makes a substantial contribution to the field by addressing a critical need for transparency and explainability in AI decision-making.

Reviewer comment (Experiments):
What reasoning was used in choosing Roberta-Large as the LLM, M? Why was the masked language model used instead of autoregressive LLMs? Is it extendable to autoregressive models as well?

Author response:
We selected RoBERTa-Large for several reasons:

- RoBERTa is well-known for its robust performance across a variety of NLP tasks. Its widespread use in the NLP community makes it a reliable base model for evaluating new methods and approaches.
- There are no human-understandable explanations for LLMs. RoBERTa is a fundamental model in the LLM family, we believe that it is important to start with such a model to establish a baseline for future research. 
  

Our current dataset primarily focuses on RoBERTa; however, it's important to emphasize that our methodology is universally applicable across a diverse range of LMs. The foundational elements of our approach, such as the integration of KGs and the interpreting component, are designed with flexibility in mind and are not specific to RoBERTa. These components can be adaptable to the architectures and functionalities of various LMs.

Key aspects of our method, such as the extraction and utilization of knowledge from KGs, as well as the generation and validation of explanations, are inherently model-agnostic. This means they can be readily applied to other LLMs with minimal adjustments. As part of our future work, we plan to extend our dataset to include a variety of LMs.

Reviewer comment (Writing):
Can the intuition behind equation 4 in page 6 be explained clearly?

Author response:
This equation is used to normalize scores obtained from our human study. In our human study, we aimed to engage participants effectively and obtain accurate evaluation results. To achieve this, we included a variety of questions, each measuring different dimensions of our dataset. Notably, one of these dimensions was 'irrelevance,' where a lower score indicated a better quality. This is opposite to the other dimensions, where a higher score indicates a better quality. 

We introduced Equation 4 to address this inconsistency and enhance the readability of our results. The primary goal of this normalization was to align all scores on a common scale where a higher value uniformly indicates better quality.

Reviewer comment (Evaluation):
How can the claim that the LLM’s reasoning is very similar to “why” explanations in the dataset since it improves the performance be made? It could merely be providing additional context which helps the LLM to provide better answer.

Author response:
Our model's reasoning is very similar to 'why' explanations in the dataset. This is because our model transforms the 'black-box' nature of LLMs into a more transparent graph structure, leading to reasoning in LLMs that is very similar to "why" explanations.

- **Transformation to Transparent Structure:** The core of our approach lies in converting the opaque reasoning process of LLMs into a more transparent and structured form. This is achieved by integrating the retrieved graph into the LM's reasoning process.
- **Narrowing Down Knowledge Search Space:** By leveraging KG, we effectively narrow down the knowledge search space for the LM. Our approach helps LM focus on the most relevant and contextually appropriate information, reducing irrelevant or inaccurate reasoning.
- **Structured Knowledge for Better Interpretability:** The graph provides structured representations of knowledge, which are inherently more interpretable than unstructured data. We utilize this structured knowledge to help LM align its reasoning process more closely with the "why" explanations.
- **Guiding the LM's Reasoning Process:** The graph not only provides structure but also guides the LM in its reasoning. The LM is constrained to utilize knowledge from the graph, which ensures that its reasoning is grounded in relevant knowledge space. 

---
We are grateful for the opportunity to clarify our work. We hope that our response has addressed your concerns. We look forward to hearing from you.

---

In our paper, we have a **decision model**, and a **generator model**. Our task is to explain the decision-making process of the **decision model**. I am assuming that you are referring to the **generator model** when you say "the LLM". The generator model is used to generate explanations based on the key reason-elements extracted from the decision model. The generator model is not used to make decisions, it is like a **"translator"** that translates the decision model's reasoning process into human-understandable explanations. More specifically, the "why explanation" is reflected how the decision model reasons, it should be "very similar" to the real **decision model's** reasoning, not the **"generator model"**. 

Under our strong constraint, the generator model will NOT reason the question and answer or make decisions. We will add this clarification in the revised version.

Our work aims to make the reasoning process of LLMs **(decision model)** more transparent and interpretable, acknowledging that the explanations are approximations designed to be as faithful as possible to the LLM's **(decision model)** outputs.

We hope this explanation addresses your concerns, and we are grateful for your feedback. Please let us know if you have any further questions or comments, we are happy to address them.