Paper: Are Large Language Models Post Hoc Explainers? (ICLR-2024)

Reviewer comment (Evaluation):
(Pareto Curve) Even if the LLMs are decisively better/worse than the existing explainability methods, the ultimate decision of the practitioner would also be based on how cheap/expensive it is to obtain the explanations. I would be interested to see an approximate cost-benefit tradeoff to have a better sense of whether the LLM-in-the-loop explainers are preferable in practice. The cost here could be with respect to $ compute and with respect to time.

Author response:
**Compute and runtime for generating explanations using LLMs**

Great point! We would like to clarify that, since submission, the **cost of GPT-4 has already reduced 10-fold** and inference times have gone down too. In response to the reviewers’ concern, we provide a detailed table below that compares the runtime for generating explanations using LLMs and existing post hoc explanation methods.

| **Method** |   LR   |   LR  |   LR  |   LR   |   ANN  |   ANN  |   ANN  |   ANN  | **Mean runtime (in secs)** |
|:----------:|:------:|:-----:|:-----:|:------:|:------:|:------:|:------:|:------:|:--------------------------:|
|            | COMPAS | Blood | Adult | Credit | COMPAS | Blood  | Adult  | Credit |                            |
| Grad        |  0.183 | 0.001 | 0.002 |  0.001 |  0.003 |  0.001 |  0.002 |  0.002 |            0.024           |
| SG         |  0.174 | 0.121 | 0.124 |  0.123 |  0.134 |  0.127 |  0.131 |  0.128 |            0.133           |
| IG         |  0.044 | 0.043 | 0.043 |  0.043 |  0.047 |  0.045 |  0.047 |  0.046 |            0.045           |
| ITG        |  0.001 | 0.001 | 0.001 |  0.001 |  0.001 |  0.001 |  0.002 |  0.001 |            0.001           |
| SHAP       |  8.93  | 9.064 | 9.151 |  8.996 |  11.21 | 11.143 | 11.165 | 11.077 |           10.092           |
| LIME       |  2.922 | 1.482 | 0.407 |  0.398 |  3.051 |  1.574 |  0.476 |  0.456 |            1.346           |
| LLM       | 1732 | 1668 | 1418 | 1624 | 1578 | 1313 | 1349 | 1723 | 1550 |


From the above table, we show that the time taken by LLMs to generate explanations is greater than the other explanation methods. However, we would like to highlight that the runtime for generating explanations using LLMs by query OpenAI APIs depends on a range of factors, including time of the day, server/requests overload, rate limit imposed by OpenAI, etc. However, the above runtime is expected to go down as the query time for OpenAI APIs improves.

Reviewer comment (Reproducibility):
(Lack of sufficient experimental details) I find that the presentation of the experiments could be significantly improved. As it stands, there are a lot of missing details in the experiments section or the Appendix, which makes me feel less confident about the reliability of the results. I believe most of these points could easily be addressed in the rebuttal phase, and I will be happy to revise my assessment during the rebuttal.

Author response:
**Lack of experimental details**

We apologize for the lack of clarity and provide additional experimental details below.
*PGI/PGU and FA/RA metrics*

We have added a detailed definition of the four metrics in Section 6.1 of our revised manuscript.


*Hyperparameters for explanation methods*

We followed OpenXAI and used the standard hyperparameters for these explanation methods. In response to the reviewers’ feedback, we detail them below for reference and have added them to the revised manuscript.

**LIME**

kernel_width           = 0.75

std_LIME               = 0.1

mode                   = 'tabular'

sample_around_instance = True

n_samples_LIME         = 1000 or 16

discretize_continuous  = False

**grad**

absolute_value = True

**Smooth grad**

n_samples_SG = 100

std_SG       = 0.005

**Integrated gradients**

method             = 'gausslegendre'

multiply_by_inputs = False

n_steps            = 50

**SHAP**

n_samples = 500

*Reproducibility - what learning rate used to train models etc*

All our models are trained in PyTorch using cross entropy and a class weighted term to encourage underrepresented classes to contribute more to the loss than the most popular class.

Optimizer: Adam

LR: 0.001

Epochs: 100

Data normalization: minmax

Batch size: 64

We have shared all these details in the Jupyter notebook shared in the supplementary material (Notebooks/TrainModels.ipynb)

*What is our parsing strategy and how many bad replies?*

The GPT-4 and GPT-3.5 responses are parseable an average of **96.4%** and **85%** of the time, respectively. We improve parsing ability by encouraging the LLM’s replies to be nicely formatted. This is done by requesting the ranked features to be on the last line in descending order of importance with no other information present. The exact parsing details can be found in our Supplementary Materials code folder **llms > response.py**. 

In general, we use regex and string manipulation to identify the last line of the reply. We then extract key parts of the last line like **“is”**, **“:”**, and **“=”** which indicate that the ranked features will begin on the right side of the matched string. An example reply is as follows: 

*“Therefore, the top five most important features, ranked from most important to least important, are: G, A, F, H, I.*

*Answer: G, A, F, H, I”*

This will be parsed by splitting the last line by **:** into two parts, namely “Answer” and “G, A, F, H, I”. The latter is split by a comma and stripped of any white space to be finally placed into an array for further processing and faithfulness evaluation.

Reviewer comment (Evaluation):
Importantly, I believe the authors should define their evaluation metrics; e.g., for metrics like PGI, PGU, RA, FA the authors refer to earlier papers without explicitly defining what they are. The reader should ideally be able to see the metrics without navigating to different papers.

Author response:
*PGI/PGU and FA/RA metrics*

We have added a detailed definition of the four metrics in Section 6.1 of our revised manuscript.

Reviewer comment (Reproducibility):
Similarly, for reproducibility purposes, the details around the models used should be better provided. E.g. it’s unclear what optimizer is used to train the models, with which learning rate, whether early stopping is applied, and so on. This would surely raise reproducibility issues for follow-up work unless addressed.

Author response:
*Reproducibility - what learning rate used to train models etc*

All our models are trained in PyTorch using cross entropy and a class weighted term to encourage underrepresented classes to contribute more to the loss than the most popular class.

Optimizer: Adam

LR: 0.001

Epochs: 100

Data normalization: minmax

Batch size: 64

We have shared all these details in the Jupyter notebook shared in the supplementary material (Notebooks/TrainModels.ipynb)

Reviewer comment (Reproducibility):
The authors describe the process of parsing the response as `We first save each LLM query’s reply to a text file and use a script to extract the features.` I believe further details are needed to better understand this process. Specifically, what is the existing parsing strategy? Are the responses always parseable? What fraction of the time they are not parseable?

Author response:
*What is our parsing strategy and how many bad replies?*

The GPT-4 and GPT-3.5 responses are parseable an average of **96.4%** and **85%** of the time, respectively. We improve parsing ability by encouraging the LLM’s replies to be nicely formatted. This is done by requesting the ranked features to be on the last line in descending order of importance with no other information present. The exact parsing details can be found in our Supplementary Materials code folder **llms > response.py**. 

In general, we use regex and string manipulation to identify the last line of the reply. We then extract key parts of the last line like **“is”**, **“:”**, and **“=”** which indicate that the ranked features will begin on the right side of the matched string. An example reply is as follows: 

*“Therefore, the top five most important features, ranked from most important to least important, are: G, A, F, H, I.*

*Answer: G, A, F, H, I”*

This will be parsed by splitting the last line by **:** into two parts, namely “Answer” and “G, A, F, H, I”. The latter is split by a comma and stripped of any white space to be finally placed into an array for further processing and faithfulness evaluation.

Reviewer comment (Experiments):
The authors present `LLMs accurately identify the most important feature` as a significant result (e.g. abstract `identify the most important feature with 72.19% accuracy,`), however for this specific task I do not see baselines. Why do the authors have baselines for faithfulness, but not for this specific task (apologies if I’m missing this and the result exists)? Specifically – how good are existing algorithms at identifying the most important feature?

Author response:
**Top-k = 1 experiments (Tables 4 and 5)**

Thank you for reporting the misleading statement. We found that GPT-4 exhibits non-trivial performance for most important feature identification, and performs on par with state-of-the-art methods in certain settings like the Credit dataset (Tables 4 and 5). Comparing GPT-3.5 and GPT-4 (Figure 6) additionally demonstrates that LLMs are closing the performance gap to post hoc explainers. We deemed this a new frontier as high performance is achieved through prompting alone, a novel approach in our field.

In light of this, we have updated the sentence in the abstract to be clearer, reading: `we observe that LLMs identify the most important feature with 72.19% accuracy, indicating promising avenues for further research into LLM-based explanation frameworks within explainable artificial intelligence (XAI).`

We hope these points can help to address the reviewer's remaining concerns, and are happy to provide further clarifications if desired.

Reviewer comment (Evaluation):
I’m slightly confused about the insights we can draw from the experiments. Specifically, the authors propose 3 different explanation strategies that seem to perform reasonably similarly. I understand the overall message that LLMs have the potential to be used as explainers. However, the confusing part to me is there are 3 algorithms presented, and it’s hard to understand which one is better or when. I’d appreciate it if the authors could provide a concise discussion around this.

Author response:
**What are the experimental insights? Why do we have 3 different strategies that perform roughly the same? Concise discussion.**

Great point! We agree with the reviewer that the key takeaway of our study is that LLMs have the potential to be used as a post hoc explainer. The three prompting strategies presented in our work perform similarly at identifying the top k = 1, 2, 3, 4, and 5 features (**see Figure 5**). However, according to Figure 3 and Table 2, Instruction-based ICL is, on average, better at Feature Agreement and Rank Agreement than the other two methods, Perturbation-based and Prediction-based ICL. For instance, Instruction-based ICL outperforms the other two methods in terms of FA and RA for the LR model on the Adult dataset (see Table 2).

To address your second concern about when to use one over another, it seems across the four datasets that **Instruction-based ICL performs better in terms of our metrics.** However, choosing which one to use is nuanced. 

1. Perturbation-based ICL is best for a simpler and more cursory way to identify key features influencing predictions where deeper analysis may not be necessary. 

2. Prediction-based ICL is ideal when the task requires analysis of both the predictive and explanatory components. 

3. Instruction-based ICL is best when a detailed step-by-step analysis of each feature’s impact is necessary. This method provides a more comprehensive understanding of the model’s behavior and feature importance. 


We have included most of the above clarifications and details in the revised version of the manuscript. We hope we addressed all your questions/concerns/comments adequately. In light of these clarifications, we kindly request you to consider increasing your score. We are very happy to answer any further questions.

Reviewer comment (Theory):
The critical weakness, in my view, is that the interpretability method itself is uninterpretable. When the task involves understanding language, this can be somewhat understood, like Bills et al.'s 2023 "Language models can explain neurons in language models." But this paper applies language models to ask them directly, "What are the most important features in determining the model's prediction?" And it does so on purely numerical datasets. It oversimplifies the task of interpretability, arbitrarily modeling logistic regression coefficients as fixed ground truths and using prediction gap -- but there are a million dramatically cheaper ways that they could have selected the most important features according to the same criteria.

Author response:
**Our method is uninterpretable**

While we agree with the reviewer that the inner workings of LLMs are inherently complex and (currently) uninterpretable by nature, it is not the primary focus of our work and is an important point worth exploring in future work. Further, we would like to clarify that the explanations generated by the LLM using our prompting strategy are not *uninterpretable* or *fundamentally flawed*. For example, we specifically instruct the LLM to explain its reasoning (**see the “Instructions” section of the prompt templates on pages 4 and 5**) before reaching a decision on which features are most important to the model. 

We would like to note that identifying key features using LLMs is an interesting future direction as our current exploration shows that they achieve non-trivial performance when compared to some state-of-the-art explanation methods. Further, LLMs provide natural language explanations and a level of textual reasoning that is more accessible/interpretable to end users when explaining a model’s feature space. For instance, LIME performs a specific job that does not adapt to any specific patterns it observes in the perturbations, whereas LLMs are not limited in this respect, and have the capacity to adapt their explanation based on the patterns they see in the local neighborhood, which offers an explanation for why GPT-4 is more sample efficient than LIME with 16 input perturbations.

We thank the reviewer for raising these points as we had not clearly articulated them in the original manuscript.

**Comparison to Bills et al. 2023**

Thank you for your valuable feedback on our paper. We appreciate your insights and the opportunity to address the concerns you raised regarding the interpretability of our proposed method.

Firstly, we acknowledge your point about the interpretability challenge inherent in using Large Language Models (LLMs) as explainers. The complexity and opaqueness of these models do indeed raise valid concerns about the 'black-box' nature of the explanations they generate. However, **our work aims to explore the potential of LLMs in a new domain**, specifically as tools for post hoc explanation in XAI (Explainable Artificial Intelligence). Regarding the application of LLMs to numerical datasets, we believe this represents an innovative step in understanding the capabilities of LLMs beyond their traditional scope. While LLMs are primarily designed for language tasks, **their ability to abstract and generalize can potentially be leveraged in a variety of contexts, including numerical data interpretation.** 

With respect to the oversimplification of experimental setup in our approach to interpreting logistic regression coefficients. It was a deliberate design choice for initial experiments as we aimed to establish a baseline understanding of LLMs before moving on to more complex scenarios.

**Motivation for using LLM instead of other cheaper alternatives**

Thank you for your insightful critique regarding the cost-effectiveness of our approach in selecting the most important features. We understand your concern about there being more straightforward and less resource-intensive methods available for feature selection. Our choice to use Large Language Models (LLMs) for this task was **driven by our desire to explore their potential beyond conventional language processing applications.** The **novelty** of our approach lies in its ability **to integrate contextual understanding and in-context learning (ICL) capabilities of LLMs** to provide richer, more nuanced explanations than what might be achievable through simpler methods. 

While we agree that there are numerous simpler and more cost-effective ways to identify important features, LLMs are useful for the following reasons:

1. Existing methods often **lack the ability to offer detailed, human-like explanations.** LLMs can bridge this gap by generating explanations that are more understandable to humans, which is particularly valuable in fields where interpretability is crucial, such as healthcare or finance. 

2. Generating explanations using LLMs is not just about identifying important features but also about understanding the rationale behind these selections in a manner that simpler methods may not provide. 

3. The ability of LLMs to process and explain complex patterns in data can be particularly beneficial when dealing with intricate, high-dimensional datasets where traditional feature selection methods might struggle.

Reviewer comment (Experiments):
A notable limitation (if the more fundamental questions did not overshadow it) is that the models interpreted are all quite simple, the datasets are themselves simple, and more standard models for tabular data are not considered (but again, this is not the paper's primary limitation).

Author response:
**more standard models for tabular data are not considered**

We would like to clarify a potential misunderstanding here. The primary reason for selecting the specific tabular datasets in our study was to align with the datasets used in OpenXAI, a widely recognized and commonly used benchmark in the field of XAI. OpenXAI provides a standardized framework for evaluating and comparing different XAI methods, ensuring that our results are comparable and relevant to the broader research community. We followed suit and adhered to a recognized benchmark that would allow our work to be directly compared with other studies in the field. This choice was crucial for ensuring the validity and relevance of our findings, especially given the novelty of using LLMs for post hoc explanations in XAI. However, we acknowledge your point that incorporating a broader range of standard models for tabular data would have enriched our analysis, which remains part of our future work.

Reviewer comment (Evaluation):
It seems presumptive to suggest that this approach is better than SHAP, at least without a deeper investigation into where this method outperforms it (and some discussion on why it ostensibly performs almost as poorly as randomly selecting features).

Author response:
**Can you give some examples where SHAP performed worst according to your metrics**

In response to the reviewers’ question, we compare SHAP and LIME explanations generated for a sample from the Recidivism dataset trained using the LR model. The Recidivism dataset has six features (A, B, C, D, E, F).

*SHAP explanation*

The SHAP algorithm generates the feature ranking magnitudes as [0.2977, -0.0028,  0.0040, -0.1067,  0.00004, -0.0026] and the feature ranked by SHAP magnitude: ['A', 'D', 'C', 'B', 'F', 'E'] which represents **['age', 'c_charge_degree_F', 'length_of_stay', 'priors_count', 'race', 'sex_Female']** Intuitively, SHAP outputs individual feature magnitude values and they are generated using a somewhat game theory based algorithm.

*LLM explanations*

In contrast to SHAP and other existing methods, LLMs provide detailed reasoning of the model behavior.

*Response from LLM:*

Feature 'A':

When 'A' increases, the output remains the same (0). When 'A' decreases, the output decreases (-1). This suggests a negative correlation between 'A' and the output.

Importance: 80. 'A' seems to have a strong influence on the output, as a decrease in 'A' consistently leads to a decrease in the output.

**Rank: A**

Feature 'B':

When 'B' increases, the output decreases (-1). When 'B' decreases, the output remains the same (0). This suggests a negative correlation between 'B' and the output.

Importance: 90. 'B' seems to have a stronger influence on the output than 'A', as an increase in 'B' consistently leads to a decrease in the output.

**Rank: BA**

Feature 'C':

When 'C' increases, the output decreases (-1). When 'C' decreases, the output remains the same (0). This suggests a negative correlation between 'C' and the output.

Importance: 70. 'C' seems to have a slightly weaker influence on the output than 'A' and 'B', as changes in 'C' do not consistently lead to changes in the output.

**Rank: BAC**

…

…

…

…


**Final Rank: BACDEF**

Hence, LLMs provide extensive reasoning for the ranked features and identify the correct top 3 feature rank (ground truth for this sample was BACFDE)

*Comparisons and Analysis* 

**Ground truth:** 1. Priors_count, 2. age, 3. length_of_stay

**Top 3 SHAP explanations:** 1. age, 2. c_charge_degree_F, and 3. Length_of_stay

**LLM explanations:** 1. Priors_count, 2. Age, 3. length_of_stay, 

LLM was able to perfectly match the explanations of the LR model SHAP only matched ‘age’ (in the wrong order) and length of stay in the correct position.