Are self-explanations from Large Language Models faithful? (2024)

Abstract: Instruction-tuned Large Language Models (LLMs) excel at many tasks and will even explain their reasoning, so-called self-explanations. However, convincing and wrong self-explanations can lead to unsupported confidence in LLMs, thus increasing risk. Therefore, it’s important to measure if self-explanations truly reflect the model’s behavior. Such a measure is called interpretability-faithfulness and is challenging to perform since the ground truth is inaccessible, and many LLMs only have an inference API. To address this, we propose employing self-consistency checks to measure faithfulness. For example, if an LLM says a set of words is important for making a prediction, then it should not be able to make its prediction without these words. While self-consistency checks are a common approach to faithfulness, they have not previously been successfully applied to LLM self-explanations for counterfactual, feature attribution, and redaction explanations. Our results demonstrate that faithfulness is explanation, model, and task-dependent, showing self-explanations should not be trusted in general. For example, with sentiment classification, counterfactuals are more faithful for Llama2, feature attribution for Mistral, and redaction for Falcon 40B.

## introduction instruction-tuned large language models (llms), such as llama2 @xcite , falcon @xcite , @xcite , or gpt4 @xcite , are increasingly becoming mainstream among the general population, due to their capabilities and availability. instruction-tuned llms can even provide very convincing explanations for their utterances and will often do so unprompted. because llms produce these explanations themselves and they provide justification for their own behavior, we term them self-explanations. importantly, one may also judge the ethicality of models not just based on session 1 (prediction and explanation) is the following candidate a good fit for a senior swe position? answer only yes/no. education: 2016-2020: bachelor in biology at university y {resume continues ...} ## interpretability-faithfulness of self-explanations this section describes the general methodology we propose and its principles, using counterfactuals as an example. section 4 then goes into detail on the different kinds of explanations and datasets where this approach is possible and how we precisely formulate the prompts. ## prompt-patterns for self-explanation and self-consistency checks this section provides the specific details and considerations behind the prompts used in the experiment section. specifically, the prompts that generate the counterfactual, feature attribution, and redaction explanations. as well as the prompt for generating the classification. to facilitate the "specific details," we use sentiment classification as an example. however, the methodology generalizes to multi-choice q&a datasets (e.g., babi) and multi-paragraph classification (e.g., nli). prompts for those tasks are in appendix e.2 and appendix e.3, respectively. all prompts are used with chat-tuned models, as these models are typically deployed and provide a well-defined framing mechanism between input and output 1 . such framing helps to avoid some out-of-distribution issues compared to purely instruction-tuned models. however, the prompts only convey a one-time input-output relation and thus could be used with purely instruction-tuned models given the proper framing. it should be noted that it's impossible to prove that the model understands the user's intent. however, it's worth prioritizing the user's intent rather than the model's understanding when evaluating the potential impact of deployed models. ## results to evaluate prompt sensitivity, all the prompt variations from section 4 (details in appendix e) are evaluated using the llama2-70b model fine-tuned for chatting @xcite . in addition, we use a default case to show differences across model types and sizes by evaluating on: mistral v0.1 (7b) @xcite , falcon (7b and 40b) @xcite , and llama2 (7b and 70b) @xcite . model details are in appendix b, with additional details (e.g., generation configuration) in appendix c. we perform all experiments with sentiment classification (imdb), multi-choice q&a tasks (babi-1 and mctest), and an entailment/nli task (rte), details of which are in appendix a. the prompts for these datasets are slightly different; see appendix e. the variability amoung the datasets, demonstrates that our methodology generalizes to both paragraph-based multi-choice questions and two-paragraph tasks. in all cases, the main paragraph is modified through the explanation. the questions, choices, or hypotheses are not modified. ## related work using self-consistency checks to measure faithfulness is not a new idea. one of the earliest uses was with feature attribution, where allegedly important information was removed from the input, and it was then checked that this removal altered the prediction @xcite @xcite . these earlier works used an external explanation algorithm rather than self-explanations. another important difference is that the classification was not discrete (e.g., posi-tive or negative sentiment) but provided logits. the logit values were then measured before and after information removal, and the difference provides a more nuanced faithfulness metric. the self-consistency idea has also been adopted for self-explanations. @xcite , applied the same idea to self-explanation feature attribution from chatgpt on a sentiment classification task (a 100 observation subset of @xcite ). because chatgpt does not provide logits, they instruct chatgpt to provide a confidence score between 1 and 0 for either positive or negative sentiment. similarly, they instruct chatgpt to provide scores for the importance of each word. using this approach, previous methods can in principle be easily applied. unfortunately, they find their approach ineffective as the confidence and importance scores are not calibrated, and chatgpt resists classifying as positive or negative sentiment when information is missing. for this reason, they call for better faithfulness metrics targeted llm self-explanations. in contrast, our approach does not depend on any scores (confidence or importance). instead, the models perform discrete classification and can predict "unknown". this solves the aforementioned issues identified by @xcite . self-consistency checks have also been used to measure the faithfulness of mathematical counterfactuals. in particular, lanham et al. (2023) applies self-consistency checks to chain-of-thought (cot) self-explanations from llms, where the cots provide mathematical context (e.g. "2 + 3 = 5") to a mathematical question. @xcite then edits the cot to contain false information (e.g. "2 + 3 = 6") and checks that the prediction follows. the issue here is that injecting false facts may create out-of-distribution results or be interpreted as typos by the llm, thus it's unclear if this method is completely valid. regardless, they find similar to our paper, that faithfulness is model and taskdependent. ## conclusion our investigation reveals that self-explanations' faithfulness is highly model and dataset-dependent. this conclusion is similar to previous works @xcite @xcite . our contribution is the ability to measure faithfulness on llms' self-explanations, specifically counterfactuals, feature attribution, and redaction explanations. the task dependence is concerning as it means llm self-explanations cannot generally be trusted. there is also no reason to trust more free-form explanations where faithfulness can not be evaluated using self-consistency checks. this increases the risk with llms, as individuals may have the misconception that llms can explain themselves @xcite .