Counterfactual Simulatability of LLM Explanations for Generation Tasks (2025)

Abstract: LLMs can be unpredictable, as even slight alterations to the prompt can cause the output to change in unexpected ways. Thus, the ability of models to accurately explain their behavior is critical, especially in high-stakes settings. Counterfactual simulatability measures how well an explanation allows users to infer the model’s output on related counterfactuals and has been previously studied for yes/no question answering. We provide a general framework for extending this method to generation tasks, using news summarization and medical suggestion as example use cases. We find that while LLM explanations do enable users to better predict their outputs on counterfactuals in the summarization setting, there is significant room for improvement for medical suggestion. Furthermore, our results suggest that evaluating counterfactual simulatability may be more appropriate for skill-based tasks as opposed to knowledge-based tasks.

## introduction while large language models (llms) have proven effective for a diverse range of applications, their outputs still often contain hallucinations of unsupported information @xcite or biases @xcite ) that hinder their reliability on critical language generation tasks. at the same time, it is important that users themselves can accurately evaluate the model capabilities, knowledge, and reliability @xcite . particularly in high stakes domains, such as medical applications, misunderstandings or the lack of ability to predict model behavior on unseen inputs can pose disastrous risks to users @xcite . to anticipate these risks, recent work has turned attention to evaluating the reliability of llm explanations @xcite @xcite . in particular, @xcite evaluates the counterfactual simulata-bility of llm explanations for yes/no question answering. counterfactual simulatability is a measure of how well a model's explanation allows humans to correctly infer the model's predictions on simulatable counterfactuals (unseen inputs where the explanation should enable the user to confidently guess the model's output). furthermore, according to @xcite , the counterfactual simulatability of an explanation can be decomposed into simulation generality, a measure of the diversity of simulatable counterfactuals and simulation precision, a measure of the proportion of these counterfactuals for which humans correctly infer the model's output. ideal model explanations should balance both generality and precision. counterfactual simulatability, however, is equally critical to generation tasks, where the larger space of possible outputs makes understanding a model's decision process more challenging. to fill this gap, we formalize a framework for evaluating counterfactual simulatability in language generation tasks (figure 1 ). to evaluate a model's explanation, we first use a separate llm to (1) decompose the explanation into atomic units and (2) generate relevant counterfactuals. then, an annotator (human or the llm) evaluates the simulatability and precision of each unit of the model's explanation using the counterfactual and the model's output on the counterfactual respectively. finally, generality and precision scores are calculated following §4.4. rather than evaluating the factual correctness of explanations, our framework measures the ability of llms to accurately describe their behavior in a generative setting. we evaluate whether llms' explanations lead to reliable human mental models that are consistent with their outputs. we apply the framework to two tasks: news summarization using cnn/dm @xcite and medical suggestion generation using the taiwan e-hospital dataset @xcite . these tasks involve different trade-offs between generality and precision. explanations for news summarization can be highly general as models may employ similar approaches for documents that contain common elements (e.g., dates, quotes) but differ in content. in contrast, medical suggestion explanations may be less general as the model's response can be highly dependent on the user input (i.e., minor variances in expressed symptoms can lead to different suggestions). this can lead to a more limited set of simulatable counterfactuals, though potentially being more precise in its outputs. with these generation tasks, we conduct an initial evaluation of llms' explanations and assess where models' explanations in these generation tasks fail. in summary, our contributions are as follows: ## counterfactual simulatability for generation tasks explanations can be evaluated by considering whether an observer, having seen a model's explanation for some input, can infer (i.e., simulate) the model's output on a related counterfactual input. for instance, if a user asks "do dolphins swim?" and a model answers "yes" with the explanation "all aquatic animals swim", then the user would infer that when asked the counterfactual "do starfish swim?", the model will similarly answer "yes". if, in reality, the model answers "no" to this question, then, as @xcite notes, the explanation is ineffective because it creates a mental model that is inconsistent with the model's behavior, even though the answer may be factually correct. more specifically, counterfactual simulatability measures how accurate these mental models are on simulatable counterfactuals, unseen inputs where the explanation allows the user to confidently predict the model's output @xcite . however, in contrast to the classification example, generation tasks have a much larger output space (e.g., there exists many possible summaries for a news document). this makes simulation extremely challenging as it is impossible to precisely identify a single possible output based on an explanation. for instance in news summarization, if a user is shown the explanation "the summary should include the key event" and a counterfactual document on "the opening ceremony of the 2024 paris olympics", while the user can logically infer that the summary will include the opening ceremony, it is impossible to predict the exact wording in which it will appear. explanations typically cannot enable humans to pinpoint a single model output. however, they are still very useful if they help humans narrow down the possible outputs (e.g., refining "all possible summaries" to "summaries that mention the opening ceremony"). ## limitations first, the generality metric relies on our llm (gpt-4 turbo) being able to generate diverse, yet simulatable counterfactuals. although we briefly experimented with counterfactual generation, more work can be done to assess whether other models or prompting strategies may lead to higher quality counterfactuals. additionally, cosine similarity might not be the optimal way to judge diversity across different tasks as it is heavily content-based. for example, while many counterfactuals for summarization followed the same structure (e.g., a headline followed by when/where it occurred followed by a quote ...), they were still rewarded with high generality because the specific topics and words differed. second, in a few cases there was ambiguity when determining whether atomic units were 'present'in the counterfactual/the counterfactual output. since these units of information may appear with different wording or even be implied, we encouraged annotators to use their best judgement and leave notes. we leave investigation into ways of improving this matching to future work. finally, we focused on demonstrating our frame-work using explanations generated from a few popular llms. testing newer types of models (e.g., reasoning models) and different task setups (e.g., multiple interactions) are a promising direction. furthermore, expanding the set of metrics to consider aspects such as explanation comprehensiveness, faithfulness, and plausibility, and assessing how they vary with counterfactual simulatability, may also be worthwhile. supplementary materials availability statement: our source code as well as an anonymized version of the data used in the human evaluation is made available at github.com/mlimpijankit/ counterfactual-simulatability-generatio n-tasks. 8 acknowledgments this research is supported in part by the office of the director of national intelligence (odni), intelligence advanced research projects activity (iarpa), via the hiatus program contract #2022-22072200005 and the national science foundation and by dod ousd (r&e) under cooperative agreement phy-2229929 (the nsf ai institute for artificial and natural intelligence). the views, opinions and/or findings expressed are those of the author and should not be interpreted as representing the official views or policies of the department of defense, the national science foundation or the u.s. government. one of the authors holds an equity interest in openai.