Inference to the Best Explanation in Large Language Models (2024)

Abstract: While Large Language Models (LLMs) have found success in real-world applications, their underlying explanatory process is still poorly understood. This paper proposes IBE-Eval , a framework inspired by philosophical accounts on Inference to the Best Explanation (IBE) to advance the interpretation and evaluation of LLMs’ explanations. IBE-Eval estimates the plausibility of natural language explanations through a combination of explicit logical and linguistic features including: consistency , parsimony , coherence , and uncertainty . Extensive experiments are conducted on Causal Question Answering (CQA) , where IBE-Eval is tasked to select the most plausible causal explanation amongst competing ones generated by LLMs (i.e., GPT 3.5 and Llama 2). The experiments reveal that IBE-Eval can successfully identify the best explanation with up to 77% accuracy ( \approx 27\% above random), improving upon a GPT 3.5-as-a-Judge baseline ( \approx+17\% ) while being intrinsically more efficient and interpretable. Additional analyses suggest that, despite model-specific variances, LLM-generated explanations tend to conform to IBE criteria and that IBE-Eval is significantly correlated with human judgment, opening up opportunities for future development of automated explanation verification tools.

## introduction large language models (llms) such as openai's gpt @xcite and llama @xcite have been highly effective across a diverse range of language understanding and reasoning tasks @xcite . while llm performances have been thoroughly investigated across various benchmarks @xcite @xcite , the principles and properties behind their step-wise reasoning process are still poorly understood @xcite . llms are no-toriously black-box and can be difficult to interpret @xcite . moreover, the commercialization of llms has led to strategic secrecy around model architectures and training details @xcite . finally, llms are susceptible to hallucinations and adversarial perturbations @xcite , often producing plausible but factually incorrect answers @xcite . as the size and complexity of llm architectures increase, the systematic study of generated explanations becomes crucial to better interpret and validate the llm's internal inference and reasoning processes @xcite @xcite . the automatic evaluation of natural language explanations presents several challenges @xcite . without resource-intensive annotation @xcite @xcite , explanation quality methods tend to rely on either weak supervision, where the identification of the correct answer is taken as evidence of explanation quality, or require the injection of domain-specific knowledge @xcite . in this paper, we seek to better understand the llm explanatory process through the investigation of explicit linguistic and logical properties. while explanations are hard to formalize due to their open-ended nature, we hypothesize that they can be analyzed as linguistic objects, with measurable features that can serve to define criteria for assessing their quality. specifically, this paper investigates the following overarching research question: "can the linguistic and logical properties associated with llmgenerated explanations be used to qualify the models' reasoning process?". to this end, we propose an interpretable framework inspired by philosophical accounts of abductive inference, also known as inference to the best explanation (ibe) -i.e. the therefore, since the balloon was pricked, it may have deflated, resulting in a decrease in air pressure inside the balloon, causing the external air pressure to make the balloon expand causal question the balloon expanded. what was the cause? a) i blew into it. b) i pricked it. explanation prompt for the provided scenario, identify which option is the most plausible cause of the context. let's think stepby-step and generate an explanation for each option. treat each option as the premise and the provided context as the conclusion. generate a short step-bystep logical proof that explains how the premise can result in the conclusion. process of selecting among competing explanatory theories @xcite . in particular, we aim to measure the extent to which llm-generated explanations conform to ibe expectations when attempting to identify the most plausible explanation. to this end, we present ibe-eval, a framework designed to estimate the plausibility of natural language explanations through a set of explicit logical and linguistic features, namely: logical consistency, parsimony, coherence, and linguistic uncertainty. to evaluate the efficacy of ibe-eval, we conduct extensive experiments in the multiple-choice causal question answering (cqa) setting. the overall results and contributions of the paper can be summarized as follows: for reproducibility, our code is made available on github foot_0 to encourage future research in the field. ## inference to the best explanation (ibe) explanatory reasoning is a distinctive feature of human rationality underpinning problem-solving and knowledge creation in both science and everyday scenarios @xcite . accepted epistemological accounts characterize the creation of an explanation as composed of two distinct phases: conjecturing and criticism @xcite . the explanatory process always involves a conflict between plausible explanations, which is typically resolved through the criticism phase via a selection process, where competing explanations are assessed according to a set of criteria such as parsimony, coherence, unification power, and hardness to variation @xcite @xcite @xcite @xcite . as llms become interfaces for natural language explanations, epistemological frameworks offer an opportunity for developing criticism mechanisms to understand the explanatory process underlying state-of-the-art models. to this end, this paper considers an llm as a conjecture device producing linguistic objects that can be subject to criticism. in particular, we focus on a subset of criteria that can be computed on explicit linguistic and logical features, namely: consistency, parsimony, coherence, and uncertainty. to assess the llm's alignment to such criteria, we focus on the task of selecting among competing explanations in a multiple-choice cqa setting (figure 1 ). specifically, given a set of competing hypotheses (i.e. the multiple-choice options), @xmath0, h 2 , . . . , h n }, we prompt the llm to generate plausible explanations supporting each hypothesis (section 3). subsequently, we adopt the proposed ibe selection criteria to assess the quality of the generated explanations (section 4). ibe-eval computes an explanation plausibility score derived from the linear combination of the computed selection criteria. the explanation with the highest score is selected as the predicted answer and additionally assessed as to the extent to which the observable ibe features are correlated with qa accuracy. we hypothesize that ibe-eval will produce higher scores for the explanation associated with the correct answer and that the ibe criteria should meaningfully differentiate between competing explanations. ## explanation generation for the first stage, the llm is prompted to generate competing explanations for the hypotheses using a modified chain-of-thought (cot) prompt @xcite . specifically, the cot prompt is modified to instruct the llm to produce an explanation for each competing hypothesis (see figure 1 ). we adopt a methodology similar to @xcite , where the generated explanation is constrained into an entailment form for the downstream ibe evaluation. in particular, we posit that a valid explanation should demonstrate an entailment relationship between the premise and conclusion which are derived from the question-answer pair. to elicit logical connections between explanation steps and facilitate subsequent analysis, the llm is constrained to use weak syllogisms expressed as if-then statements. additionally, the llm is instructed to produce the associated causal or commonsense assumption underlying each explanation step. this output is then post-processed to extract the explanation steps and supporting knowledge for evaluation via the ibe selection criteria. additional details and examples of prompts are reported in appendix a.2. ## linguistic & inference criteria to perform ibe, we investigate a set of criteria that can be automatically computed on explicit logical and linguistic features, namely: consistency, parsimony, coherence, and uncertainty. consistency. consistency aims to verify whether the explanation is logically valid. given a hypothesis, comprised of a premise p i , a conclusion c i , and an explanation consisting of a set of if-then statements @xmath1, ..., s i , we define e to be logically consistent if p i ∪ e ⊨ c i . specifically, an explanation is logically consistent if it is possible to build a deductive proof linking premise and conclusion. to evaluate logical consistency, we leverage external symbolic solvers along with autoformalization -i.e., the translation of natural language into a formal language @xcite . specifically, the hypotheses and explanations are formalized into a prolog program which will attempt to generate a deductive proof via backward chaining @xcite . to perform autoformalization, we leverage the translation capabilities of gpt 3.5. specifically, we instruct gpt 3.5 to convert each if-then explanation step from the generated explanation into an implication rule and the premise statement into grounding atoms. on the other end, the entailment condition and the conclusion are used to create a prolog query. the query instructs the prolog solver to attempt to find a path through the implications rules such that the conclusion be directly connected to the premise. further details about the autoformalization process can be found in appendix a.3. after autoformalization, following recent work on neuro-symbolic integration for llm explanations @xcite , we adopt an external prolog solver for entailment verification 2 . the explanation is considered consistent if the prolog solver can satisfy the query and successfully build a deductive proof. technical details can be found in appendix a.5. parsimony. the parsimony principle, also known as ockham's razor, favors the selection of the simplest explanation consisting of the fewest elements and assumptions @xcite . epistemological accounts posit that an explanation with fewer assumptions tends to leave fewer statements unexplained, improving specificity and alleviating the infinite regress @xcite . further, parsimony is an essential feature of causal interpretability, as only parsimonious solutions are guaranteed to reflect causation in comparative analysis @xcite . in this paper, we adopt two metrics as a proxy of parsimony, namely proof depth, and concept drift. proof depth, denoted as depth, is defined as the cardinality of the set of rules, r, required by the prolog solver to connect the conclusion to the premise via backward chaining. let h be a hypothesis candidate composed of a premise p and a conclusion c, and let e be a formalized explanation represented as a set of rules r ′ . the proof depth is the number of rules |r|, with r ⊆ r ′ , traversed during backward chaining to connect the conclusion c to the premise p: depth(h) = |r| concept drift, denoted as drif t, is defined as the number of additional concepts and entities, outside the ones appearing in the hypothesis (i.e., premise and conclusion), that are introduced by the llm to support the entailment. for simplicity, we consider nouns as concepts. let @xmath2, n oun c , n oun e } be the unique nouns found in the premise, conclusion, and explanation steps. concept drift is the cardinality of the set difference between the nouns found in the explanation and the nouns in the hypothesis: intuitively, the parsimony principle would predict the most plausible hypothesis as the one supported 2 https://github.com/neuro-symbolic-ai/ explanation_based_ethical_reasoning by an explanation with the smallest observed proof depth and concept drift. implementation details can be found in appendix a.6. coherence. coherence attempts to measure the logical validity at the level of the specific explanation steps. an explanation can be formally consistent on the surface while still including implausible or ungrounded intermediate assumptions. coherence evaluates the quality of each intermediate if-then implication by measuring the entailment strength between the if and then clauses. to this end, we employ a fine-tuned natural language inference (nli) model. formally, let s be a set of explanation steps, where each step s consists of an if-then statement, @xmath3, t hen s ). for a given step s i , let es(s i ) denote the entailment score obtained via the nli model between if s and t hen s clauses. the step-wise entailment score sw e(s) is then calculated as the averaged sum of the entailment scores across all explanation steps |s|: we hypothesize that the llm should generate a higher coherence score for more plausible hypotheses, as such explanations should exhibit stronger step-wise entailment. additional details can be found in appendix a.7. uncertainty. finally, we consider the linguistic certainty expressed in the generated explanation as a proxy for plausibility. hedging words such as probably, might be, could be, etc typically signal ambiguity and are often used when the truth condition of a statement is unknown or improbable. @xcite found that the strength of scientific claims in research papers is strongly correlated with the use of direct language. in contrast, they found that the use of hedging language suggested that the veracity of the claim was weaker or highly contextualized. to measure the linguistic uncertainty (u c) of an explanation, we consider the explanation's underlying assumptions (a i ) and the overall explanation summary (s). the linguistic uncertainty score is extracted using the fine-tuned sentencelevel roberta model from @xcite . the overall linguistic uncertainty score (u c overall ) is the sum of the assumption and explanation summary scores: where u c(a) is the sum of the linguistic uncertainty scores (u c(a)) across all the assumptions |a| associated with each explanation step i: and linguistic uncertainty of the explanation summary u c(s). we hypothesize that the llm will use more hedging language when explaining the weaker hypothesis resulting in a higher uncertainty score. further details can be found in appendix a.8. ## experimental setting causal question-answering (cqa) requires reasoning about the causes and effects given an event description. we specifically consider the task of cause and effect prediction in a multiple-choice setting, where given a question and two candidate answers, the llm must decide which is the most plausible cause or effect. causal reasoning is a challenging task as the model must both possess commonsense knowledge about causal relationships and consider the event context which would make one option more plausible than the other. for our experiments, we use the choice of plausible alternatives (copa) @xcite and e-care @xcite datasets. copa. copa is a multiple-choice commonsense causal qa dataset consisting of 500 train and test examples that were manually generated. each multiple-choice example consists of a question premise and a set of answer candidates which are potential causes or effects of the premise. copa is a well-established causal reasoning benchmark that is both a part of superglue @xcite and the calm-bench @xcite . ## preliminary analysis we conduct a preliminary analysis as a sanity check to measure the extent to which llms generate selfevident or tautological explanations -i.e., explanations that simply restate the premises and conclusions. tautological explanations present a risk for ibe-eval as the metrics would be theoretically uninformative if the llm adopts the tested causal relation as the explanation itself (e.g. @xmath4. we consider the parsimony metric to compute the percentage of explanations with proof depth equal to 1 (i.e, explanations containing only one inference step) and concept drift equal to 0 (i.e. no additional concepts other than the ones stated in premises and conclusions appear in the explanation). in such cases, the llm is effectively generating a self-evident or tautological explanation. we found that about 2% of the cases consist of self-evident explanations. for gpt 3.5, llama 2 13b, and llama 2 7b, 2% of the generated explanations exhibit a concept drift of 0, and on average 1.5% of the explanations have a proof depth of 1. we then conducted an error analysis to evaluate the cases where ibe-eval selected a self-evident explanation as the best one. across all llms, less than 0.1% of the errors were caused by the selection of such explanations. our analysis suggests that the impact of self-evident explanations is not sig-nificant and that the ibe framework can be robustly applied to identify such cases. ## results to assess the llm's alignment with the proposed ibe framework and evaluate the efficacy of ibe-eval, we run a regression analysis and conduct a set of ablation studies to evaluate the relationship between ibe and question accuracy. the main results are presented in figure 2 and our regression analysis finds that the ibe criteria are generally consistent across the llms as demonstrated by similar correlation patterns found on both the copa and e-care tasks (figure 2 ). gpt 3.5 exhibits the strongest alignment with ibe expectations as we observe nearly all the ibe criteria have statistically significant and directionally aligned correlations across both tasks. thus our proposed ibe criteria can serve as promising build blocks for future work on automated explanation evaluation. in next, we explore each explanation feature in further detail to better understand the variances across the ibe criteria and llms. consistency. we find that the llms are surprisingly strong conjecture models. the llms can generate logically consistent explanations for any hypothesis as observed by similar consistency scores for correct and incorrect (figure 3 ) explanations. moreover, we observe that consistency tends to be a statistically insignificant predictor for the llama models. therefore, we conclude that evidence of logical consistency provides a limited signal for plausibility and is better understood in the context of other ibe criteria. for the incorrect candidate explanations, we find that llms overrationalize and introduce additional premises to demonstrate entailment in their explanations. parsimony. the results suggest that parsimony has a more consistent effect and is a better predictor of explanation quality. we observe negative correlations between proof depth, concept drift, and question-answering accuracy, suggesting that llms tend to introduce more concepts and explanation steps when explaining less plausible hypotheses. on average, we found the depth and drift to be 6% and 10% greater for the incorrect option across all llms (figure 4 ). moreover, the results suggest that as the llm parameter size increases, the tendency to over-rationalize increases as well. this is attested by the fact that the average difference in depth and drift is the greatllms are strong rationalizers and can generate logically consistent explanations at equal rates for explanations associated with both correct and incorrect answers options. figure 4 : explanation parsimony is evaluated using proof depth and concept drift. both metrics are consistently lower for explanations supporting the correct answers suggesting that llms are able to generate efficient explanations for the more plausible hypothesis. coherence. similarly to parsimony, we found coherence to be a better indicator of explanation quality being statistically significant for both gpt 3.5 and llama 2 13b on copa and both llama 2 models on e-care. we found that the average coherence score is consistently greater for the stronger hypothesis across all llms and datasets (see figure 5 ). both gpt and llama 2 13b exhibit a higher relative difference between the correct and incorrect hypotheses in contrast to llama 2 7b. uncertainty. the results reveal that linguistic uncertainty is the strongest predictor of explanation quality and is a statistically significant feature for all llms. this suggests that llms use more qualifying language when explaining weaker hypotheses (see figure 6 ). we found that uncertainty can improve accuracy by 13pp on copa and 4pp on e-care. we also examine the uncertainty cues expressed by llms by analyzing both the frequency of hedge words and the types of hedge cues employed in incorrect explanations. we find the distribution of hedge cues across llms tends to be similar, with only minor differences be-tween llms (figure 6 ). epistemic cues were most frequently used by all three models, with llama 2 7b being more likely to use conditional cues. see appendix a.8 for further details.. ## related work explorations of llm reasoning capabilities across various domains (e.g. arithmetic, commonsense, planning, symbolic, etc) are an emerging area of interest @xcite . prompt-based methods @xcite @xcite , such as cot, investigate strategies to elicit specific types of reasoning behavior through direct llm interaction. @xcite investigate automatic proof generation and propose a neurosymbolic framework with an llm semantic parser and external solver. @xcite propose an inference framework where the llm acts as both a selection and inference module to produce explanations consisting of causal reasoning steps in entailment tasks. research on llm faithfulness @xcite investigates if llm explanations are robust to spurious input alterations. @xcite propose a self-consistency measure cc-shap which measures how specific alterations to a model's input contribute to the generated explanation. this paper primarily draws inspiration from recent work on the evaluation of natural language explanations @xcite @xcite @xcite . however, differently from previous methods that require extensive human annotations or specific domain knowledge, we are the first to propose a set of criteria that can be automatically computed on explicit linguistic and logical features. ## conclusion this paper proposed ibe-eval, an interpretable framework for llm explanation evaluation inspired by philosophical accounts of inference to the best explanation (ibe). ibe-eval can identify the best explanation supporting the correct answer with up to 77% accuracy in cqa scenarios, improving upon a gpt 3.5 judge baselines by +17%. our regression study suggests that llm explanations tend to conform to ibe expectations and that ibe-eval is strongly correlated with human judgment. linguistic uncertainty is the stronger ibe predictor for explanation quality closely followed by parsimony and coherence. however, we also found that llms tend to be strong conjecture models able to generate logically consistent explanations for less plausible hypotheses, suggesting limited applicability for the logical consistency criterion in isolation. we believe our findings can open new lines of research on external evaluation methods for llms as well as interpretability tools for understanding the llm's underlying explanatory process.