Explanations explained. Influence of Free-text Explanations on LLM s and the Role of Implicit Knowledge (2025)

Abstract: In this work, we investigate the relationship between the quality of explanations produced by different models and the amount of implicit knowledge the are able to provide beyond the input. We approximate explanation quality via accuracy on a downstream task with a standardized pipeline (GEISER) and study its correlation with three different association measures, each capturing different aspects of implicitness, defined as a combination of relevance and novelty. We conduct experiments with three SOTA LLMs on four tasks involving implicit knowledge, with explanations either confirming or contradicting the correct label. Our results demonstrate that providing quality explanations consistently improves the accuracy of LLM predictions, even when the models are not explicitly trained to take explanations as input, and underline the correlation between implicit content delivered by the explanation and its effectiveness.

## introduction large language models (llms) excel at numerous language processing tasks, including text generation, translation, and question answering @xcite . still, understanding their reasoning is challenging, hindering trust and adoption in high-stakes domains @xcite @xcite . one approach towards "intrinsic explainability" is to have llms generate explanations for their predictions. existing methods, like pipeline models @xcite and self-rationalizing models @xcite , often focus on extractive rationales suitable for information extraction @xcite . however, complex reasoning tasks require free-text explanations, especially when implicit knowledge is involved @xcite . also, generating explanations raises concerns about their faithfulness, as llms might produce plausiblesounding explanations with no genuine connection to their reasoning @xcite . this is particularly problematic for implicit knowledge, which relies on the model's internal representations of the world @xcite . with the rise of retrieval-augmented generation (rag, @xcite ), language models are increasingly supplemented with external information, such as explanations, retrieved from knowledge bases or provided via in-context learning (icl). the effectiveness of these approaches depends on the quality of the retrieved or injected text, which serves as additional context for the model's reasoning. while traditional rag studies focus on improving retrieval mechanisms (e.g., optimizing factual correctness), less attention has been paid to evaluating the quality of explanations used in these frameworks. recent work by @xcite shows that augmenting icl with natural language explanations (nles) improves model robustness. however, their study focuses on performance benefits rather than the quality of different explanation types, and their evaluation is limited to downstream accuracy without assessing what makes an explanation effective in guiding a model's decision. our work addresses this gap by providing a principled evaluation of explanation quality, particularly in sentence pair reasoning tasks, measured by downstream task performance. moreover, we show that explanation effectiveness correlates with the degree of implicit content, i.e., novel yet relevant information they provide. we test this hypothesis by examining the relationship between explanation effectiveness and three metrics approximating novelty and relevance, and show that they have high, yet different correlation with explanation quality according to the examined task. the main contributions of this paper are: the role of explanations in nlp has been extensively studied. for instance, cambria et al. (2023) provide a comprehensive survey of natural language explanation generation approaches, and hartmann and sonntag (2022) examine the benefits of explanations for improving nlp models. paranjape et al. (2021) focus on template-based explanations, while lampinen et al. (2022) and ye and durrett (2022) highlight the advantages of incontext explanations for complex reasoning tasks. jansen et al. (2016) provide a comprehensive characterization of different kinds of explanations, each one with different insight into model behavior. traditionally, explanation quality has been assessed using automated metrics like bleu @xcite , rouge @xcite , or bert-score @xcite , which compare outputs to human-written references. however, these metrics may not fully capture explanation quality or align with human judgment, and collecting human references is often costly. more recently, (human) simulatability scores have emerged as an alternative to overlap metrics, based on the idea that explanation quality can be defined as the "utility to an end-user" @xcite . this approach evaluates how explanations improve predictive performance on downstream tasks rather than overlap with ground truth explanations and, while humans were initially the predictors @xcite , trained models now automate this process, showing strong correlations with human judgments @xcite . for example, @xcite measures explanation quality by training a student model on teacher-generated explanations for downstream tasks. prior work has largely focused on eliciting explanations from models or evaluating them based on task performance, our work shifts the focus toward understanding how explanations can reveal implicit knowledge, offering a novel perspective on explanation quality assessment. while, to the best of our knowledge, there are no previous works addressing implicit content measures directly, in the context of information retrieval, relevance and novelty have been recognized as key aspects of novelty detection tasks @xcite , and similarly to us exploit textual entailment @xcite for sentence level novelty mining. metrics such as the cosine similarity between high-dimensional embeddings has been traditionally used to quantify semantic similarity of texts, but has also been recently questioned as a faithful representation @xcite . other works, on the other hand, have focused on estimating the causal strength between textual fragments, and proposed learned metrics such as ceq @xcite or cesar @xcite , in the attempt to improve more simplistic yet effective metrics such as pointwise mutual information (pmi). ## methodology we address the problem of explaining the semantic relationship between two textual fragments under the assumption that the relationship involves implicit knowledge, and the hypothesis that explanations eliciting more implicit knowledge represent higher-quality explanations. ## explanation types we test two different modes of explanation generation: explanations that confirm the given relationship between s 1 and s 2 , explaining why it holds (why, gold) and explanations that potentially contradict the relationship between s 1 and s 2 explaining the circumstances when the relationship may not hold (counterfactual). why explanations. this kind of machinegenerated explanation (why) is the most typical way to provide an explanation, i.e., the answer to a "why" question. in our setting, a why explanation is an answer to the question "why is r c the relation holding between s 1 and s 2 ?". gold explanations. these explanations (gold) are the explanations provided in the original dataset, either directly written or manually checked by humans given the correct relation r c , thus falling into the label-confirming explanation type like why explanations. while the quality of human-generated explanations is generally considered high (e.g., we expect that they point out relevant and implicit information), there is no guarantee that, when used by a model m 2 , they will perform better than modelgenerated ones. therefore, for the purposes of this study, we evaluate them along with the generated ones and take them as a strong baseline, rather than consider them a target or reference explanation. ## results and discussion accuracy trends. in the preferred explanations were those of deepseek with all m 2 s in all tasks, with the exception of erteit, where llama-m2 and gemma-m2 scored higher with llama-m1's why explanations. label-confirming explanations (why) consistently led to the highest accuracy across all datasets and models, confirming that explanations aligned with the gold label can meaningfully support the m2 model's decision-making. on the other hand, label-contradicting explanations generally scored lower than the noexp baseline, as was expected, indicating that "bad" explanations can indeed be detrimental to the model's accuracy. however, there are a few cases where cf explanations im-proved over the noexp baseline, specifically on esnli and ecare and mainly with deepseek-m2's explanations. we manually inspected a sample of the cf explanations that led to a correct prediction, and noticed a common trend: in fact, in these cases either the model "refused" to support the opposite label, or it produced a long "chain of thought" style explanation which was truncated, and therefore did not contain the section of the explanation supporting the opposite label. this was especially the case with deepseek, which produced an initial "reasoning" independently of the supported relationship, which was helpful for the downstream model to predict the correct label. finally, it is worth noticing that machinegenerated why explanations consistently outperform the human-generated ones (gold), which nonetheless are beneficial to prediction accuracy compared to noexp. association measures. we computed three association measures between the explanation and the input: causal strength, cosine similarity, and entailment probability ( ## conclusion in this study, we tested the effects of explanations on llms, showing that they can significantly improve their accuracy in predicting relations between sentences. this improvement is consistent across different models, datasets, and explanation types. our experiments also show a correlation between explanation effectiveness and the degree of implicit knowledge conveyed by the explanations, suggesting that explanations that introduce novel and relevant information are more likely to be helpful to llms. furthermore, our analysis reveals that different llms exhibit varying sensitivity to different explanation types. our findings contribute to research on the role of explanations in enhancing llm performance. by understanding the nuances of model sensitivity to different explanation types and the ways in which explanations contribute to implicit knowledge acquisition, we can develop more effective techniques for explaining and improving the reasoning capabilities of llms. future work should explore how to automatically generate or filter explanations with high causal alignment to further boost downstream model performance. limitations knowledge in language. finer-grained techniques are needed for a comprehensive evaluation of implicitness. future work should explore additional features, such as explanation length and syntactic complexity, to better understand their interplay with model performance. we utilized a controlled experimental setup, where explanations are provided in a specific format and injected into the model during inference. real-world applications might involve more complex scenarios with less controlled input and output formats. also, while our study focused on sentence-pair tasks, the geiser pipeline can in principle be extended to multi-hop reasoning chains and other explanation-rich settings by iteratively injecting intermediate explanations, which we plan to explore in future work.