X plain LLM : A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLM s (2024)

Abstract: Large Language Models (LLMs) have achieved remarkable success in natural language tasks, yet understanding their reasoning processes remains a significant challenge. We address this by introducing XplainLLM, a dataset accompanying an explanation framework designed to enhance LLM transparency and reliability. Our dataset comprises 24,204 instances where each instance interprets the LLM’s reasoning behavior using knowledge graphs (KGs) and graph attention networks (GAT), and includes explanations of LLMs such as the decoder-only Llama-3 and the encoder-only RoBERTa. XplainLLM also features a framework for generating grounded explanations and the debugger-scores for multidimensional quality analysis. Our explanations include why-choose and why-not-choose components, reason-elements , and debugger-scores that collectively illuminate the LLM’s reasoning behavior. Our evaluations demonstrate XplainLLM’s potential to reduce hallucinations and improve grounded explanation generation in LLMs. XplainLLM is a resource for researchers and practitioners to build trust and verify the reliability of LLM outputs. Our code and dataset are publicly available.

## introduction as the capabilities and applications of large language models (llms) continue to expand @xcite @xcite , the need for transparency and interpretability in their reasoning behavior has become increasingly urgent @xcite . traditional methods @xcite @xcite allow us to get insights into the reasoning behind language model outputs, but they fall short of providing a complete picture, leaving the logic behind complex decision opaque @xcite . this gap presents a significant barrier in applications where 1 https://lmexplainer.github.io/xplainllm model decision transparency is important, such as healthcare @xcite , law @xcite , and public services @xcite . current methods for explaining llm's reasoning behavior primarily focus on the analysis of parameter changes @xcite @xcite and chain-of-thought (cot) based self-explanation @xcite . analysis of parameter changes bases the explanations on self-attention weights in models like @xcite and gpt-2 @xcite , deducing correlations between input tokens and the model's predictions. however, the relationships highlighted in these generated explanations are difficult to understand for humans. cot-based self-explanation, on the other hand, iteratively generates rationales step-by-step. due to the inherent constraints in llms, these explanations often have hallucinations and can not reflect the real reasoning process @xcite . we introduce xplainllm, a dataset accompanying an explanation framework designed to enhance transparency, explainability, and understandability in llm reasoning behaviors. by integrating knowledge graphs (kgs) and graph attention networks (gat) @xcite , we construct a structured and reliable dataset that anchors explanations in reasoning-relevant knowledge. we link the llm reasoning process to the entities and relations within kgs to help provide an intuitive and interpretable representation of the llm's decision-making process. our process also helps facilitate model tuning, debugging, robustness evaluation and demonstration in in-context learning. xplainllm provides a structured explanation of two distinct types of llms: llama-3-8b @xcite (decoder-only model) and robertalarge @xcite (encoder-only model). a total of 24,204 instances are included in the dataset. the explanations are tied to two models' reasoning dataset size answer format expl. format source model match? self-explanatory? "why not" included? cos-e 9,500 mc nl human × × × ecqa 10,962 mc nl human × ✓ × neuron 307,200 neuron nl + score model ✓ × × xplainllm 24,204 mc nl model ✓ ✓ ✓ we evaluate the quality of the explanations in xplainllm through human and automated evaluations. the overall quality of explanations achieves an average score of 0.87/1.00 by human evaluators, and an average of 0.89/1.00 by automated evaluators. we evaluate our framework by comparing the performance of llms with and without our framework, and the results show that llms under our framework outperform the benchmark, with a performance gap extending to 20%. we also evaluate the quality of the explanations generated by our framework, and the results underscore the quality of our explanations on multiple metrics. in summary, we make two key contributions to the field of explainable ai for llms: (1) an explanation dataset of model reasoning behavior, and (2) a framework for improving the interpretability of llms through structured, grounded explanations. to the best of our knowledge, xplainllm is the first dataset to provide structured and grounded explanations for llm reasoning behavior. ## related work interpretability in llms explainable ai (xai) aims to address the issue of interpreting the outcomes of language models @xcite @xcite . one of its goals is to generate explanations that enable humans to easily understand the decision-making process. zelikman et al. (2022); zhang and gao (2023 2023 ) utilize the cot to find the rationale and apply the reasoning capabilities of llms to domain tasks. however, these explanations are inherently constrained in capturing prompt-specific reasoning, which often generates hallucinations and can not reflect the real reasoning of llms @xcite . another goal of xai is to explain the model in a trustworthy way. @xcite introduce an explainable factor to minimize the risk of unreasonable explanation generation. @xcite integrate the external knowledge to generate why and why-not counterfactual explanations. zelikman et al. (2022) apply self-checker mechanism to ensure trusted rationals. however, these methods fail to accurately capture the core reasoning of llms. in contrast, our work enhances llm trustworthiness and deepens human understanding of its reasoning behavior, improving their potential in end-user applications. explanation datasets the explainable datasets for language models can be categorized into three types @xcite : (1) highlights: provide input elements such as words and phrases, as explanations to a predicted output @xcite @xcite ; (2) free-text explanations: provide readable textual explanations in words or sentences @xcite @xcite ; (3) structured explanations: provide natural language explanation but are constrained by the explanation writing process @xcite @xcite . different from these, our explanation incorporates highlighted reason-elements and guided instruction to generate a free-text explanation. our explanation is structured and grounded in the reasoning process, enhancing the trustworthiness and comprehensiveness of the content. we present a comparison with prevalent explanation datasets @xcite @xcite in ## conclusion we introduce xplainllm: a knowledge-augmented dataset paired with an explanation framework designed to enhance the interpretability of llms. our dataset and framework provide a way for llms to generate reliable and grounded explanations without additional training. through the use of debugger-score, we provide a multidimensional analysis of quantitatively evaluate the quality of explanations. our evaluations demonstrate that xplainllm not only grounds explanations in reasoning behavior, but also helps llms reduce hallucinations and improve their performance. the dataset and code are available at https: //lmexplainer.github.io/xplainllm . we release them under the mit license to encourage further research in explainable ai.