LLMC heckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-Explanations (2024)

Abstract: Interpretability tools that offer explanations in the form of a dialogue have demonstrated their efficacy in enhancing users’ understanding (Slack et al., 2023; Shen et al., 2023), as one-off explanations may fall short in providing sufficient information to the user. Current solutions for dialogue-based explanations, however, often require external tools and modules and are not easily transferable to tasks they were not designed for. With \texttt{LLMCheckup} , we present an easily accessible tool that allows users to chat with any state-of-the-art large language model (LLM) about its behavior. We enable LLMs to generate explanations and perform user intent recognition without fine-tuning, by connecting them with a broad spectrum of Explainable AI (XAI) methods, including white-box explainability tools such as feature attributions, and self-explanations (e.g., for rationale generation). LLM-based (self-)explanations are presented as an interactive dialogue that supports follow-up questions and generates suggestions. \texttt{LLMCheckup} provides tutorials for operations available in the system, catering to individuals with varying levels of expertise in XAI and supporting multiple input modalities. We introduce a new parsing strategy that substantially enhances the user intent recognition accuracy of the LLM. Finally, we showcase \texttt{LLMCheckup} for the tasks of fact checking and commonsense question answering. Our code repository: https://github.com/DFKI-NLP/LLMCheckup

## introduction to unravel the black box nature of deep learning models for natural language processing, a diverse range of explainability methods have been developed @xcite @xcite . nevertheless, practitioners often face difficulties in effectively utilizing *equal contribution 1 https://github.com/dfki-nlp/llmcheckup explainability methods, as they may not be aware of which techniques are available or how to interpret results provided. there has been a consensus within the research community that moving beyond one-off explanations and embracing conversations to provide explanations is more effective for model understanding @xcite @xcite and helps mitigate the limitations associated with the effective usage of explainability methods to some extent @xcite . in the field of nlp, two dialogue-based interpretability tools, interrolang @xcite and convxai @xcite , have been introduced. both tools employ multiple, separately fine-tuned lms to parse user intents and dedicated external lms to provide explanations. by contrast, our framework, llmcheckup, only requires a single llm and puts it on "quadruple duty": (1) analyzing users' (explanation) requests ( §2.1, §5.1), (2) performing downstream tasks ( §4), (3) providing explanations for its outputs ( §3), and (4) responding to the users in natural language ( §2.3). instead of using many different lms to explain the behavior of another llm, llmcheckup allows us to directly employ the same llm used for user intent recognition to selfexplain its own behavior. the advantage of a singlemodel approach is that it simplifies the engineering aspect of building an xai system without multiple external modules and separately fine-tuned models. at the same time, we consistently achieve good performance even with a single model, as modern llms are very powerful and can handle a wide range of tasks including user intent recognition and explanation generation. thus, llmcheckup provides a unified and compact framework that is useful for future user studies in the context of human-computer interaction and explainability. ## llmcheckup llmcheckup is an interface for chatting with any llm about its behavior. we connect several white-box and black-box interpretability tools ( §3), s.t. llmcheckup takes into account model internals, datasets and documentation for generating self-explanations. user requests for explanations are recognized via a text-to-sql task performed by the llm under investigation ( §2.1-2.2). we showcase a short dialogue between the user and llmcheckup in figure 1 and a longer dialogue featuring different operations in appendix b. llmcheckup can answer various questions related to the data as well as the model's behavior. for example, in figure 1 the user is interested in the rationale for a specific prediction and the model generates an explanation to justify the assigned label. llmcheckup also suggests to have a look at another related operation (token-level importance scores) that can help explain model's behavior ( §2.4), but the user asks for a modified (aug-mented) version of the same instance instead. as a result, the model paraphrases the original question which can be then treated as a new sample and the user can further examine it by using the custom input functionality of llmcheckup ( §2.4). ## nlp explainability tools while we introduce each explainability method individually, these methods can be interconnected through follow-up questions from users or suggestions provided by llmcheckup to preserve context. ## use cases in this paper, we demonstrate the workflow of llmcheckup on two typical nlp tasks: fact checking and commonsense question answering. figure 1 and appendix b show sample dialogues where user asks questions regarding rationalization, data augmentation and other operations based on the ecqa data @xcite for commonsense question answering. the llmcheckup repository includes all the necessary configuration files for different lms and our use cases. they can be easily adopted to many other downstream tasks, data and transformer-type models, demonstrated in a tutorial which will be available with the camera-ready version of our repository. ## evaluation we conducted evaluations for parsing and data augmentation with llms using automated evaluation metrics foot_7 . among all the supported methods presented in ## discussion in contrast to previous dialogue-based xai frameworks convxai @xcite and inter-rolang @xcite , which require a fine-tuned model for each specific use case, llms used in llmcheckup possess remarkable zero-/few-shot capabilities @xcite for effectively handling many tasks without requiring fine-tuning. although the quality of an explanation could be enhanced with further fine-tuning, llmcheckup uses model outputs out of the box. our empirical results underline the feasibility of conversational interpretability and the usefulness of llmcheckup for future studies, especially human evaluation. we focus on the ground work in terms of engineering, implementation and user interface, for connecting the human with the model. this provides user studies @xcite @xcite on conducting their study. we see evaluation measures for differences between users' mental models and model behavior and objective metrics beyond simulatability as the most important gaps to fill. ## related work interfaces for interactive explanations lit @xcite ) is a gui-based tool available for analyzing model behaviors across entire datasets. however, lit has less functionalities in terms of prompting and lower accessibility, e.g. no tutorial and a lower level of integration with huggingface. crosscheck @xcite exhibits the capability to facilitate quick crossmodel comparison and error analysis across various data types, but adapting it for other use cases needs substantial code modification and customization. xmd's @xcite primary purpose is model debugging, but it shares similarities in the focus on feature attributions, visualization of single instances and user feedback options. it is, however, limited to feature attribution explanations and smaller, efficiently retrainable models. ifan @xcite enables real-time explanationbased interaction with nlp models, but is limited to the sequence-to-class format, restricting its applicability to other tasks and it offers only a limited set of explainability methods. dialogue-based systems for interpretability @xcite point out that conversational interfaces have the potential to greatly enhance the transparency and the level of trust that human decision-makers place in them. according to @xcite user studies, delivering explanations in a conversational manner can improve users' understanding, satisfaction, and acceptance. @xcite emphasizes the necessity of interactive interrogation in order to build understandable explanation narratives. convxai @xcite , talktomodel @xcite , interrolang @xcite and @xcite share some similarities with our framework, but are more complex in their setup and consider fewer explainability methods. additionally, they might overrely on external lms to explain the deployed lm's behavior, whereas llmcheckup places a strong emphasis on selfexplanation, which is crucial for faithfulness. finally, llmcheckup uses auto-regressive models, as they have become increasingly dominant in various nlp applications nowadays. in isee @xcite , a chatbot adapts explanations to the user's persona, but they do not consider llms. ## conclusion we present the interpretability tool llmcheckup, designed as a dialogue-based system. llm-checkup can provide explanations in a conversation with the user facilitated by any auto-regressive llm. by consolidating parsing, downstream task prediction, explanation generation and response generation within a unified framework, llm-checkup streamlines the interpretability process without switching between different lms, modules or libraries and serves as a baseline for future investigation. future work includes exploring rag models @xcite combined with explainability, as currently llmcheckup relies on search engines for external information retrieval. we also want to add multi-modal models, so that converting image or audio input to texts would no longer be necessary, but the current state of interpretability on such models lags behind unimodal approaches @xcite . integrating our framework into hug-gingchat 12 would further increase the visibility and accessibility through the web.