I nterro L ang: Exploring NLP Models and Datasets through Dialogue-based Explanations (2023)

Abstract: While recently developed NLP explainability methods let us open the black box in various ways (Madsen et al., 2022), a missing ingredient in this endeavor is an interactive tool offering a conversational interface. Such a dialogue system can help users explore datasets and models with explanations in a contextualized manner, e.g. via clarification or follow-up questions, and through a natural language interface. We adapt the conversational explanation framework TalkToModel (Slack et al., 2022) to the NLP domain, add new NLP-specific operations such as free-text rationalization, and illustrate its generalizability on three NLP tasks (dialogue act classification, question answering, hate speech detection). To recognize user queries for explanations, we evaluate fine-tuned and few-shot prompting models and implement a novel adapter-based approach. We then conduct two user studies on (1) the perceived correctness and helpfulness of the dialogues, and (2) the simulatability, i.e. how objectively helpful dialogical explanations are for humans in figuring out the model’s predicted label when it’s not shown. We found rationalization and feature attribution were helpful in explaining the model behavior. Moreover, users could more reliably predict the model outcome based on an explanation dialogue rather than one-off explanations.

## introduction framing explanation processes as a dialogue between the human and the model has been motivated in many recent works from the areas of hci and ml explainability @xcite @xcite @xcite . with the growing popularity of large language models (llms), the research community has yet to present a dialogue-based interpretability framework in the nlp domain that is both capable of conveying faithful explanations foot_0 in human-understandable terms and is generalizable to different datasets, use cases and models. one-off explanations can only tell a part of the overall narrative about why a model "behaves" a certain way. saliency maps from feature attribution methods can explain the model reasoning in terms of what input features are important for making a prediction @xcite , while counterfactuals and adversarial examples show how an input needs to be modified to cause a change in the original prediction @xcite . semantic similarity and label distributions can shed a light on the data which was used to train the model @xcite , while rationales provide a natural language justification for a predicted label @xcite . these methods do not allow follow-up questions to clarify ambiguous cases, e.g. a most important token being a punctuation (figure 1 ) (cf. @xcite , or build a mental model of the explained models. in this work, we build a user-centered, dialoguebased explanation and exploration framework, in-terrolang, for interpretability and analyses of nlp models. we investigate how the talkto-model (ttm, @xcite ) framework can be implemented in the nlp domain: concretely, we define nlp-specific operations based on the aforementioned explanation types. our system, in-terrolang, allows users to interpret and analyze the behavior of language models interactively. we demonstrate the generalizability of interrolang on three case studies -dialogue act classification, question answering, hate speech detection -for which we evaluate the intent recognition (parsing of natural language queries) capabilities of both finetuned (flan-t5, bert with adapter) and few-shot llm @xcite . we find that an efficient adapter setup outperforms few-shot llms, but that this task of detecting a user's intent is far from being solved. in a subsequent human evaluation ( §5.3), we first collect subjective quality assessments on each response about the explanation types regarding four dimensions (correctness, helpfulness, satisfaction, fluency). we find a preference for mistakes summaries, performance metrics and free-text rationales. secondly, we ask the participants about their impressions of the overall explanation dialogues. all of them were deemed helpful, although some (e.g., counterfactuals) have some potential for improvement. finally, a second user study on simulatability (human forward prediction) provides first evidence for how various nlp explanation types can be meaningfully combined in dialogical settings. attribution and rationales resulted in very high simulation accuracies and required the least number of turns on average, revealing a need for a longer conversation than single-turn explanations. we open-source our tool foot_1 that can be extended to other models and nlp tasks alongside a dataset collected during the user studies including various operations and manual annotations for the user inputs (parsed texts): free-text rationales and templatebased responses for the decisions of nlp models include explanations generated from interpretability methods, such as attributions, counterfactuals, and similar examples. ## methodology talktomodel @xcite is designed as a system for open-ended natural language dialogues for comprehending the behavior of ml models for tabular datasets (including only numeric and categorical features). our system interrolang retains most of its functionalities: users can ask questions about many different aspects and slices of the data alongside predictions and explanations. in-terrolang has three main components (depicted in figure 2 ): a dialogue engine parses user inputs into an sql-like programming language using either adapters for intent classification or llm that treats this task as a seq2seq problem, where user inputs are the source and the parses are the targets. an execution engine runs the operations in each parse and generates the natural language response. a text interface (figure 4 ) lets users engage in open-ended dialogues and offers pre-defined questions that can be edited. this reduces the users' workload to deciding on what to ask, essentially. ## nlp models we selected three use cases in nlp with bert-type transformer models trained on standard datasets, all of which we offer users to explore. ## interpretability and analysis components for our implementation and experimental setup, we use the following tools and methods to realize the operations in perturbation for counterfactual generation, we use the official hugging face implementation of polyjuice @xcite foot_3 . adversarial examples are generated via openattack @xcite foot_4 , where we choose pwws @xcite as the attacker for our models on a single instance. for data augmentation we use the nlpaug library foot_5 and replace some tokens in the text based on their embedding similarity computed with the bert-based-cased model. the percentage of words that are augmented for each text is set to 0.3. we display the replaced words in bold, so that the user can easily distinguish between the original instance and the augmented one. rationalization as a baseline, we use the parsing model (gptneo) in a zero-shot setup to produce free-text explanations based on a concatenation of the input, the classification by the explained bert-type model @xcite and an instruction asking for an explanation. for an improved version, we produce plausible rationales from chatgpt foot_6 and then prompt a dolly-v2-3b foot_7 for few-shot rationales. the rationales are precomputed for all datasets. natural language understanding for computing the semantic similarity, we embed the data point using sentence transformers @xcite and compute the cosine similarity to other points (excluding the instance in question) in the respective dataset. in order to retrieve frequent keywords from the whole dataset, we apply the stopwords set defined in nltk @xcite and get a word frequency set. the operation can then return the n most frequent keywords, with n being defined through the user query. ## evaluation we conduct our evaluation based on parsing accuracy and two user studies. after introducing the partitions we used to obtain the parsing (intent recognition) results ( §5.2), we describe the setup free-text rationale 62.07 62.50 65.45 ## results and discussion parsing accuracy human preferences simulatability based on ## related work dialogue systems for interpretability in ml dalvi mishra et al. (2022) proposed an interactive system to provide faithful explanations using previous interactions as a feedback. despite being interactive, it does not provide feasibility of generating rationales on multiple queries subsequently. bertrand et al. (2023) wrote a survey on prior studies on "dialogic xai", while fig. 6 of jacovi et al. (2023) highlights that interactive interrogation is needed to construct complete explanation narratives: feature attribution and counterfactuals complement each other, s.t. the users can build a generalizable mental model. visual interfaces for interpretability in nlp lit @xcite , azimuth @xcite , ifan @xcite and webshap (wang and chau, 2023) offer a broad range of explanations and interactive analyses on both local and global levels. robustness gym @xcite , seal @xcite , @xcite , interactive model cards @xcite and datalab @xcite offer model evaluation, dataset analysis and accompanying visualization tools in practice. there are overlaps with interrolang in the methods they integrate, but none of them offer a conversational interface like ours. user studies on nlp interpretability most influential to our study design are simulatability evaluations @xcite @xcite @xcite . in terms of preference ratings, strout et al. (2019) evaluated how extractive rationales (discretized attributions) from different models are rated by human annotators. helpfulness and satisfaction ratings were used in schuff et al. (2020) and ray et al. (2019). ## conclusion we introduce our system, interrolang, which is a user-centered dialogue-based system for exploring the nlp datasets and model behavior. this system enables users to engage in multi-turn dialogues. based on the findings from our conducted user study, we have determined that one-off explanations alone are usually not sufficient or beneficial. in many cases, users may require multiple explanations to obtain accurate predictions and gain a better understanding of the system's output. future work includes making the bot more proactive, so that it can suggest new operations related to the user queries. we also want to investigate the feasibility of using a singular llm for all tasks (parsing, prediction, explanation generation foot_9 , response generation) over the modular setup that we currently employ; redesigning operations as api endpoints and training llms to call them @xcite , s.t. they can autonomously take care of the entire dialogue management at once. lastly, refining language models (increasing faithfulness or robustness, aligning with user expectations) through dialogues has gained traction @xcite . while we are already collecting valuable data, our framework misses an automated feedback loop to iteratively improve the models.