Enhancing Model Transparency: A Dialogue System Approach to XAI with Domain Knowledge (2024)

Abstract: Explainable artificial intelligence (XAI) is a rapidly evolving field that seeks to create AI systems that can provide human-understandable explanations for their decision-making processes. However, these explanations rely on model and data-specific information only. To support better human decision-making, integrating domain knowledge into AI systems is expected to enhance understanding and transparency. In this paper, we present an approach for combining XAI explanations with domain knowledge within a dialogue system. We concentrate on techniques derived from the field of computational argumentation to incorporate domain knowledge and corresponding explanations into human-machine dialogue. We implement the approach in a prototype system for an initial user evaluation, where users interacted with the dialogue system to receive predictions from an underlying AI model. The participants were able to explore different types of explanations and domain knowledge. Our results indicate that users tend to more effectively evaluate model performance when domain knowledge is integrated. On the other hand, we found that domain knowledge was not frequently requested by the user during dialogue interactions.

## introduction explainable artificial intelligence (xai) has emerged as an important and evolving domain within the field of ai, with the goal of enabling ai systems to explain their decision-making in ways that are understandable and accessible to humans @xcite @xcite . one potential strategy for attaining this objective is the use of dialogue systems that facilitate seamless and effective access to explanations in a natural manner. the goal of this paper is to explore the impact of integrating domain knowledge into an explanatory dialogue system, aiming to enhance user comprehension in ai-driven decisions. dialogue, by its very nature, facilitates the dissemination of information in a structured manner @xcite . through dialogue, users cannot only receive explanations but also pose questions tailored to their specific needs. this enables a dynamic interaction in which mental models can be scrutinized and refined through question-and-answer exchanges @xcite . however, in the case of explanatory dialogue systems utilizing xai, prevailing conversational interfaces @xcite @xcite directly map user intents to xai operations and furnish templatebased responses. while expedient, this approach often overlooks the nuances of dialogue context, potentially leading to misunderstandings and impeding the natural flow of interaction. in @xcite , fundamental requirements for explanatory dialogue systems tailored to xai contexts were delineated. contextual information is essential for a comprehensive understanding of a given situation. although ai models and xai methodologies are adept at processing data-centric information, they are constrained by their inability to incorporate domain-specific context, which limits their capacity to provide insights beyond the scope of the underlying data. a deeper understanding of a model can be achieved by acquiring additional knowledge from the field in question. incorporating domain knowledge into xai systems can create more transparent and trustworthy models that better support human decision-making. in this paper, we present an approach for modeling domain knowledge within explanatory dialogues ( §2), highlighting its importance in fostering richer interactions. in a study with 32 participants, we evaluate the effectiveness of our dialogue system which integrates xai explanations and domain knowledge( §3, §4). our results show that users can better assess a model's predictions through domain knowledge ( §5). ## modeling domain knowledge in the field of explainable artificial intelligence (xai), there are two main categories of explanations: local and global explanations @xcite . local explanations focus on individual predictions, illustrating how specific input features affect the outcome for a single instance. in contrast, global explanations provide an overarching view of how the model behaves across the entire dataset, showing general patterns and the importance of features. two prominent types of local explanations are counterfactual explanations @xcite and shapley values @xcite . counterfactual explanations describe how altering certain input values can change the prediction. shapley values, on the other hand, allocate the contribution of each feature to the prediction. these explanations typically illustrate the relationship between input features and the classifier's outcomes. however, this relationship is always described through statistical means. we assume that domain knowledge provides additional information for these statistical explanations, thereby assisting users in their understanding process. for example, shapley values might reveal gender as the most important feature in a titanic survival prediction model. a user might then ask why gender is so important. domain knowledge can provide context, such as historical or societal factors like women are more likely to survive than men because they were prioritized for the lifeboats (see also to facilitate effective access to domain knowledge in dialogue, we propose using bipolar argumentation structures @xcite . arguments and explanations are inherently similar, as both aim to clarify and justify information. argument components can relate to each other with a polarity, either supporting or attacking another argument. to create a robust dialogue, these arguments can be organized within an argument tree @xcite . the main claim, which is the central topic of discussion, acts as the root of the argument tree, facilitating selective access to the most relevant information. this structure prevents information overload and provides coherent, contextually relevant explanations. by mapping main claims to the statistical explanations provided by the xai system, we en- system okay, let's have a look. my prediction is that a person with the values you provided would be accepted. if you want a more detailed explanation, please tell me what you'd like to know. for example, you might ask why i decided the way i did or what would have changed the outcome. user why do you think i would have been accepted? system all right, let's have a look. in your case, the most influential value was amount, as you can see in the figure below. you can also see the influence of all other values, ordered by their importance. purple bars indicate a positive contribution, blue bars a negative contribution. is there anything else i can help you with? user why does a high credit amount increase my chances of being accepted? system the amount of credit requested can affect acceptance as larger loan amounts may pose higher risks for lenders, requiring more stringent evaluation of the borrower's financial capacity to repay. we propose to have one argument tree with a specific claim for each feature-outcome relation. each tree can contain supporting or opposing arguments, providing a wide range of information on the domain. this results in multiple argument trees within an explanatory dialogue, effectively representing the necessary domain knowledge. by implementing these argument trees, we can ensure that users receive comprehensive and contextually relevant explanations, enhancing their understanding and engagement with the ai model. ## explanatory dialogue system we implement the proposed approach ( §2) in an existing explanatory dialogue system, which was introduced in @xcite . the generic dialogue system supports various datasets and operates on two scenarios: german credit data (hofmann, 1994) and the titanic dataset @xcite . the focus is on numerical and categorical datasets, utilizing a random forest classifier for real-time computation, enabling faster xai methods calculation and thus a more natural, steady conversation. the system supports two types of local explanations: shapley values and counterfactuals (see §2). figure 1 shows the architecture of the evaluated system. for integrating domain knowledge, we introduce a module providing suitable arguments. these arguments can be obtained either through manual acquisition from consulting domain experts or in case of widely studied topics by using automated procedures, e.g. large language models or semantic databases. to exemplify, we manually extracted arguments for the titanic dataset from existing literature @xcite (domain experts) and used chatgpt foot_0 to generate arguments for the credit domain, which were then manually verified for accuracy. additionally, each argument was manually annotated in order to align it with the desired argument tree structure and to provide a reference link to the ai features addressed in the argument 2 . however,research indicates that this process can also be automated in the future @xcite . since adding domain knowledge creates new user queries, we replaced the original keywordbased natural language understanding with a finetuned bert model @xcite to provide a more natural interaction. feature values and names were replaced by placeholders, ensuring the model is not fine-tuned on specific scenarios but rather on explanatory dialogue, thus keeping the system generic. the data for training this model was manually generated with chatito foot_2 . the rule-based dialogue interaction involves asking the user for feature-specific information and providing a prediction. the system then asks the user if they would like to receive an explanation of the prediction. we revise the explanation policy and introduce new interaction steps, by distinguishing between xai explanations and domain knowledge explanations (see figure 2 ). instead of directly mapping user intentions to specific xai explanations, we implement a more abstract intent for requesting an explanation, which determines the most suitable explanation based on the dialogue state and provided information. initially, we offer shapley explanations to give the user insight into the features impacting the outcome, presenting a simplified graph foot_3 of the values for lay users. we then provide additional information from domain knowledge, either based on the previous interaction (e.g., shapley values, specific argument) or on requested feature values (see ## study setup to assess the initial impacts of integrated domain knowledge, we conduct a user study, presenting the dialogue system ( §3) in a web environment (see appendix a) with two distinct models trained on the titanic and credit datasets. in one scenario, the ai is trained on accurate training data (true ai), while in the other, the expected class is inverted in the training set to simulate a malfunctioning ai (false ai). this study setting has been designed to ascertain whether users can discern the false ai using authentic domain knowledge. we assess the user's impression of the ai using the questions: i agree with the decisions made by the system (q1) and the system decisions are plausible (q2). in addition to evaluate the overall performance of the system, we employed the sassi questionnaire @xcite . each of the 32 participants interacted with the system twice, experiencing both scenarios (credit/ titanic) and one ai setting (true or false), with and without domain knowledge activated. this resulted in four groups based on ai truthfulness and scenario variance (see the study began with general instructions and a task description. users were encouraged to interact and explore explanations, with the task designed to be open-ended for a natural conversation. after each scenario, the participants completed the questionnaire on a five-point likert scale (q1, q2 and sassi). finally, we collected demographic information and participants' attitudes towards and experiences with ai (see ## evaluation we discover notable differences in the interactions between the true and false ai setting, as shown in these findings underline the importance of domain knowledge in explanatory dialogues and highlight both the system's strengths and areas for improvement, guiding future enhancements to better support user understanding and interaction. however, given the small sample size, these results only indicate trends. a more extensive evaluation with a larger participant pool is planned for the future to validate these findings more robustly. ## conclusion and future work in this paper, we highlighted the need for domain knowledge integration in explanatory dialogue systems. our approach employs argumentation structures to incorporate domain knowledge into explanatory dialogue systems, enhancing the transparency and comprehensibility of ai model explanations. by extending an existing explanatory dialogue system with domain knowledge, we demonstrate the practicality of our approach and con-ducted a study to evaluate the performance of this enhanced system. while we observed the supportive role of domain knowledge in enhancing explanations in a way that users can more effectively evaluate model performance, several challenges remain. enhancing interaction and optimizing the explanation policy are essential to ensure that users are capable to address their questions and receive the most relevant and comprehensive explanations to them, including alternative information such as feature descriptions. additionally, improving the nlu component based on our observed explanation interaction patterns is crucial for facilitating more natural conversations.