Explaining Decision-Tree Predictions by Addressing Potential Conflicts between Predictions and Plausible Expectations (2021)

Abstract: We offer an approach to explain Decision Tree (DT) predictions by addressing potential conflicts between aspects of these predictions and plausible expectations licensed by background information. We define four types of conflicts, operationalize their identification, and specify explanatory schemas that address them. Our human evaluation focused on the effect of explanations on users’ understanding of a DT’s reasoning and their willingness to act on its predictions. The results show that (1) explanations that address potential conflicts are considered at least as good as baseline explanations that just follow a DT path; and (2) the conflict-based explanations are deemed especially valuable when users’ expectations disagree with the DT’s predictions.

## introduction machine learning (ml) models have become increasingly accurate in recent times, leading to their widespread adoption by decision makers in a variety of vital domains, including healthcare, defense and energy. this underscores the need for explanations of the outcomes of these models that support decision making by practitioners. ml models may be classified into transparent and opaque models based on their interpretability @xcite . transparent models are "interpretable by a machine learning expert or a statistician" @xcite . these models, e.g., decision trees (dts), decision rules and linear models, are built on the basis of interpretable features, which are typically obtained through feature engineering. transparent models are often less accurate than opaque models, in particular neural networks, provided large training datasets are available. nonetheless, it is necessary to explain transparent models because (1) large datasets may not always be available, as is the case in our evaluation datasets ( § 4.1); (2) it is common practice to clarify the outcomes of opaque models by approximating them with transparent models ( § 2); and (3) even if these transparent models are understandable by ml experts, they may still be unclear to practitioners. in this paper, we generate textual explanations of predictions made by a particular transparent model: dt. our explanations address potential conflicts between aspects of these predictions and plausible expectations licensed by background information (i.e., expectations that "make sense" in light of this information). specifically, we identify four types of conflicts whereby events that appeared unlikely or likely on the basis of background information happened or did not happen respectively; we then specify schemas that address these conflicts ( § 3). we generated explanations for two datasets: telecom and nursery. in telecom, a dt predicts whether a customer will churn (leave) or stay with the company based on their profile (e.g., whether they have a phone service and what are their monthly charges); in nursery, a dt predicts the acceptance status of a child to a childcare center on the basis of the circumstances of the child and their family (e.g., how satisfactory are the current childcare arrangements and how demanding is the parents' employment). the bottom part of our human evaluation of the explanations generated for the two datasets ( § 4) considers users' overall preferences for different explanation types, and the effect of explanations on two explanatory goals: users' understanding of the dt's reasoning, and feature value parents' employment: challenging current childcare: good child's health: average from the data, one might expect that children with good current childcare will be a great deal more likely to get wait listed than to get a priority acceptance (54% vs 11%). however, the ai system has learned from the data that among children with challenging parents' employment and average health, those with good current childcare are almost certain to get a priority acceptance (close to 100%). the main findings of our user study are: (1) explanations that address potential conflicts are generally considered at least as good as baseline explanations that just follow a dt path; and (2) the conflictbased explanations are deemed especially valuable when users' expectations disagree with dt predictions. we stress that these findings pertain to explanations that address conflicts due to plausible expectations from background information. we do not claim that these explanations address actual user expectations. ## related work in 1990-2000, explanations derived from knowledge bases were enhanced by addressing aspects of users' reasoning. specifically, @xcite and @xcite considered potential inferences from explanations, omitting easily inferable information and addressing erroneous inferences; @xcite took into account reasoning fallacies when explaining the reasoning of bayesian networks; and @xcite generated instructions from which users could draw appropriate inferences about actions to take. recently, @xcite identified additional triggers that should be addressed in explanations. current research on explanation generation focuses on explaining the predictions made by ml models -a sub-field called explainable ai (xai). in particular, neural networks have received a lot of attention owing to their superior performance on one hand, and their opaqueness on the other hand. a common first step in explaining the predictions of neural networks is to build a local surrogate explainer model that uses a transparent model to approximate the neighbourhood of an instance of interest. linear regression @xcite @xcite , decision rules @xcite and dts @xcite @xcite have been employed for this purpose. a dt's prediction is generally explained by tracing the path from the root to a predicted outcome @xcite . recently, researchers have generated class-contrastive counterfactual explanations to enhance the explanations of dt predictions. @xcite generated explanations that have a factual and a counterfactual component; the former is the dt trace, while the latter was found by ranking all the paths leading to alternative outcomes according to their distance from the factual explanation. @xcite studied counterfactual explanations for dts in an interactive system where users could change or remove features, or request an explanation for a hypothetical instance. counterfactual explanations were generated by representing the tree structure as binary meta-features, and minimizing an l1-like metric to retrieve the shortest statement. however, these works do not determine when a counterfactual enhancement is required. the need for an enhancement was studied in (biran and mckeown, 2017) -they identified and addressed unexpected effects of individual features on predictions made by logistic regression. however, they did not consider unexpected predictions. @xcite argued that good explanations must be written for a specific purpose and audience, have a narrative structure, and use vague language to communicate uncertainty. the explanations generated in @xcite and @xcite have a narrative structure, and only those in @xcite use vague language to convey strength of evidence. the approach described in this paper complements explanations by addressing both unexpected predictions and unexpected effects of features, thereby enhancing their narrative structure. in addition, we leverage the work of @xcite to address reiter's desideratum of using vague language to convey probabilities. finally, and more broadly, expectation-theory posits that the surprisingness of an event may stem from a discrepancy between the state of the world and propositions that are deducible from presented information @xcite . @xcite offer a bayesian formulation of the influence of surprisingness on visual attention shifts in terms of the difference between prior and posterior probabilities. in our research, we employ a probabilistic formulation to identify potential conflicts between plausible expectations and aspects of dt predictions. ## justifying dt predictions in this work, we explain the outcome predicted by a dt for sample instances, where an instance comprises a set of features, each associated with a value, and an outcome is a discrete class. for example, the top of like @xcite approach, ours hinges on identifying discrepancies, but it differs from their approach in that (1) we propose addressing potential conflicts as a guiding principle for selecting content that complements explanations of dt predictions; (2) these conflicts pertain to predicted outcomes and to the impact of variables; and (3) we identify these conflicts by comparing aspects of a dt prediction with plausible expectations derived from probabilistic relations. ## empirical evaluation our evaluation considers two main questions: next, we describe our datasets and classifier, followed by our experimental design and results. foot_7 ## conclusion our approach for explaining dt predictions addresses potential conflicts between aspects of these predictions and plausible expectations licensed by background information. to this effect it operationalizes the identification of four types of conflicts, and specifies schemas for generating explanations that address these conflicts. our approach is model agnostic, except for the determination of the actual impact of a variable, which is readily available in most ml models. our evaluation on the nursery and telecom datasets shows that (1) explanations addressing potential conflicts between dt predictions and plausible expectations from background information are considered at least as good as baseline explanations; and (2) the conflict-based explanations are deemed especially valuable when users' expectations disagree with dt predictions. these insights are of practical import, since users' expectations are often not available to explanation systems, and conflict-based explanations provide clear benefits, or at worst are neutral, regardless of the particulars of these expectations. our approach has the following limitations, which we propose to address in the future: (1) it does not perform feature selection to reduce long paths in a dt; (2) conflict-based explanations address only one pivot feature; and (3) the explanations omit information about dt accuracy for particular instances. our evaluation has the following limitations: (1) we cannot divorce length from explanation type, as conflict-based explanations are about twice as long as basic ones; (2) the cohorts for the two datasets had different demographics, so, given the size of our population, it is not possible to attribute differences in our results for each dataset to domain or demographic differences; and (3) we could not recruit participants with relevant experience, but in light of our narrative immersion and the general accessibility of the concepts in the explanations, we believe that our results are informative. a datasets feature value description parents' employment challenging frequent relocations, transfers, long leaves of absence; parents are not employed in the school district and need to travel more than one hour for work. somewhat difficult hard working conditions that allow for an early retirement (e.g., miners, policemen, soldiers), night work, additional work engagements. ordinary normal condition. current childcare very critical there is no possibility of childcare with family, and previous level of childcare was inadequate (child does not live with parents, problematic private care). critical there is no possibility of childcare with family, and previous level of care was less than adequate (frequent change of care, termination of care, alternate care by parents, occasional care). insufficient no possibility of childcare with family (both parents or single parent work full-time or are full-time students, no alternative care with relatives), but previous level of care was adequate (with own family, adequate private care, educational care organizations). sufficient childcare is possible with some relatives (healthy and unemployed grandparents living in the school district, other able-bodied and unemployed members of the household). good normal condition (childcare is possible in the family -father or mother unemployed and able to care). housing condition inadequate subleased or emergency housing; cramped; has lack of sanitation facilities or water. somewhat inadequate subleased or cramped apartment. adequate normal condition. social situation problematic inadequate educational ability of parents (gross neglect of education and care, violence); inadequate family relationships (serious conflicts between parents, between grandparents, between parents and grandparents, more severe forms of disturbance of parents or other family members); social and antisocial forms of restraining behavior by parents and other family members (alcoholism and other addictions, delinquency, quitting, etc). somewhat problematic less than adequate educational ability of parents (uneven, inconsistent education, excessive difficulty or indulgence, neurotic reaction of parents); less than adequate family relationships (milder forms of parental personality disorders, privileged or neglected children, family conflicts). unproblematic normal condition. child's health poor admission is not recommended due to the health conditions of the child. average the child has a mental or physical disorder that influences their admission status; the child's development is affected by health conditions of family members. good normal condition (healthy). the nursery dataset originally had five classes, three of which account for about 97% of the instances; we therefore removed the other two classes, which resulted in a balanced dataset with 12630 instances. the classes, features and feature values in the dataset were originally in slovenian, and their english translation in @xcite was somewhat peculiar. with the help of one of the authors of the original paper, we recoded the features and feature values in the nursery domain to those in the telecom dataset had only two classes, stay and churn, but it was imbalanced towards stay (73%). the dt had an accuracy of 79% when trained with a cost-sensitive setting for imbalanced datasets. this accuracy is comparable to those reported in kaggle for several predictive models. however, in order to avoid biasing participants' class expectations, we decided to even out the class distribution. to this effect, we retained only customers with a month-to-month contract, which had both outcomes, and randomly removed half of the incorrectly predicted cases. this yielded a more balanced dataset (60% stay) and a slightly improved dt accuracy of 80% (trained without the cost-sensitive setting).