Communicating Uncertainty in Explanations of the Outcomes of Machine Learning Models (2024)

Abstract: We consider two types of numeric representations for conveying the uncertainty of predictions made by Machine Learning (ML) models: confidence-based (e.g., “the AI is 90% confident”) and frequency-based (e.g., “the AI was correct in 180 (90%) out of 200 cases”). We conducted a user study to determine which factors influence users’ acceptance of predictions made by ML models, and how the two types of uncertainty representations affect users’ views about explanations. Our results show that users’ acceptance of ML model predictions depends mainly on the models’ confidence, and that explanations that include uncertainty information are deemed better in several respects than explanations that omit it, with frequency-based representations being deemed better than confidence-based representations.

## introduction there is a large body of research on how to communicate the uncertainty associated with predicted outcomes, in particular in healthcare @xcite @xcite . in that research, the uncertainty is derived from simple historical population averages, e.g., iprevent provides such information to enable patients to assess their risk of breast cancer. however, in the age of personalised medicine, the uncertainty is obtained from the predictions of machine learning (ml) models, which are tailored to individuals by learning complex relationships between a prediction (e.g., a disease) and a large number of variables. understanding this uncertainty is essential to improve medical decision making @xcite . however, there is relatively little research on conveying the uncertainty of predictions made by ml models. in this paper, we consider two types of numeric representations for conveying the uncertainty of ml predictions: confidence and confidence+frequency (denoted %frequency). the confidence representation was proposed in @xcite to convey how certain an ai is of its prediction (e.g., "the ai is 80% confident of the predicted outcome"); and the %frequency representation, which is best practice for conveying population-based statistics in healthcare @xcite , gives a frequency out of a reference class (a base population), and the corresponding percentage. the reference class may be generic (e.g.,"out of 200 people, 160 (80%) will develop this side effect") or tailored (e.g., "out of 200 people like you, . . ."). we chose the latter, as recommended in @xcite . we describe a user study that examines (1) the influence of these two representations of uncertainty and other factors on users' acceptance of the predictions of an ml model; and (2) users' views about explanations featuring these representations of uncertainty. our study was conducted in a healthcare scenario, sourced from the busselton dataset @xcite , where an ai uses demographic, medical and lifestyle information to predict whether a person is at risk of coronary heart disease (chd). concessive-contrastive explanations for these predictions, without uncertainty information, were used as a baseline. we chose these explanations owing to their support in the literature @xcite @xcite . at-risk scenario -residentid 83: you are a 76 year old female whose weight is optimal, does not drink, but smokes 10 cigarettes a day. you also have optimal blood pressure, borderline total and hdl cholesterol, and high triglycerides. but on the upside, you are not diabetic. concessive-contrastive explanation (baseline) even though you have optimal blood pressure, the ai predicts that you are at risk of a coronary event because you are between 72 and 79 years old and have a high level of triglycerides. ## related work there has been substantial research in communicating the uncertainty associated with predicted outcomes, in particular in healthcare @xcite @xcite . most of that research con-siders how to convey probabilities derived from historical population-based statistics, focusing on modality selection (i.e., words, numbers or graphs), and within each modality, on selecting a specific format (e.g., probabilities, percentages or natural frequencies for numeric representations). @xcite demonstrated that natural frequencies are more understandable than probabilities, and that it is essential to provide a reference class. but in later review articles, @xcite and @xcite argued that both percentages and frequencies are required. these insights have informed best practice in uncertainty representations shown to patients (e.g., iprevent). research on communicating uncertainty also considered the effect of other factors on users' perceptions of risk, such as communicative intent @xcite , risk type (absolute or relative) @xcite , framing of an outcome (positive or negative) @xcite , context (e.g., information about a population at a lower risk) @xcite , and users' numeracy @xcite . our work is inspired mainly by the research of @xcite and @xcite . @xcite studied the interaction between the specificity of the reference class in frequency representations (generic versus tailored) and presentation format (words only versus words and numbers) when communicating population-based statistics. they found that patients deemed tailored risks to be less accurate and higher than generic risks when the risks were presented in words only, but not when words were combined with numbers. @xcite examined the interaction between the correctness of an ml model, the explanation style and the model's confidence in its prediction (expressed as a percentage), e.g., "the ai is 45% confident that the price will increase". the research described in this paper advances the state-of-the-art in that (1) it compares the influence of confidence and %frequency representations of uncertainty on users' acceptance of ml predictions (which differ from population-based historical predictions); (2) it considers the influence of three new factors, viz predicted outcome, size of the reference class and level of concern about a coronary event, on users' acceptance of a prediction, in addition to factors from the literature, viz confidence percentage @xcite , (dis)agreement between ai and user predictions (similar to @xcite ) and users' numeracy @xcite ; ## experimental setup we describe our dataset, the design of our user study, foot_1 our experiments and our participant cohorts. ## results we report the results for research questions rq1-rq4. statistical significance was adjusted with holm-bonferroni correction for multiple comparisons @xcite , where applicable; results with 0.05 < p-@xmath0.1 are designated as trends. ## conclusion this research focuses on the influence of uncertainty information on the acceptance of predictions made by ml models. our main contributions are: (1) determining factors that influence users' acceptance of these predictions; and (2) comparing the influence of confidence and %frequency uncertainty representations on users' views about explanations. our results show that when uncertainty information is incorporated in an explanation of the prediction of an ml model, users' likelihood of accepting the prediction is influenced by the model's confidence percentage -high percentages (90%) increase the likelihood of acceptance (compared to a baseline explanation without uncertainty information), while low percentages (65%) decrease this likelihood. this finding suggests that people may require a high level of confidence in order to increase their likelihood of accepting an ml prediction. reference class size influenced the likelihood of prediction acceptance, with a large class (200 out of 1000) increasing this likelihood and a small class (20 out of 1000) decreasing it (for the %frequency cohort). predicted outcome and (dis)agreement between ai and user predictions influenced prediction acceptance for baseline explanations (without uncertainty information), with participants being more likely to accept at-risk predictions than not-at-risk predictions, and ml model predictions that agreed with their own predictions than ml model predictions that disagreed. however, uncertainty information moderated these effects, increasing the likelihood of accepting the less-acceptable predictions and decreasing the likelihood of accepting the moreacceptable ones. users deemed explanations that include uncertainty information to be better, in terms of the four explanatory attributes, than baseline explanations that omit uncertainty information. when the two types of uncertainty representations were seen separately, users deemed them to be similar in terms of their effect on the four explanatory attributes. however, when seen together, %frequency representations were deemed to be better than confidence representations by the vast majority of users.