On Evaluating Explanation Utility for Human- AI Decision Making in NLP (2024)

Abstract: Is explainability a false promise? This debate has emerged from the insufficient evidence that explanations help people in situations they are introduced for. More human-centered, application-grounded evaluations of explanations are needed to settle this. Yet, with no established guidelines for such studies in NLP, researchers accustomed to standardized proxy evaluations must discover appropriate measurements, tasks, datasets, and sensible models for human-AI teams in their studies. To aid with this, we first review existing metrics suitable for application-grounded evaluation. We then establish criteria to select appropriate datasets, and using them, we find that only 4 out of over 50 datasets available for explainability research in NLP meet them. We then demonstrate the importance of reassessing the state of the art to form and study human-AI teams: teaming people with models for certain tasks might only now start to make sense, and for others, it remains unsound. Finally, we present the exemplar studies of human-AI decision-making for one of the identified tasks — verifying the correctness of a legal claim given a contract. Our results show that providing AI predictions, with or without explanations, does not cause decision makers to speed up their work without compromising performance. We argue for revisiting the setup of human-AI teams and improving automatic deferral of instances to AI, where explanations could play a useful role.

## introduction decision makers can make use of imperfect models if they can detect when they are correct. explanations of individual predictions are proposed to this end as they are expected to reveal useful signals about the model's reasoning process (jacovi et al., 2021). before undertaking realistic evaluations involving people, nlp researchers aspired to first implement working methods. thus, prior nlp explainability work has mostly focused on overcoming technical challenges and used proxy evaluations. consequently, human-centered evaluations of explanations grounded in real nlp applications are scarce. there is a prevailing perspective that this now needs to change since explainability methods passed proof-of-concept tests (see human-centered evaluations of explanations tutorial). however, given that this is a nascent nlp research space and the notable variation among prior studies (see an existing resource for the development and evaluation of explanations in nlp already includes over 50 datasets. can these be used for applicationgrounded evaluations of explanations? to answer this, in §3, we establish criteria to assess each dataset's suitability for computing explanation usefulness with the measurements overviewed in §2. we discover that 17/53 datasets are apt for studying appropriate reliance and complementary human-ai team performance but only involve low risk. 4/53 additionally involve higher risk and do not have quality concerns. we recommend using these four, as high-stakes scenarios benefit more from effective explanations. we introduce the final criterion for dataset selection based on how the likelihood of hazards, and therefore risk, changes if the performance of a state-of-the-art model peaks or is so low the model cannot be collaborated with. we show that this criterion requires continuous assessment of model performance in a rapidly evolving field like nlp. finally, we present explanation usefulness studies for a task identified by our meta-analysis: verifying a legal claim given a contract. these serve as exemplars to nlp researchers planning similar studies. we use both the common human-ai setup, where people make all final decisions with ai assistance, and an overlooked setup, where people decide only for those referred to them by a deferral model. we isolate the effect of explanations, use strong baseline conditions, deploy multiple cognitive forcing functions, integrate real-world situations, and implement attention checks tailored to the application. we show that input highlights and influential training examples do not improve human decision making assisted with the model's predictions and confidence. however, we find that major advances are needed for functional human-ai teams in nlp, regardless of explanations. we discover that successful deferral is a promising direction to this end. ## review of application-grounded explanation evaluation in there is a notable variation in the choice of models explained, explanations used, evaluation measurements, baseline conditions, datasets, and outcomes among them. to conclusively establish, or disprove, the value of explanations for human-ai decisionmaking in nlp, more research is needed together with a more rigorous evaluation protocol designed to collectively guide us towards settling this matter. to assist with running such studies, we start with an overview of suitable explanation usefulness measurements as they are not yet widely adopted in nlp research. taxonomy of explanation evaluation. doshi-velez and kim (2017) categorize evaluations of explanations as: (1) proxy (no humans, proxy tasks; e.g., the proportion of all features selected as important), (2) human-grounded (with humans, but simplified tasks; e.g., simulatability), or (3) application-grounded (with humans, realistic tasks; e.g., human-ai decision making). human-ai decision-making, which is the focus of this paper, is one of the six usage contexts within explainable ai (liao et al., 2022). forward and counterfactual simulatability, that are common in nlp (xie et al., 2022; arora et al., 2022), are human-but not application-grounded; buçinca et al. (2020) show that explanations affect simulatability and human-ai decision-making differently. reliance definitions. it is often asserted that explanations can deter people from rejecting correct predictions, i.e., underreliance. this expectation stems from assuming that the model is correct for the right reasons, and explanations are anticipated to unveil this. explanations could also aid people in rejecting incorrect predictions, thereby countering overreliance. this becomes possible when explanations present information that appears illogical, self-contradictory, or inconsistent with what the person already knows. the ultimate goal is appropriate reliance -have people accept correct predictions and dismiss erroneous ones. a gain in the average rate at which people do so upon seeing explanations quantifies their usefulness. fok and weld (2023) define desired reliance behavior based on expected performance, which implies it is fine to accept a "super-human" model's predictions (or reject a "sub-human" model's predictions) even if some are wrong (correct). ## analysis of task appropriateness in this section, we present criteria that can be used to determine the suitability of tasks for application-grounded human evaluations of explanations ( §3.1) and analyze 53 existing datasets introduced for developing and evaluating explanations in nlp ( §3.2). we refer to a task as its realization in the data. ## study i: people make all decisions in §2, we overview two strategies for human-ai decision-making: (1) people make all final decisions with ai assistance, and (2) a deferral model refers only a fraction to people. we aim to provide an exemplar for user study design and initial insights into the usefulness of explanations in improving human-ai decision making in both of these setups for one of the tasks identified in our meta-analysis: verifying claims based on the con-tractnli non-disclosure agreements (ndas). in this section, we use (1), and in §5, we look at (2). ## study ii: people decide only for deferred instances this human-ai decision making setup defers a fraction of all instances to experts. we investigate explanation usefulness for finetuning/prompting llms to defer, and to human decision makers. ## conclusions we provide guidelines for specifying a sound experimental setup for application-grounded evaluation of explanations in nlp. the dataset selection criteria we set should be used to determine a dataset's suitability for such evaluations. future studies should consider 4 datasets we identify by validating over 50 datasets against our criteria, or validate/create new datasets guided by them. we show that the performance of a resourceappropriate state-of-the-art model should be continuously reassessed: it should neither peak nor be low, otherwise teaming the model with people is unsound. we recommend following our user study design that isolates the effect of explanations, uses strong baseline conditions, deploys multiple cognitive forcing functions, integrates real-world situations, implements attention checks tailored to the application, among other things. we believe that the main reason ai assistance is unhelpful to specialists, regardless of explanations, is the difficulty of making the right decision without investing enough effort into a given problem. yet, more effort makes teaming with ai unnecessary. we argue that integrating deferral could be a more viable approach to speeding up work without sacrificing overall performance.