Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems. Explainable artificially intelligent (XAI) systems form part of sociotechnical systems, e.g., humanAI teams tasked with making decisions. Yet, current XAI systems are rarely evaluated by measuring the performance of humanAI teams on actual decisionmaking tasks. We conducted two online experiments and one inperson thinkaloud study to evaluate two currently common techniques for evaluating XAI systems: (1) using proxy, artificial tasks such as how well humans predict the AIu0027s decision from the given explanations, and (2) using subjective measures of trust and preference as predictors of actual performance. The results of our experiments demonstrate that evaluations with proxy tasks did not predict the results of the evaluations with the actual decisionmaking tasks. Further, the subjective measures on evaluations with actual decisionmaking tasks did not predict the objective performance on those same tasks. Our results suggest that by employing misleading evaluation methods, our field may be inadvertently slowing its progress toward developing humanAI teams that can reliably perform better than humans or AIs alone.

Leveraging Rationales to Improve Human Task Performance. Machine learning (ML) systems across many application areas are increasingly demonstrating performance that is beyond that of humans. In response to the proliferation of such models, the field of Explainable AI (XAI) has sought to develop techniques that enhance the transparency and interpretability of machine learning methods. In this work, we consider a question not previously explored within the XAI and ML communities: Given a computational system whose performance exceeds that of its human user, can explainable AI capabilities be leveraged to improve the performance of the human? We study this question in the context of the game of Chess, for which computational game engines that surpass the performance of the average player are widely available. We introduce the RationaleGenerating Algorithm, an automated technique for generating rationales for utilitybased computational methods, which we evaluate with a multiday user study against two baselines. The results show that our approach produces rationales that lead to statistically significant improvement in human task performance, demonstrating that rationales automatically generated from an AIu0027s internal task model can be used not only to explain what the system is doing, but also to instruct the user and ultimately improve their task performance.