Generating Impact and Critique Explanations of Predictions made by a Goal Recognizer (2025)

Abstract: In this paper, we generate two types of explanations, Impact and Critique, of predictions made by a Goal Recognizer (GR) – a system that infers agents’ goals from observations. Impact explanations describe the top-predicted goal(s) and the main observations that led to these predictions. Critique explanations augment these explanations with evidence that challenges the GR’s predictions if so warranted. Our user study compares users’ goal-recognition accuracy for Impact and Critique explanations, and users’ views about these explanations, under three prediction-correctness conditions: correct, partially correct and incorrect. Our results show that (1) users stick with a GR’s predictions, even when a Critique explanation highlights its flaws; yet (2) Critique explanations are deemed better than Impact explanations in most respects.

## introduction goal recognition is a field that infers agents' goals from observations. foot_0 sample applications include games @xcite and autonomous vehicles @xcite . goal recognizer (gr) predictions are characterized by the existence of a partial order between observations, i.e., in general, certain actions can be performed only after some other actions, which differs from feature independence or correlational dependencies between features in machine learning (ml). in this paper, we offer two approaches for generating explanations of the predictions of grs; and compare their effect on users' goal recognition accuracy (how accurately they assign likelihoods to goals) and users' views of the explanations. our explanation-generation approach follows @xcite 's human-centered view, whereby an explanation is "not about the model, but about the evidence that led to the prediction". our explanations are aimed at non-expert users wishing to understand the reasons for a prediction. specifically, we offer domain-agnostic algorithms for generating two types of explanations: xai systems generally do not vary the type of explanation they generate, irrespective of whether the ml model produced correct or wrong predictions, leaving it to users to discern between these options. in contrast, the critique explanations generated in this research differ for predictions that seem plausible and predictions that require further scrutiny. in our evaluation, we distinguish between three levels of gr prediction correctness (correct, partially correct and incorrect) and three explanatory conditions (prediction only, impact and critique), and examine their influence on users' goal recognition accuracy and opinions about the explanations. for the latter we employ six explanatory attributes from @xcite . our results show that users stick with the gr's prediction, even when a critique explanation highlights its flaws; yet critique explanations are deemed better than impact ones in most respects. our main contributions are: (1) two explanation types with algorithms that select goals and observations to be mentioned; (2) an evaluation approach that distinguishes between different levels of prediction correctness; and (3) results about the effect of explanations on users' goal recognition accuracy and about users' explanation-type preferences. this paper is organized as follows. section 2 discusses related work. our demonstration domain appears in section 3, and our explanation-generation algorithms in section 4. section 5 describes our user study, followed by its results in section 6. section 7 presents concluding remarks. we focus on explainable goal recognition (xgr) which generates explanations for gr predictions @xcite . even though the number of gr applications is growing, e.g., traffic monitoring @xcite , human-robot interaction @xcite , games @xcite and autonomous vehicles @xcite , generating explanations for gr predictions has gathered relatively little attention. a notable exception is the autonomous vehicle domain, where several models have been developed to explain the predictions of grs regarding the intentions of other vehicles @xcite @xcite . these xgr models leverage gr transparency to extract intuitive explanations from a gr's internal features -an approach that makes sense in this high-stakes domain, but may not be feasible for every domain. hence, it seems advisable to develop model-agnostic xgr approaches which treat grs as a black box. goal recognition resembles time-series classification (tsc) in that both process observations over time, and yield a distribution over predicted outcomes. however, unlike goal recognition, an entire distribution must be explained for tsc, which is often done using heatmaps @xcite . feature attribution methods explain how individual features influence ml model predictions by distributing importance across features @xcite , which aligns with human-centered xai goals @xcite . while some feature-attribution methods can handle correlational dependencies between features @xcite , the dependencies in planning, and hence in goal recognition, are structural (e.g., preconditions, goal hierarchies). this motivates our critique method, which explicitly accounts for such dependencies. @xcite adapted feature attribution to goal recognition, identifying one key observation supporting or opposing a goal g or g ′ to address "why g?" or "why not g ′ ?" questions respectively; they also generate counterfactual explanations that suggest alternative actions leading to different predictions. our work differs from @xcite 's in several key respects: (1) we automatically identify top-ranked goals (section 4.1), removing the need for users to specify goals initially; (2) we avoid the potential introduction of bias during observation selection, and determine which observations to mention -often more than one (section 4.2); and (3) we introduce evidence-based critique explanations that highlight potential inconsistencies between the gr's predictions and domain knowledge (sections 4.3 and 4.4). ## domain and dataset we use the rovers dataset @xcite to demonstrate our algorithms. this dataset contains (1) a set of missions (goals) attempted by planetary rovers, which require different types of data from specific waypoints (locations); (2) rovers with possibly different capabilities (e.g., collecting rocks or soil, traveling between specific waypoints), who collaborate to achieve a mission; and (3) a sequence of observations (which may miss some entries) of the actions performed by the rovers and their waypoints. given a sub-sequence of observations, the gr predicts which mission(s) are being attempted by the rovers. ## generating explanations to generate impact explanations, we select toppredicted goals and identify observations with the highest impact on these predictions. these are then fed into explanatory templates. additionally, critique explanations require evidence that supports or challenges a gr's predictions. this evidence is according to the ai, missions #2 and #6 are the most likely (probabilities of 23.7% and 22.5%, respectively). the observation that most influenced this result was: critique explanation according to the ai, missions #2 and #6 are the most likely (probabilities of 23.7% and 22.5%, respectively). the observation that most influenced this result was: up to now: ## experimental setup we conducted a user study to address the following research questions: rq1. do explanations help users attain a better goal recognition accuracy than no explanations? if so, which explanation type is better? rq2. which independent variables influence users' performance? rq3. what are users' views of impact and critique explanations? we consider the extent to which an explanation is liked @xcite , and five up to now: ## results we applied the following procedures to adjust statistical significance for multiple comparisons. for rq1 and rq2, we used tukey's honestly significant difference (hsd) @xcite for comparisons among estimated means. for rq3, we used the holm-bonferroni correction @xcite . results with 0.05 < p-@xmath0.1 after adjustment are designated as trends. rq1 and rq2. we compare participants' goal recognition accuracy under impact, critique and none explanatory conditions for the three correctness levels of gr predictions. the only statistically significant results were for the average likelihoods assigned to correct goals (l cor ) for the critique explanatory condition compared to none (betweensubjects critique cohort and within-subject cohort) for the correct and partially correct scenarios (first two rows of it is worth noting that critique explanations for correct predictions only add one (essentially summary) sentence to the corresponding impact explanations: "all the observations contribute to missions x and y , which are the most likely missions". thus, it is surprising that critique explanations statistically significantly increased l cor for the correct and partially correct scenarios, while impact explanations had no statistically significant effects. we also control for the interaction between time spent and ncs score and each explanatory condition, and for the effect of scenario group (presentation order had no effect, section 5.4). in the between-subjects experiment, critique explanations were deemed more helpful than impact explanations for assessing mission likelihood and judging when to trust the gr for partially correct gr predictions, and more complete for incorrect predictions (p-value ∼ 0.05). however, when participants in the within-subject experiment directly compared the two types of explanations, the critique explanations were deemed better than impact explanations in terms of all explanatory attributes (p-@xmath3.05) except for presence of extraneous information, for which they were equivalent. these findings, like those in @xcite , are not surprising, as when each cohort saw only one type of explanation, it was deemed adequate, but when the explanations were seen side-by-side, critique explanations were preferred. this result may be explained by the observations in @xcite , whereby users generally prefer longer explanations (critique explanations are markedly longer than the corresponding impact explanations for partially correct and incorrect predictions), though in our case, the content of these explanations was not absorbed. ## discussion we presented domain-agnostic algorithms that generate impact and critique explanations for predictions made by a gr. our algorithms treat the gr as a black box, but derive information from domain knowledge to generate critiques of predictions. as mentioned in section 1, xai systems generally do not vary the type of explanation they generate, irrespective of whether the ml model produced correct or wrong predictions. in contrast, our approach distinguishes between explanations of predictions that seem plausible and predictions that require further scrutiny. our algorithms were tested in several domains (e.g., blocks world and sokoban), demonstrating their generalizability. however, our user study was restricted to the rovers domain, owing to budgetary constraints. in this study, we evaluated our explanations in terms of their effect on users' goal recognition accuracy, considering the influence of five other independent variables; and assessed these explanations in terms of six explanatory attributes. critique explanations led participants to increase the likelihood of correct goals for correct and partially correct predictions, but did not affect the likelihood assigned to incorrect goals. also, additional time spent increased the likelihood assigned to correct goals for critique explanations, while a higher ncs score increased the likelihood assigned to correct and incorrect goals for these explanations. despite their lack of impact with respect to incorrect goals, users viewed critique explanations favourably, when compared directly with impact explanations, in terms of five explanatory attributes. this result calls into question evaluations that are based solely on such attributes, without considering task performance. the results of our user study could be affected by our critique explanations not being critical enough, or by the crowdworkers we recruited not being sufficiently engaged with the experiment (or by recruiting crowdworkers at all @xcite ). unfortunately, we could not recruit real users who would be engaged with our rovers domain, which is a well-known problem in nlg evaluation. to address this limitation, we are supplementing our study with think-aloud sessions involving a small group of highly engaged participants. finally, it is worth noting that explanations may become unwieldy when many goals or observations are selected to be mentioned. this suggests another avenue of investigation, which involves interacting with users @xcite .