When to explain: Identifying explanation triggers in human-agent interaction (2020)

Abstract: With more agents deployed than ever, users need to be able to interact and cooperate with them in an effective and comfortable manner. Explanations have been shown to increase the understanding and trust of a user in human-agent interaction. There have been numerous studies investigating this effect, but they rely on the user explicitly requesting an explanation. We propose a first overview of when an explanation should be triggered and show that there are many instances that would be missed if the agent solely relies on direct questions. For this, we differentiate between direct triggers such as commands or questions and introduce indirect triggers like confusion or uncertainty detection.

## introduction the introduction of artificial agents into our daily lives means that an increasing number of lay users interact with them, often even collaborating. this has major societal implications since they are used in domains ranging from healthcare over finance to the military. as a result, special care must be taken to ensure that users understand agents' decisions, can effectively collaborate with them, and even hold them accountable if necessary. while research emphasising the need for explanations is not new @xcite , interest has picked up over the past few years @xcite . recent advances in artificial intelligence and machine learning have led to a rapid increase in quality of artificial agents. since most state-of-the-art models are black boxes, it is often not clear to the end-user why the agent made certain decisions. trust, however, relies on understanding the decision-making of the agent @xcite and trust is a prerequisite for successful collaboration and use. explanations have been shown to increase the understanding of the agent in human-agent teams @xcite and thus increase trust. within human-human interaction, people resolve conflicts or uncertainties by explaining the reasoning behind their arguments or decisions. users have a tendency to anthropomorphise agents @xcite and expect them to behave human-like; thus, they expect them to give explanations for their decisions and actions. most work assumes that the user directly asks for an explanation @xcite . we claim that there are many situations where explanations are needed, even if not explicitly requested by the user. in our work, we aim to provide an overview of direct as well as indirect explanation triggers. this overview will be the basis of designing future system experiments and evaluation metrics that target explanations to those needs. while our primary goal is to investigate this in the context of human-robot interaction, we believe that the impact of these findings is not limited solely to this domain. ## related work in this section, we review recent papers covering explainability, explanations and explanations specifically for human-agent interaction. as our focus lies on human-agent interaction we will mostly refer the reader to survey papers for the parts on explainability and explanations as they give a much more in-depth overview than what would be possible within this space. ## triggers all the work on explanations for human-robot agents mentioned before makes the assumption that the user is explicitly going to ask for an explanation and to the best of our knowledge, the question when during communication an explanation is actually needed remains unanswered. @xcite pose the question in their overview paper on explainability in humanagent systems, but only consider the task of the agent-system, not the communicative aspect or any flexible trigger detection that takes the users current state into account. we argue that it is vital to fill this gap in order to make use of the full potential of explanations for human-agent interaction, as there are many situations in which an explanation is needed, even if not explicitly requested by the user. we therefore provide a first overview of possible direct and indirect triggers. when users interact with explainable agents, the agent constantly has to evaluate whether it has to inform the user about its decisions. to do this efficiently it needs clear triggers when to explain. ## conclusion we have shown the need for detecting triggers of explanations and given a first classification of possible internal and external triggers. next steps will be to implement this classification system into an agent. further work will then correlate the triggers to specific types of explanations and their generation.