Learning from Explanations and Demonstrations: A Pilot Study (2020)

Abstract: AI has become prominent in a growing number of systems, and, as a direct consequence, the desire for explainability in such systems has become prominent as well. To build explainable systems, a large portion of existing research uses various kinds of natural language technologies, e.g., text-to-speech mechanisms, or string visualizations. Here, we provide an overview of the challenges associated with natural language explanations by reviewing existing literature. Additionally, we discuss the relationship between explainability and knowledge transfer in reinforcement learning. We argue that explainability methods, in particular methods that model the recipient of an explanation, might help increasing sample efficiency. For this, we present a computational approach to optimize the learner’s performance using explanations of another agent and discuss our results in light of effective natural language explanations for humans.

## introduction the process of gaining knowledge from the interaction between individuals needs to allow a two-way flow of information, i.e., reciprocally active communication. during this process explainability is key to enabling a shared communication protocol for effective information transfer. to build explainable systems, a large portion of existing research uses various kinds of natural language technologies, e.g., text-to-speech mechanisms, or string visualizations. however, to the best of our knowledge, few works in the existing literature specifically address how the features of explanations influence the dynamics of agents learning within an interactive scenarios. interactive learning scenarios are a much less common but similarly interesting context to study explainability. explanations can contribute in defining the role of each agent involved in the interaction or guide an agent's exploration to relevant parts of the learning task. here, some of the known benefits of explanability (e.g., increased trust, causality, transferability, informativeness) can improve the learning experience in interactive scenarios. although feedback and demonstration have been largely investigated in reinforcement learning @xcite , the design and evaluation of natural language explanations that foster knowledge transfer in both human-agent and agent-agent scenarios is hardly explored. our contribution aims to optimize this knowledge transfer among agents by using explanationguided exploration. we refer to explanations as the set of information that aims to convey a causality by comparing counterfactuals in the task, i.e, providing the reward that could have been obtained if a different action would have been chosen. instead of providing the optimal solution for the task, this approach lets the learner infer the best strategy to pursue. in this work, we provide (1) an overview on the topic of natural language explanations in interactive learning scenarios, and (2) a preliminary computational experiment to evaluate the effect of explanation and demonstration on a learning agent performance in a two-agents setting. we then discuss our results in light of effective natural language explanations for both agents and humans. ## on natural language explanations in interactive learning scenarios humans use the flexibility of natural language to express themselves and provide various forms of feedback, e.g., via counterfactuals. to be successful, artificial agents must therefore be capable of both learning from and using natural language explanations; especially in unstructured environments with human presence. recent advances in groundedlanguage feedback state that, although there is a conceptual difference between natural language explanations and tuples that hold information about the environment, natural language is still a favorable candidate for building models that acquire world knowledge @xcite @xcite . along this line, training agents to learn rewards from natural language explanations has been widely explored @xcite @xcite @xcite @xcite . the interestigness of @xcite approach lays in grounding the implementation of two artificial agents on a corpus of naturalistic forms of feedback studied in educational research. the authors presented a general method that uses sentiment analysis and contextualization to translate feedback into quantities that reinforcement learning algorithms can reason with. similarly, @xcite build a training corpus of state-action pairs annotated with natural language explanations with the intent of rationalizing the agent's action or behavior in a way that closely resemble how a human would most likely do. existing literature reviews and experimental studies paired natural language feedback with demonstrations of the corresponding tasks to learn the mapping between instructions and actions @xcite . this aspect has been studied also in the context of real-time interactive learning scenarios in which the guidance and the dialog with a human tutor is often realized by providing explanations @xcite . following the idea of ai rationalization introduced by @xcite , our work approaches the generation of explanations as a problem of translation between ad-hoc representations of an agent's behavior and the shape of the reward function. on the contrary, to achieve our goal we use counterfactuals that can be easily encoded in natural language. ## experiments to operationalize the constructs discussed above, we have created an interactive learning scenario allowing both human-agent, and agent-agent interaction. we present initial results that use this interactive scenario to compare different kinds of information provided to the learner. ## discussion in above section we organized the literature on the topic of natural language in interactive learning scenarios involving humans and agents. to date, several excellent works exist on the topic of explainability and natural language technologies, but there it seems to be a gap for experimental work that aims to investigate the concept of explainable ai for transfer learning in both human-agent and agent-agent scenarios. we expected that the proposed counterfactual structure of an agent's explanations would affect the learning of another agent interacting in the same environment. overall, the data did not confirm this hypothesis. we assume that the impact of the formalization of the demonstrations and the explanations is less strong than other learning parameters. furthermore, the access to both demonstrations and the explanations might have influenced erroneously the agent's reasoning about the task. future work should consider isolating the problem of comparing different types of information employing other rationale that can be suitable, such as inverse reinforcement learning. another challenging future direction is represented by the implementation of methods that model the recipient of an explanation. inferring the learner understanding of the task through partial observations of its state would help in driving the explainer's selection of informative examples. one of the aspect we neglected in the current study is more realistic and reactive behaviors on both the part of the learner and the explainer. on this subject, while any given agent may not be an expert during learning, accounting for the explainable agency of agents that are not experts remains a topic of future work. using counterfactuals to allow agents to understand the effects of their actions seems a promising approach. however, this is not always applicable in complex environment involving humans. if we consider the hex game with a number of states of around 10 92 , generating counterfactuals in natural language might conduct to probabilistic explanations and increase mental overload, leading to performance degradation. considering a training corpus of annotated natural language explanations provided by humans appear to be a necessary requirement to extend our findings to human-agent scenarios. following the same line, testing the effect of agents' explainability on human learning requires challenging longterm studies. the evaluation framework is, in fact, an open challenge. further evaluation about the effects of the provided explanations on several metrics beyond the human's performance is needed to support our claims. ## conclusion throughout this paper, we contextualize natural language explanations with a specific focus on learning scenarios. we gave an overview of the existing literature bridging the concept of explanation in humans and artificial agents and showing that explainability is receiving attention in the context of multi-agent settings. we proposed a preliminary computational experiment for comparing demonstrations and explanations and discuss limitations and future work.