Inherently Explainable Reinforcement Learning in Natural Language
Abstract
We focus on the task of creating a reinforcement learning agent that is inherently explainable—with the ability to produce immediate local explanations by thinking out loud while performing a task and analyzing entire trajectories post-hoc to produce temporally extended explanations.
This Hierarchically Explainable
Reinforcement Learning agent (HEX-RL), operates in Interactive Fictions, textbased game environments in which an agent perceives and acts upon the world using textual natural language.
These games are usually structured as puzzles or quests with long-term dependencies in which an agent must complete a sequence of actions to succeed—providing ideal environments in which to test an agent’s ability to explain its actions.
Our agent is designed to treat explainability as a first-class citizen, using an extracted symbolic knowledge graph-based (KG) state representation coupled with a Hierarchical Graph Attention mechanism that points to the facts in the internal graph representation that most influenced the choice of actions.
Experiments show that this agent provides significantly improved explanations over strong baselines, as rated by human participants generally unfamiliar with the environment, while also matching state-of-the-art task performance.
Introduction
[Figure]
different categories of KG facts seen in Fig.
Explainable AI refers to artificial intelligence methods and techniques that provide humanunderstandable insights into how and why an
AI system chooses actions or makes predictions.
Such explanations are critical for ensuring reliability and improving trustworthiness by increasing user understanding of the underlying model.
In this work we specifically focus on creating deep reinforcement learning (RL) agents that can explain their actions in sequential decision making environments through natural language.
In contrast to the majority of contemporary work in the area which focuses on supervised machine learning problems which require singular instance level local explanations (You et al., 2016; Xu et al., 2015; Wang et al., 2017; Wiegreffe and Marasovic, 2021), such environments—in which agents need to reason causally about actions over a long series of steps—require an agent to take into account both environmentally grounded context as well as goals when producing explanations.
Agents implicitly contain beliefs regarding the downstream effects—the changes to the world—that actions taken at the current timestep will have.
This requires explanations in these environments to contain an additional temporally extended component taking the full trajectory’s context into account—complementary to the immediate step-by-step explanations.
Interactive Fiction (IF) games (Fig. 1) are partially observable environments where an agent perceives and acts upon a world using potentially incomplete textual natural language descriptions.
They are structured as long puzzles and quests that require agents to reason about thousands of locations, characters, and objects over hundreds of steps, creating chains of dependencies that an agent must fulfill to complete the overall task.
They provide ideal experimental test-beds for creating agents that can both reason in text and explain it.
We introduce an approach to game playing agents— Hierarchically Explainable Reinforcement
Learning (HEX-RL) —that is designed to be inherently explainable, in the sense that its internal state representation—i.e. belief state about the world—takes the form of a symbolic, humaninterpretable knowledge graph (KG) that is built as the agent explores the world.
The graph is encoded by a Graph Attention network (GAT) (Veliˇckovi´c et al., 2017) extended to contain a hierarchical graph attention mechanism that focuses on different sub-graphs in the overall KG representation.
Each of these sub-graphs contains different information such as attributes of objects, objects the player has, objects in the room, current location, etc.
Using these encoding networks in conjunction with the underlying world KG, the agent is able to create immediate explanations akin to a running commentary that points to the facts within this knowledge graph that most influence its current choice of actions when attempting to achieve the tasks in the game on a step-by-step basis.
While graph attention can tell us which elements in the KG are attended to when maximizing expected reward from the current state, it cannot explain the intermediate, unrewarded dependencies that need to be satisfied to meet the long term task goals.
For example, in the game zork1 , the agent needs to pick up a lamp early on in the game—an unrewarded action—but the lamp is only used much later on to progress through a location without light.
Thus, our agent additionally analyzes an overall episode trajectory—a sequence of knowledge graph states and actions from when the agent first starts in a world to either task completion or agent death—to find the intermediate set of states that are most important for completing the overall task.
This information is used to generate a temporally extended explanation that condenses the immediate step-by-step explanations to only the most important steps required to fulfill dependencies for the task.
Our contributions are as twofold: (1) we create an inherently explainable agent that uses an ever-updating knowledge-graph based state representation to generate step-by-step immediate explanations for executed actions as well as performing a post-hoc analysis to create temporal explanations; and (2) a thorough experimental study against strong baselines that shows that our agent generates significantly improved explanations for its actions when rated by human participants unfamiliar with the domain while not losing any task performance compared to the current state-of-the-art knowledge graph-based agents.
Background and Related Work
Interactive Fiction (IF) games are simulations featuring language-based state and action spaces.
It provides a platform for exploring lifelong open-domain dialogue learning (Shuster et al., 2020) and action elimination with deep reinforcement learning (Zahavy et al., 2018).
In this paper, we use IF games as our test-bed because they provide an ideal platform for collecting data, linking game states and actions to the corresponding natural language explanations.
We use the definition of text-adventure games as seen in Côté et al.
(2018) and Hausknecht et al.
We take Jericho (Hausknecht et al., 2020), a framework for interacting with text games, as the interface connecting learning agents with interactive fiction games.
A text game can be defined as a partially-observable
Markov Decision Process: G = 〈 S , P , A , O , Ω , R , γ 〉 , representing the set of environment states, conditional transition probabilities between states, the vocabulary or words used to compose text
[Figure]
House You are behind the white house.
A path leads into the forest to the east.
In one corner of the house there is a small window which is slightly ajar.
You are carrying: A jewel-encrusted egg
[Figure]
animate
[Figure]
commands, observations, observation conditional probabilities, reward function, and discount factor, respectively.
The reinforcement learning agent is trained to learned a policy π G ( o ) → a . Knowledge Graphs for Text Games. Ammanabrolu et al.
(2020) proposed Q*BERT, a reinforcement learning agent that learns a KG of the world by answering questions.
Xu et al.
(2020) introduce the SHA-KG, a stacked Hierarchical Graph Attention mechanism to construct an explicit representation of the reasoning process by exploiting the structure of the KG.
Adhikari et al.
(2020) present the Graph-Aided Transformer Agent (GATA) which learns to construct a KG during game play and improves zero-shot generalization on procedurally generated TextWorld games.
Other works such as Murugesan et al.
(2020) explore how to use KGs to endow agents with commonsense.
While these works showcase the effectiveness of KGs on task performance and do not focus on how explainable their architectures are.
We further note that these architectures do now allow for as fine-grained attention-based attribution as HEX-RL’s architecture does—e.g. Q*BERT does not use relationship information in their policy and
SHA-KG averages attention across large portions of the graph and is unable to point to specific triples in its KG representation to explain an action.
Explainable Deep RL.
Contemporary work on explaining deep reinforcement learning policies can be broadly categorized based on: (1) how the information is extracted, either via intrinsic motivation during training (Shu et al., 2017; Hein et al., 2017; Verma et al., 2018) or through post-hoc analysis (Rusu et al., 2015; Hayes and Shah, 2017; Juozapaitis et al., 2019; Madumal et al., 2020); and (2) the scope—either global (Zahavy et al., 2016;
Hein et al., 2017; Verma et al., 2018; Liu et al., 2018) or local (Shu et al., 2017; Liu et al., 2018; Madumal et al., 2020; Guo et al., 2021).
In our work, we create an agent that spans more than one of these categories providing immediately local explanations through extracted knowledge graph representations and post-hoc temporal explanations.
Inspired by Madumal et al.
(2020), we learn a graphical causal model which focuses on using relations between steps in a puzzle to generate temporal explanations instead of generating counterfactuals.
Hierarchically Explainable RL
Our work aims to generate (1) immediate step-by-step explanations of an agent’s policy by capturing the importance of the current game state observation and (2) temporally extended explanations that take into context an entire trajectory via a post-hoc analysis.
Formally, let X = { s t , a t } t = 1: T be the set of game steps that compose a trajectory.
Each game state s t consists of a knowledge graph
G t representing all the information learned since the start of the game.
This graph is further split into four sub-knowledge graphs each containing different, semantically related relationship types.
This section first describes a graph attention based architecture that uses these sub-graphs to produce immediate explanations.
We then describe how to filter the game states in a trajectory into a condensed set of the most important ones X ′ ⊂
X that best capture the underlying dependencies that need to be fulfilled to complete the task—enabling us to produce temporal explanations.
Knowledge Graph State Representation.
Building on Ammanabrolu et al.
(2020), constructing the knowledge graph is treated as a question-answering task.
KGs in these games take the form of
RDF triples (Klyne, 2004) of 〈 subject , relation , object 〉 —extracted from text observations and update as the agent explores the world.
The agent answers questions about the environment such as, “What am I carrying?”
or “What objects are around me?”.
A specially constructed dataset for question answering in text games—JerichoQA—is used to fine-tune ALBERT (Lan et al., 2019) to answer these questions (See Appendix A.3).
The answers form a set of candidate graph vertices V t for the current step and questions form the set of relations R t .
Both V t and R t are then combined with the graph at the previous step G t − 1 to update the agent’s belief about the world state into G t .
The left side of Figure 2 showcases this.
In an attempt to enable more fine grained explanation generation and inspired by Xu et al.
(2020), we divide the knowledge graph G into multiple sub-graphs G atr , G inv , G obj , G loc , each representing (1) attributes of objects, (2) objects the player has, (3) objects in the room, and (4) other information such as location (right side of Fig. 2) based on the corresponding relationship types extracted by the ALBERT-QA module.
The union of all sub-graphs is equivalent of V t and R t extracted from the current game state.
The full knowledge graph G t captures the overall game state since the start of the game.
The sub-graphs easily reflect different relationships of the current game state.
Template Action Space.
Agents output a language string into the game to describe the actions that they want to perform.
To ensure tractability, this action space can be simplified down into templates.
Templates consist of interchangeable verbs phrases ( V P ), optionally followed by prepositional phrases ( V P PP ), e.g. ([ carr y / take ] ) and ([ throw / discard / put ] [ against / on / down ] ), where the verbs and prepositions within [.] are aliases.
Actions are constructed from templates by filling in the template’s blanks using words in the game’s vocabulary.
Size of action space is shown in Appendix A.1.
3.1 Immediate Explanations
Our immediate explanations consist of finding the subset of triplets in sub-graphs
G atr , G inv , G obj , G loc c the action decision made at the current step—is capable of explaining the action.
We introduce a deep RL architecture capable of this.
Hierarchical Knowledge Graph Attention Architecture.
At each step, a total score R t and an observation o t is received—consisting of (cid:179)
o t desc , o t game , o t inv , a t − 1 (cid:180) corresponding to the room description, game feedback, inventory, and previous action and are processed using a GRU based encoder using the hidden state from the previous step, combining them into a single observation embedding o t ∈ R d text × c (bottom of Fig. 2).
The full knowledge graph G t is processed via Graph Attention Networks (GATs) (Veliˇckovi´c et al., 2017) followed by a linear layer to get the graph representation g t ∈ R d text (middle of Fig. 2).
We compute LSTM attention between o t and g t as:
[Equation]
where ⊕ denotes the addition of a matrix and a vector.
W l ∈ R d text × d text , W g ∈ R d text × d text , W o ∈
R d text × d text are weights and b l ∈ R d text , b o ∈ R d text are biases.
The overall representation vector is updated as:
[Equation]
where ⊙ denotes dot-product and c is the number of o t ’s components.
Sub-graphs are also encoded by GATs to get the graph representation g ′ t ∈ R d Gt , sub × m (no. of subgraphs).
The Hierarchical Graph Attention between q t ∈ R d Gt , sub 1 and g ′ t is calculated by:
[Equation]
where W H ∈ R d Gt , sub × d Gt , sub , W g ′ ∈ R d Gt , sub × d Gt , sub , W q ∈ R d Gt , sub × d Gt , sub are weights and b H ∈
R d Gt , sub , b q ∈ R d Gt , sub are biases.
Then we get state representation, consisting of the textual observations full knowledge graph and sub-knowledge graph.
[Equation]
where s is the number of sub-graphs (4 in our paper).
The full architecture can be found in
Figure 2.
The agent is trained via the Advantage Actor Critic (A2C) (Mnih et al., 2016) method to maximize long term expected reward in the game in a manner otherwise unchanged from Ammanabrolu et al.
(2020) (See Appendix A.2).
These attention values thus reflect the portions of the knowledge graphs that the agent must focus on to best achieve this goal of maximizing reward.
Hierarchical Graph Attention Explanation.
The graph attention α Hierarchical is used to capture the relative importance of game state observations and KG entities in influencing action choice.
For each sub-graph, the graph attention, α Hierarchical,i ∈ R n nodes × m is summed over all the channels m to obtain α ′ Hierarchical,i ∈ R n nodes × 1 , showing the importance of the KG nodes in the i th sub-graph.
The top- k valid entities (and corresponding edges) with highest absolute value of its attention form the set of knowledge graph triplets that best locally explain the action a t .
In order to make the explanation more readable for a human reader, we further transform knowledge graph triplets to natural language by template filling.
We create templates for each type of sub-graphs G atr , G inv , G obj , G loc .
• 〈 object , is , attribute 〉→ “ Object is attribute ” • 〈 player , has , object 〉→ “ I have object ” • 〈 object , in , location 〉→ “ Object is in location ” • 〈 location 1, direction , location 2 〉→ “ location 1 is in the direction of location 2 ”, e.g. 〈 f orest , nor th , house 〉 is converted to “ Forest is in the north of house ”
More examples can be found in Appendix A.4.
3.2 Temporally Extended Explanations
Graph attention tells us which entities in the KG are attended to when making a decision, but is not enough alone for explaining “why” actions are the right ones in the context of fulfilling dependencies that may potentially be unrewarded by the game—especially given the fact that there are potentially multiple ways of achieving the overall task.
HEX-RL thus saves trajectories for hundreds of test time rollouts of the games, performed once a policy has been trained (Table 1 and Appendix A.5).
The game trajectories consist of all the game states, actions taken, predicted critic values, game scores, the knowledge graphs, and the immediate step level explanations generated as previously described.
HEX-RL produces a temporal explanation by performing a post-hoc analysis on these game trajectories.
The agent then analyzes and filters these trajectories in an attempt to find the subset of states that are most crucial to achieving the task as summarized in Figure 3—then using that subset of states to generate temporal trajectory level explanations.
Bayesian State Filter.
We first train a Bayesian model to predict the conditional probability P ( A | B i ) of a game step ( A ) given any other possible game step ( B i ) in the game trajectories.
More specifically, each game step is composed of 3 elements, game state o t , action a t and the current knowledge graph G t .
The key intuition here being that state, action pairs that appear in a certain ordering in multiple trajectories are more likely to dependant on each other.
The set of game steps with the highest P ( A | B i ) is used to explain taking the action associated with game state A .
For example, “take egg” ( A ) is required to “open egg” ( B ), and P ( A | B ) = 1, hence “open egg” is used as a reason why action “take egg” must be taken first.
The initial set of game states X is filtered into X 1 by working backwards from the final goal state by finding the set of states that form the most likely chain of causal dependencies that lead to it.
Details can be found in
Appendix A.5 and A.6.
[Figure]
Language Model Action Filter.
Following this, we apply a GPT-2 (Radford et al., language model trained to generate actions based on transcripts of text games from human play-throughs to further filter out important states—known as the Contextual Action
Language Model (CALM) (Yao et al., 2020).
As this language model is trained on human transcripts, we hypothesize that it is able to further filter down the set of important states by finding the states that have corresponding actions that a human player would be more likely to perform—thus potentially leading to more natural explanations.
CALM takes into observation o t , action a t and the following observation o t + 1 , and predicts next valid actions a t
+ 1 .
In our work, we use CALM as a filter to look for the relations between a game step A and the explanation candidates B i ∈
X 1 .
We feed CALM with the prompt o A , a A , o B i to get an action candidate set.
When the two game steps A and B i are highly correlated, given o A , a A and o B i , CALM should successfully predict a B i with high probability.
The game steps B i , whose associated action a B i is in this generated action candidates set, are saved as the next set of filtered important candidate game states ( X 2 ).
[Table]
Semantic State-Action Filter.
To better account for the irregularities of the puzzle like environment, we adopt a semantic filter to obtain the final important state set X 3 .
Here, given A , B i ∈
X 2 , states are further filtered on the basis of whether one of these scenarios occurs: (1) a A and a B i contain the same entities, e.g. “ take egg ” and “ open egg ”.
(2) G A and G B i share the same entities, e.g. “lamp” occurs in both observations.
(3) A and B i occur in the same location, e.g. after taking action a A , the player enters “kitchen” and B occurs in “kitchen”.
The state has a non-zero reward or a high absolute critic value, indicating that it is either a state important for achieving the goals of the game or it is a state to be avoided.
The final set of important game states X 3 is used to synthesize post-hoc temporal explanations for why an action was performed in a particular state—as seen in Figure 1—taking into account the overall context of the dependencies required to be satisfied and building on the immediate step level explanations for each given state in X 3 .
Ablation studies pin-pointing the relative contributions of the different filters are found in Section 4.4.
We concluded that all three steps of the filtering process to identify important states are necessary for creating coherent temporal explanations that effectively take into account the context of the agent’s goals. [Table]
Evaluation
Our evaluation consists of three phases: (1) We show that HEX-RL has the comparable performance to state-of-art reinforcement learning agents on text games in Section 4.1.
Then in Section 4.2, we evaluate our immediate attention explanation model by comparing the explanations generated by HEX-RL and agents that do not use knowledge graphs (See Fig. 2 and
Section 3.1).
In Section 4.3 we compare immediate to temporal explanations, focusing on the effects that including trajectory level context when evaluating explanations in the context of agent goals.
In Section 4.4 we conduct human participant ablation study evaluating the individual contributions of the filtration pipeline for generating temporal explanations seen in Figure 3.
4.1 Task Performance Evaluation
We compare HEX-RL with four strong state-of-art reinforcement learning agents—focusing on contemporary agents that use knowledge graphs—on an established test set of 9 games from the
Jericho benchmark (Hausknecht et al., 2020).
• LSTM- A2C is a baseline that only uses natural language observations as state representation that is encoded with an LSTM-based policy network.
• KG-A2C. Instead of training a question-answering system like Q*BERT to build knowledge graph state representation, KG-A2C (Ammanabrolu and Hausknecht, 2020) extracts knowledge graph triplets from the text observations using a rules based approach built on OpenIE (Angeli et al., 2015).
• SHA-KG is adapted from Xu et al.
and uses a rules-based approach to construct a knowledge graph for the agent which is then fed into a Hierarchical Graph Attention network as in HEX-RL.
This agent separates the sub-graphs out using a rules-based approach and makes no use of any graph edge relationship information.
• Q*BERT.
Ammanabrolu et al.
(2020) uses a similar method of creating the knowledge graph through question answering but does not use the hierarchical graph attention architecture combined with the sub-graphs.
These baselines are all trained via the Advantage Actor Critic (A2C) (Mnih et al., 2016) method— further comparisons to other contemporary agents can be found in Appendix A.7.
It is also worth noting that most contemporary state of the art deep RL agents for text games use recurrent neural policy networks as opposed to transformer networks due to their improved performance in this domain.
HEX-RL Training.
We trained HEX-RL on two reward types: (a) game only , which indicates that we only use score obtained from the game as reward.
(2) game with intrinsic motivation (game and IM) , which contains an additional intrinsic motivation reward based on knowledge graph expansion as seen in Ammanabrolu et al.
(2020)—where the agent is additionally rewarded for learning more about the world by finding new facts for knowledge graph (see Appendix A.8, A. and A.10).
Table 2 shows the performance of HEX-RL and the other four baselines.
We can see that designing the HEX-RL agent to be inherently explainable through the use of Hierarchical Graph Attention and the sub-graphs improves the overall maximum score seen during training when compared to any of the other agents.
In terms of the average score seen during the final 100 episodes, HEX-RL wth intrinsic motivation outperforms all baselines with the exception of Q*BERT—there HEX-RL significantly outperforms Q*BERT on one game, is outperformed on two games, and comparable on the remaining six games.
HEX-RL thus performs comparably to other state-of-the-art baselines in terms of overall task performance while also boasting the additional ability to explain its actions.
4.2 Immediate Explanation Evaluation
Having established that HEX-RL’s performance while playing text games is comparable to other state-of-the-art agents, we attempt to answer the question of exactly how useful the knowledge graph based architecture is when generating immediate step-by-step explanations by comparing
HEX-RL to a baseline that doesn’t use knowledge graphs in a human participant study.
Two models for step-by-step explanations are compared:
• LSTM Attention explanations.
Extracts the most important substring in the observations through LSTM attention α LSTM
and then uses those words to create an explanation.
• Hierarchical Graph Attention explanations.
Extracts KG triplets most influenced the choice of actions by Hierarchical Attention α Hierarchical and then transforming them into readable language explanations through templates.
We recruited 40 participants—generally unfamiliar with the environment at hand—on a crowd sourcing platform.
Each participant reads a randomly selected subset of 10 explanation pairs (drawn randomly from a pool totaling 60 explanation pairs), generated by Hierarchical Graph
Attention and LSTM attention explanation on three games in the Jericho benchmark: zork1 , library , and balances .
We choose three games with very different structures and genres as defined in Hausknecht et al.
They each require a diverse set of action types and solutions to complete and thus provide a wide area of coverage when used as test beds for human evaluation of explanations.
Then they are given the following metrics and asked to choose which explanation they prefer for that metric:
• Confidence :
This explanation makes you more confident that the agent made the right choice.
• Human-likeness : This explanation expresses more human-like thinking on the action choice.
• Understandability : This explanation makes you understand why the agent made the choice.
Variations of these questions have been used to evaluate other explainable AI systems (eg.
Ehsan et al.
At least 5 participants give their preference for each explanation pair.
We take the majority preference from humans participants as the result.
More details are shown in Appendix B.1.
Figure 4 shows the result of the human evaluation of attention explanations.
Hierarchical graph attention explanation is preferred over LSTM attention explanation in all three dimensions.
These results are statistically significant ( p < 0.05) with fair inter-rater reliabilities.
We also observe that these three dimensions are highly, positively correlated using Spearman’s Rank Order Correlation.[-
-]
A slightly higher proportion of participants preferred the LSTM Attention explanations in the human-likeness dimension compared to the other two.
The participants preferring LSTM Attention explanation stated that they found it intuitive but often incoherent and found the Hierarchical
Graph Attention explanations to be more robotic.
LSTM attention explanations are substrings of the human-written observation and thus have the potential to be more natural sounding than the templated Hierarchical Graph Attention explanations when they are coherent enough to be understood.
The KG sacrifices a small amount of human-likeness in return for much greater overall coherence and accuracy.
KGs with Hierarchical Graph Attention give us explanations that are more easily understood and inspire greater confidence in the agent’s decisions.
Example qualitative LSTM Attention and Hierarchical Graph Attention explanations can be found in Appendix C.2.
As our system relies on graph hierarchical graph attention to generate immediate explanations, a well-trained knowledge graph representation module of the world knowledge is required.
Most cases where the agent fails to provide satisfactory immediate explanations are either when: (1) the explanation is not directly linked to the one of the facts we choose to extract from the knowledge graph, such as object/location information; and (2) due to the error of knowledge graph extraction models themselves.
4.3 Immediate vs. Temporal Explanations
Having proved the effectiveness of the knowledge graph at the immediate step-by-step explanation level, we evaluate our method of producing temporal explanations and how they compare to the immediate explanations along two dimensions: (1) coherence; and (2) explanation accuracy when taken in the context of the agent’s goals.
Participants first read a trajectory of the game combined with step-by-step immediate explanations and the game goal, then indicate how much they agree with the statements on a Likert scale of(strong disagree) to 5 (strong agree).
Here, we add two metrics from the previous study:
• Goal context : You are able to understand why the agent takes this particular sequence of actions given what you know about the goal.
• Readability : This explanation is easy to read.
Error bars indicate a confidence level of 95%.
Figure 5 shows the average scores for each question for the immediate and temporal explanations.
The temporal explanations achieve comparable performance to the immediate explanations on all metrics except for the the metric relating to goal context where they significantly out-perform the immediate explanations.
A majority of participants stated that a condensed trajectory level explanation made the goals of the agent easier to understand than reading through each step level explanation.
These results indicate that HEX-RL can generally successfully identify the most important states in a trajectory and use them to create temporal explanations that are on par with immediate explanations in terms of coherence but provide significantly more context in terms of explaining an agent’s actions with respect to its task-based goals.
Cases where the agent does not provide a temporally coherent and goal-driven explanation revolve around failures—particularly by the Bayesian State Filter—in detecting the most important states in the trajectory.
A qualitative analysis (as seen in Appendix C.3) suggests that this occurs in cases where there are a large number of branching paths that lead to the same end state.
Thus, the quality of the generated temporal explanations appears to be inversely proportional to the relative complexity of the game as measured by its branching factor.
4.4 Temporal Explanation Ablation Study
Having established the overall effectiveness of the filters in HEX-RL that create the temporal explanations, we perform pair-wise ablation studies to pinpoint the relative contributions of the different filters seen in Fig.
We first compare explanations generated using a set of important states filtered from the trajectory using the Bayes model compared to Bayes+CALM explanation.
This how applying the language model action filter affects the quality of the temporal explanations.
As before, we recruited 30 participants on a crowd sourcing platform.
Each participant reads a randomly selected subset of explanation pairs, comprised of temporal explanations filtered by Bayes and Bayes+CALM models. Figure 6a shows that after applying the CALM model to filter explanation candidates, generated explanations are significantly preferred on the “Confidence” and “Understandability” dimensions. Similarly, we then conducted another ablation study to validate the contribution of semantic filter by comparing the Bayes+CALM filtering method to the full HEX-RL using Bayes+CALM+Semantic filters.
The experiment setup is the same as the previous ablation study.
Figure 6b shows that
Bayes+CALM+Semantic performs significantly better than Bayes+CALM on all three dimensions.
We additionally observe that these three metrics are highly, positively correlated using Spearman’s
Rank Order Correlation in both of these ablation studies 3 .
When asked to justify their choices, participants indicated that the full HEX-RL system with Bayes+CALM+Semantic filters provided temporal explanations that they felt was more understandable than alternatives.
These results indicate that all three steps of the filtering process to identify important states are necessary for creating coherent temporal explanations that effectively take into account the context of the agent’s goals.
[Figure]
Conclusions
Explaining deep RL policies for sequential decision making problems in natural language is a sparsely studied problem despite a steadily growing need.
An oft given reason for this phenomenon is that deep RL methods perform better without the additional burden of being explainable.
In an attempt to encourage work in this area, we create the Hierarchically Explainable Reinforcement
Learning (HEX-RL) agent which treats explainability as a first-class citizen in its design by using a readily interpretable knowledge graph state representation coupled with a Hierarchical Graph
Attention network.
This agent is able to produce step-by-step commentary-like immediate explanations and also a condensed temporal trajectory level explanation via a post-hoc analysis.
We show that with careful design, it is possible to create inherently explainable RL agents that do not lose performance when compared to contemporary state-of-the-art agents and simultaneously are able to generate significantly higher quality explanations of actions.
A Implementation Details A.1 Full Action Space Size
Jericho provides the capability to extract game-specific vocabulary and action templates (Section 3).
These templates contain up to two blanks, so a typical game with 200 templates and aword vocabulary yields an action space of O ( TV 2 )
≈ 98 million, three orders of magnitude smaller than the 240-billion space of 4-word actions using vocabulary alone.
A.2 A2C Architecture
Further details of what is found in Figure 2.
The sequential action decoder consists two GRUs that are linked together as seen in Ammanabrolu and Hausknecht (2020).
The first GRU decodes an action template and the second decodes objects that can be filled into the template.
These objects are constrained by a graph mask , i.e. the decoder is only allowed to select entities that are already present in the knowledge graph.
Same with Ammanabrolu and Hausknecht (2020), the loss consists of template loss, object loss, value loss, actor loss and entropy loss.
The template loss given a particular state and current network parameters is applied to the decoder.
Similarly, the object loss is applied across the decoder is calculated by summing cross-entropy loss from all the object decoding steps.
Entropy loss over the valid actions, is designed to prevent the agent from prematurely converging on a trajectory.
The following hyperparameters are taken from the original paper and known to work well on text games.
[Table]
A.3 Knowledge Graph Representation QA Model
The question answering network based on ALBERT (Lan et al., 2019) has the following hyperparameters, taken from the original paper and known to work well on the SQuAD 2.0 (Rajpurkar et al., 2018) dataset.
No further hyperparameter tuning was conducted.
[Table]
A.4 Templates of Immediate Explanation
We consider four types of sub-graphs G atr , G inv , G obj , G loc , each representing (1) attributes of objects, (2) objects the player has, (3) objects in the room, and (4) other information such as location (see right side of Figure 2).
Hence, we create one template for each sub-graph with converting conjugated forms of verbs,
• 〈 object , is , attribute 〉 is converted to “ Object is attribute ”.
For example, 〈 trees , is , interactable 〉 is converted to “ trees are interactable. ”.
〈 egg , is , interactable 〉 is converted to “ egg is interactable ”.
• 〈 player , has , object 〉 is converted to “ I have object ”.
For example, 〈 player , has , eggs 〉 is converted to “ I have eggs ”.
〈 player , has , kni f e 〉 is converted to “ I have knife ”.
• 〈 object , in , location 〉 is converted to “ Object is in location ”.
For example, 〈 egg , in , f orest 〉 is converted to “ egg is in forest ”.
〈 trees , in , f orest 〉 is converted to “ trees are in forest ”.
• 〈 location _1, direction , location _2 〉 is converted to “ location_1 is in the direction of location_2 ”.
For example, 〈 f orest , nor th , house 〉 is converted to “ forest is in the north of house ”.
A.5 HEX-RL explanation parameters
There are some other parameters which may affect the results of explanations.
Our initial experiments suggested that the larger the number of trajectories we use, the more accurate the Bayesian
State Filter.
[Table]
time rollouts of the games, which is performed once a policy has been trained
[Table]
topk of CALM
Topk
[Table]
model A.6 Bayesian State Filter Detail
We first train a Bayesian model to predict the conditional probability P ( A | B i ) of a game step ( A ) given any other possible game step ( B i ) in the game trajectories.
More specifically, current game step ( A ) is composed of 3 elements, game state o t , action a t and knowledge graph G t .
We count the occurrence of A ( C ( A )), and all the game steps occurred in the game logs ( C ( B i )), and also count the co-occurrence of A and B i , C ( A ∩ B i ) in the same trajectory.
X
=
{ A , B 1 ,..., B i }.
The conditional probability P ( A | B i ) is calculated by,
[Equation]
(7) where C ( A ) and C ( B i ) stand for the raw count of game step A and B i in the collected trajectories.
The key intuition here being that state, action pairs that appear in a certain ordering in multiple trajectories are more likely to dependant on each other.
Higher P ( A | B i ) indicates the necessity of A to B .
The set of game steps with the highest P ( A | B i ) is used to explain taking the action associated with game state A .
For example, “take egg” ( A ) is required to “open egg” ( B ), and P ( A | B ) = 1, hence “open egg” is used as a reason why action “take egg” must be taken first.
The initial set of game states X is filtered into X 1 by working backwards from the final goal state by finding the set of states that form the most likely chain of causal dependencies that lead to it.
As shown in Figure 3, we obtain the explanation candidate game steps X 1 by filtering all the possible game steps following current game step A in the game logs with P ( A | B i ) > p , where p is the threshold.
A.7 Raw scores across Jericho supported games
We present results on and IM reward
[Table]
A.8 Reward types
To alleviate the issue that rewards are sparse and often delayed, Ammanabrolu et al. defined an intrinsic motivation for the agent that leverages the knowledge graph being built during exploration.
The motivation is for the agent to learn more information regarding the world and expand the size of its knowledge graph.
They formally define game _ and _ IM reward in terms of new information learned.
[Equation]
where KG global = t − 1 (cid:83) i = 1 KG
i
Here KG global is the set of all edges that the agent has ever had in its knowledge graph and the subtraction operator is a set difference.
A.9 HEX-RL Architecture Hyperparameters
The additional hyperparamters used for training HEX-RL are detailed below, same with (Ammanabrolu et al., 2020).
graph dropout and mask dropout are used for encouraging graph network to actually learn a sparse representation.
Parameters
[Figure]
A.10 Task Performance
We plot the training reward curve for 9 games in Figure 7 and Figure 8.
Reward curves are shown until they reach asymptotic performance, i.e. the number of steps until the score no longer increases.
All agents were trained with the same number of steps (100,000) and the results at the end of this training is what is reported in Table 2.
[Figure] [Figure]
A.11 Immediate Explanation Evaluation
We plot the immediate explanation evaluation result per game in Figure 9.
[Figure]
B Human Evaluation Details B.1 Immediate Explanation Evaluation
We firstly ask participants to read an interactive game description and then ask them to answer a set of questions about this game to make sure they are qualified.
They will also play a demo of an interactive text game and answer a question based on the game they played.
The details can be found in Figure 10 and Figure 11.
These questions are designed to improve the quality of human evaluation.
At least 5 participants give their preference for each explanation pair.
Each participant reads a randomly selected subset of 10 explanation pairs (drawn randomly from a pool totaling 60 explanation pairs), generated by Hierarchical Graph Attention and LSTM attention explanation on three games in the Jericho benchmark, zork1 , library , and balances .
The following three questions are asked,
• Which explanation makes you more confident that the agent made the right choice?
• Which explanation expresses more human-like thinking on the action choice?
• Which explanation makes you understand why the agent made the choice?
B.2 Immediate vs. Temporal Explanation Evaluation
Participants first read the full trajectory of the game (Figure 13) combined with step-by-step immediate explanations, along with summary of the game goal, and indicate how much they agree with the five statements on a Likert scale (Figure 14).
The following five statements are used in human study.
• I am confident that I can get the same score as the agent when following this explanation.
• This explanation look like it was made by human.
This explanation is easy to understand.
• I am able to understand why the agent takes this particular sequence of actions given what I know about the goal.
This explanation is easy to read.
At least 5 crowd workers rated each explanation.
We also plot the causal explanation ablation study result per game in Figure 15 and Figure 16.
[Figure] [Figure]
C Explanations Analysis
In this section we provide descriptions of each of the games as well as qualitative samples of trajectories in the format shown to the human participants.
We pick out two types of examples where the HEX-RL explanations of the trajectories were rated highly by the human participants and rated poorly in terms of the quality of the explanations—attempting to analyze the failure cases of our method.
C.1 Text Game Descriptions
We list the descriptions 4 of our selected text games for a better undestanding of our explanations and game trajectories in Appendix C.2 and C.3.
zork1 :
Many strange tales have been told of the fabulous treasure, exotic creatures, and diabolical puzzles in the Great Underground Empire.
As an aspiring adventurer, you will undoubtedly want to locate these treasures and deposit them in your trophy case.
Zork creates a wondrous, magical realm that is a veritable feast for the imagination.
You find that you have stumbled upon the ancient ruins of a vast empire lying far underground.
Yes, you will find many more treasures for your trophy case.
But to do so, you will have to search far and wide, solve diabolical puzzles, and defend your treasures (and yourself!) from a few very nasty characters... and one monster, a vicious GRUE that lurks in the dark! 2.
Library : Escape the library using knowledge and objects found in and about the library.
Balances : Balances is a relatively short, old-fashioned puzzle game set in the world of the Enchanter series and riffing on Spellbreaker in particular: the player must find scrolls, learn their spells, and cast them in order to collect white cubes.
It shares a number of design characteristics with those games: rooms represent fairly large open spaces, there are more animal NPCs than humans to interact with, and a loose, playful approach to world-building means that the various areas don’t have a great deal to do with one another.
C.2 Immediate Explanations Analysis
Table 4 shows some example trajectories with its immediate explanations where Hierarchical
Graph Attention Explanations did a better job than LSTM Attention.
We can see when the player interacts with objects or the subset of sub knowledge graph can explain the action, Hierarchical
Graph Attention is able to present a high-quality immediate explanation.
Table 5 shows some example trajectories with its immediate explanations where HEX-RL fails to produce correct immediate explanations.
We see that when there is no clear clue in the observations, i.e. the potential gold truth explanation is does not belong knowledge graph, HEXRL fails to produce immediate explanations.
Hence, we conclude that the performance of immediate explanations is limited: (1) to the cases in which explanation is directly linked to the one of the facts we choose to extract from the knowledge graph, such as object/location information; and (2) by the relative error of knowledge graph extraction models themselves.
C.3 Temporally Extended Explanations Analysis
Table 6, 7 and 8 show example game trajectories with its corresponding temporally extended explanations for game zork1 , library and balances .
HEX-RL finds the subset of states that are most crucial to achieving the task goals.
We also show one trajectory where HEX-RL fails to produce a high-quality temporally extended explanations in zork1 .
Here, in order to achieve the goal “open window”, the player has to get to the location “North of House”.
However, there exists too many different combinations of actions, i.e. branching paths, which the player can take to go to “North of House”.
Bayesian State Filter thus fails to find the most essential steps for achieving this goal.
The performance of the temporally extended explanation system is hindered by the complexity of the game—the more possible ways of achieving the same overall goal, the harder it is to generate an explanation for why the agent has chosen to perform an action to achieve the goal.
[Table]
You are about 10 feet above the ground nestled among some large branches.
The nearest branch above you is above your reach.
Beside you on the branch is a small birds nest.
In the birds nest is a large egg encrusted with precious jewels, apparently scavenged by a childless songbird.
The egg is covered with fine gold inlay, and ornamented in lapis lazuli and motherofpearl.
Unlike most eggs, this one is hinged and closed with a delicate looking clasp.
The egg appears extremely fragile.
[Table]
is
[Table]
large egg encrusted with precious jewels
[Table]
The shelves are nearly bare, although there is a complete set of the "New ork Times", a box labeled "Avalon" [Table] and several biographies of various computer game authors.
The door out is to the north You can see a copy of "Debt of Honor" here
[Table]
The shelves are nearly bare.
[Table] [Table]
[Table]
KitchenYouareinthe kitchen of the white house.
A table seems to have been used recently for the preparation of food.
A passage leads to the west and a dark staircase can be seen leading upward.
A dark chimney leads down and to the east is a small window which is open.
On the table is an elongated brown sack, smelling of hot peppers.
Sitting on the kitchen table is A glass bottle The glass bottle contains A quantity of water
[Table]
Youarebehind the white house.
A path leads into the forest to the east.
In one corner of the house there is a small window which is open
[Table]
You are facing the north side of a white house.
There is no door here, and all the windows are boarded up.
To the north a narrow path winds through the trees.
[Table]
Thisisapath winding through a dimly lit forest.
The path heads northsouth here.
One particularly large tree with some low branches stands at the edge of the path.
[Table]
Youareabout 10 feet above the ground nestled among some large branches.
The nearest branch above you is above your reach.
Beside you on the branch is a small birds nest.
In the birds nest is a large egg encrusted with precious jewels, apparently scavenged by a childless songbird.
The egg is covered with fine gold inlay, and ornamented in lapis lazuli and mother of pearl.
Unlike most eggs, this one is hinged and closed with a delicate looking clasp.
The egg appears extremely fragile.
Action [Table] take egg
[Table]
This is a path winding through a dimly lit forest.
The path heads northsouth here.
One particularly large tree with some low branches stands at the edge of the path.
[Table]
Youareabout 10 feet above the ground nestled among some large branches.
The nearest branch above you is above your reach.
Beside you on the branch is a small birds nest.
In the birds nest is a large egg encrusted with precious jewels, apparently scavenged by a childless songbird.
The egg is
[Table] [Table]
cavernous room is filled with shelves
[Table]
far
[Table]
eye
[Table]
doorway to
[Table]
east is labelled
[Table]
stairwell lies to the north
[Table]
The door
[Table]
unlocked
[Table]
shelves
[Table]
nearly bare
[Table]
ork Times
[Table]
box
[Table]
Avalon
[Table]
game authors
[Table]
door out
[Table]
to
[Table]
[Table]
The
grasslands sway over
low
hills in all
directions
it is a peaceful wilderness, broken only by this hut and a faint path to
[Table]
A pleasant pocket valley in the grassy hills, through which a trail runs northtosouth
There is a chestnut horse here, munching on a pile of oats.
[Table] [Table]
D Broader Impacts
The ability to explain the reinforcement learning actions in text games has downstream applications beyond understanding how to play text games.
Text games are regarded as simplified analogues for systems capable of long-term dialogue with humans, such as in assistance with planning complex tasks, and also discrete planning domains such as logistics.
Our work is applicable to provide understanding of domains where change in the world is affected via language.
Systems prone to error, both in performing a given task as well as in explaining them, should not be relied upon in more critical applications.