Explainable Reinforcement Learning via Model Transforms
Abstract
Understanding the emerging behaviors of reinforcement learning agents may be difﬁcult because such agents are often trained using highly complex and expressive models.
In recent years, most approaches developed for explaining agent behaviors rely on domain knowledge or on an analysis of the agent’s learned policy.
For some domains, relevant knowledge may not be available or may be insufﬁcient for producing meaningful explanations.
We suggest using formal model abstractions and transforms, previously used mainly for expediting the search for optimal policies, to automatically explain discrepancies that may arise between the behavior of an agent and the behavior that is anticipated by an observer.
We formally deﬁne this problem of Reinforcement Learning Policy Explanation (RLPE), suggest a class of transforms which can be used for explaining emergent behaviors, and suggest methods for searching efﬁciently for an explanation.
We demonstrate the approach on standard benchmarks.
Introduction
A major limitation of AI models is the performance-transparency trade-off Rudin (2019).
As the inner workings of a model increase in complexity, it becomes more powerful, but the process through which it makes decisions becomes harder to understand.
Accordingly, interest in Explainable Artiﬁcial
Intelligence and the development of transparent, interpretable AI models has increased rapidly in recent years Arrieta et al.
This trend is particularly prevalent in reinforcement learning (RL), and deep reinforcement learning (DRL), where an agent autonomously learns how to operate in its environment.
While RL has been successfully applied to solve many challenging tasks, including trafﬁc control Arel et al.
(2010), robotic motion planning Kober, Bagnell and Peters (2013), board games Silver et al.
; Schrittwieser et al.
(2020), and more, it is becoming more and more challenging to explain the behavior of RL agents, especially when they do not operate as anticipated.
Therefore, in order to allow humans to collaborate with them effectively, it is important to develop methods for reasoning and explaining the agents’ behaviors.
While there has been a variety of recent works on explainability of DRL (see Puiutta and Veith (2020) for a recent survey), most of these methods do not exploit the full formal model of the environment, typically modeled as a Markov Decision Process (MDP), but instead focus on one element of the model (e.g., the agent’s reward function Juozapaitis et al.
or reasoning about the structure of the underlying network Greydanus et al.
In this work, we suggest a novel approach to explainability that makes use of formal transforms of the underlying environment to automatically generate explanations of emerging behavior.
As is common in most RL work, we assume the stochastic environment is modeled as a Markov Decision Process (MDP) Bertsekas (2005).
This makes it possible to generate explanations without relying on domain knowledge. Instead, it uses transforms (abstractions) used so far in the literature to expedite RL and planning solutions Bonet and Geffner (2005);
Li, Walsh, and Littman (2006); Abel et al.
Yoon et al.
While for planning, the beneﬁt of such transforms is in highlighting the features of the environment that inﬂuence the solution quality and time, we use them to isolate features of the environment that cause an agent to deviate from a behavior that is anticipated by an observer.
We consider an explainability setting, which we refer to as Reinforcement Learning Policy Explanation (RLPE), as comprised of three entities.
The ﬁrst entity, the actor , is an RL agent that seeks to maximize its accumulated reward in the environment.
The second entity, the observer , expects the actor to behave in some way and to follow a certain policy, which may differ from the one actually adopted by the actor.
We refer to this policy as the anticipated partial policy , which speciﬁes which actions an observer expects the actor to perform in some set of states.
Finally, the explainer is an entity that has access to the description of the environment, to the approach (algorithm) used by the actor to compute its policy, to the anticipated partial policy, and to a set of MDP transforms.
The explainer seeks a sequence of transforms to apply to the environment so that the actor’s policy in the transformed environment aligns with the observer’s anticipated policy.
Example 1 To demonstrate RLPE, consider Figure 1, which depicts a variation of the Taxi domain
Dietterich (2000).
In this setting, the actor represents a taxi that operates in an environment with a single passenger.
The taxi can move in each of the four cardinal directions, pick up, and drop off the passenger.
The taxi incurs a small cost for each action it performs in the environment, and gains a high positive reward for dropping off the passenger at her destination.
There are walls in the environment that the taxi cannot move through.
The observer has some partial view of the environment and actor’s model.
She knows which actions the taxi can perform and how it can collect rewards.
With the information available to her and the possibly incorrect assumptions she makes about the actor’s reasoning, the observer anticipates that the taxi will start its behavior by moving towards the passenger (so it can later pick her up and drop her off at her destination).
This description of the anticipated behavior over a subset of the reachable states in the environment is the anticipated partial policy .
The preﬁx of this policy is depicted by the green arrow in the ﬁgure.
However, the actual policy adopted by the actor, for which the preﬁx is represented by the red arrow in Figure 1, is to visit some other location before moving towards the passenger.
In order to explain the actor’s behavior, the explainer applies different action and state transforms to the environment.
The objective is to ﬁnd a transformed model in which the actor follows the anticipated partial policy.
In our example, the explainer ﬁrst applies an action abstraction that allows the taxi to move through walls and trains the actor in the transformed environment.
Since the policy in the transformed model still does not match the anticipated policy, the explainer can infer that the reason for the discrepancy is not the fact that the observer may be unaware of the walls in the environment, and therefore this transform would not represent a meaningful explanation.
As a second attempt, the explainer applies an abstraction that relaxes the constraint that a car needs enough fuel to be able to move, and allows the taxi to move regardless of its fuel level.
After training, the actor’s policy in the transformed environment aligns with the anticipated partial policy.
This indicates that the observer may not be aware of the fuel constraint, and therefore does not expect the actor to drive towards the gas station before driving to the passenger.
This transform can therefore be used to explain the discrepancy between the anticipated and actual policies.
Beyond this illustrative example, the ability to understand the “anticipation gap” between an agent and its user is important in many applications.
Examples include autonomous driving Kiran et al. (2020), where it might be critical to know why a vehicle deviates from an anticipated course of action, medical applications, where it is crucial to explain why an AI system recommends one treatment over another Komorowski et al.
(2018), and search and rescue missions Kulkarni et al.
(2020), where a robot is moving in an unknown environment, far from its operator, and may behave unpredictably.
The contributions of this work are threefold.
First, we present the ﬁrst use of formal model transforms and abstractions to produce explanations of the behavior of RL agents.
Second, we formulate the Reinforcement Learning Policy Explanation (RLPE) problem and specify classes of state and action abstractions that can be used to produce meaningful explanations.
Finally, we implement our approach and provide an empirical evaluation of this approach on a set of standard RL benchmarks.
Related Work
A variety of recent studies investigate the use of abstractions to provide meaningful post-hoc explanations for the behavior of RL agents.
Most of these methods focus on a particular element of the model and investigate its effect on the agent’s behavior.
For example, Juozapaitis et al.
(2019) decompose the reward function into an aggregation of meaningful reward types, and then classify actions according to their reward type.
Lin, Lam, and Fern (2021) use human-designed features to represent action-value functions.
The features correspond to meaningful information about the agent’s predictions for its future, such as the distance from the goal.
Huber et al.
(2020) use human user-studies to extract saliency maps for RL agents, which evaluate the relevance of features with regards to mental models, trust, and user satisfaction.
Greydanus et al.
(2017) and Mundhenk, Chen, and Friedland (2019) use saliency maps to produce visual explanations.
Amir and Amir (2018) produce a summary of an agent’s behavior by extracting important trajectories from simulated behaviors of the agent.
Our approach supports arbitrary transforms that can be applied to the environment model without relying on domain-speciﬁc knowledge or on a particular learning approach of the agent.
This variety of transforms that can be used for generating explanation relies on the rich literature of methods suggested for expediting the search for RL setting solutions. Li, Walsh, and Littman (2006); Abel, Hershkowitz, and Littman (2016).
Our approach is similar to Sreedharan et al.
(2020) work which suggests learning partial symbolic models of the environment and task and identifying missing preconditions to explain the actor’s behavior.
However, this work focuses on a deterministic environment where the agent is assumed to compute an optimal plan.
In contrast, our work supports arbitrary transforms applied to stochastic environments with partially informed RL agents.
Background: Reinforcement Learning (RL) and Markov Decision Processes (MDP)
Reinforcement learning (RL) deals with learning policies for sequential decision making of agents that operate in an environment for which the dynamics is not fully known Sutton and Barto (2018).
A common assumption is that the environment can be modelled as a Markov Decision Process (MDP)
Bertsekas (2005), which is a widely used formalization for sequential decision-making in stochastic environments.
An MDP is typically deﬁned as a tuple (cid:104) S, s 0 , A, R, P, γ (cid:105) , where S is a ﬁnite set of states, s 0 ∈ S is an initial state, A is a ﬁnite set of actions, R : S ×
A × S → R is a Markovian and stationary reward function that speciﬁes the reward r ( s, a, s (cid:48) )
an agent gains from transitioning from state s to s (cid:48) by the execution of a , P : S ×
A → P [ S ] is a transition function denoting a probability distribution p ( s, a, s (cid:48) ) over next states s (cid:48)
when action a is executed at state s , and γ ∈ [0 , 1] is a discount factor, representing the deprecation of agent rewards over time.
In this work we use factored MDPs Boutilier, Dean, and Hanks (1999), where the set of states is described via a set of random variables X
= X 1 , . . .
, X n , and where each variable X i takes on values in some ﬁnite domain Dom ( X i ) .
A state is an assignment of a value X i ∈
Dom
( X i ) for each variable X i .
A solution to an RL problem is either a stochastic policy , indicated π : S → P [ A ] , representing a mapping from states s ∈ S to a probability of taking an action a at that state, or a deterministic policy , indicated π : S → A , mapping from states to a single action.
The agent’s objective is to ﬁnd a policy that maximizes the expected accumulated reward.
There are a variety of approaches for solving RL problems Szepesvári (2010); Sutton and Barto (2018); Levine et al.
(2020), that can be generally categorized as either policy gradient methods (e.g.,
Stochastic Gradient Ascent) which learn a numerical preference for executing each action, valuebased methods (e.g., Q-learning), which estimate the values of state-action pairs, and actor-critic methods (e.g., A3C Mnih et al.
(2016)), which combine the value and policy optimization approaches.
Another important distinction exists between model-based methods, where a predictive model is learned, and model-free methods, which learn a control policy directly.
In this work, we support this variety by assuming the algorithm that is used by the actor to compute its policy is part of our input.
Our requirement is each state can be characterized by a set of possible actions for each state.
MDP Transforms
In this work we use formal MDP transforms to explain the behaviors of RL agents.
The explanation is generated by searching for a set of transforms to the MDP used to model the underlying environment such that the actor’s behavior in the modiﬁed model aligns with the observer’s expectations.
If the transition from the original to the transformed environment is meaningful, the difference between the models can help the observer reason about the actor’s behavior, thus representing an explanation.
We formally deﬁne this process in following sections and devote this section to describing various transforms from the literature for expediting the search for optimal MDP policies and suggesting using for explainability.
To account for different modiﬁcations that can be applied, we deﬁne a transform as any mapping T : M (cid:55)→ M that can be applied to an MDP to produce another MDP.
We use the term “transforms" rather than other common terms used in the literature to avoid restricting our discussion to speciﬁc kinds of mappings.
In particular, we do not use the terms “abstractions" or “relaxations", since these are terms typically used to represent mappings that expedite and facilitate planning, and we want to account for arbitrary transforms that can be relevant as explanations.
Transforms may modify any element of the MDP.
We provide some examples of transforms that can be applied, but our framework supports any transform to produce explanations.
We start by deﬁning transforms that modify the representation of the MDP’s state space.
Deﬁnition 1 (State mapping function)
A state-mapping function φ : S (cid:55)→ S
φ maps each state s ∈ S , into a state s (cid:48)
∈ S φ .
The inverse image φ − 1 ( s (cid:48) ) with s (cid:48) ∈ S φ , is the set of states that map to s (cid:48) under mapping function φ .
When changing the state space of an MDP, we need to account for the induced change to the other elements of the model.
For this, we use a state weighing function that distributes the probabilities and rewards of the original MDP between the states in the transformed MDP.
Deﬁnition 2 (State weighting function)
Li, Walsh, and Littman (2006)
A state weighting function of a mapping function φ is function w φ : S (cid:55)→ [0 , 1] where for every ¯ s ∈ S φ , (cid:80) s ∈ φ − 1 (¯ s ) w ( s ) = 1 .
Deﬁnition 3 (State-Space Transform) Li, Walsh, and Littman (2006)
Given a state mapping function φ and a state abstraction weighting function w φ , a state space transform T φ,w maps an MDP
M = (cid:104) S, s 0 , A, R, P, γ (cid:105) to T ( M ) =
(cid:104) ¯ S, ¯ s 0 , A, ¯ R, ¯ P, γ (cid:105) where for every action a :
• ¯ S = S φ • ¯ s 0 = φ ( s 0 ) • ¯ R
(¯ s, a ) =
(cid:80) s
∈ w − 1 (¯ s ) w ( s ) R ( s, a )
• ¯ P (¯ s, a, ¯ s (cid:48) )
= (cid:80) s ∈ φ − 1 (¯ s ) (cid:80) s (cid:48) ∈ φ − 1 (¯ s (cid:48) )
w ( s ) P ( s, a, s (cid:48) )
State-space transforms can, for example, group states.
In factored representations, this can be easily implemented by ignoring a subset of the state features.
In Example 1, a state-space transform can ignore the fuel level, grouping states that share the same taxi and passenger locations.
Another family of transform change the action space.
Deﬁnition 4 (Action mapping function)
An action mapping function ψ : A (cid:55)→
A ψ maps of every action in A to an action in A ψ .
The inverse image ψ − 1 ( a (cid:48) ) for a (cid:48) ∈
A ψ , is the set of actions that map to a (cid:48) under mapping function ψ .
Various action space transforms have been suggested in the literature for planning with MDPs Mausam and Kolobov (2012);
Abel et al.
(2020a)
and can be used for the RL settings we consider here.
For example, in cases where the transition function of the MDP is assumed to be (partially) speciﬁed, it is possible to apply the single-outcome determinization , where all outcomes of an action are removed (associated with zero probability) except for one.
In most practical applications, the most likely or desired outcome is preserved.
Another widely used transform is the all outcome determinization where the planner can choose a desired outcome Yoon, Fern, and Givan (2007b); Keller and Eyerich (2011a).
This transform is typically implemented by creating a separate deterministic action for each possible outcome of the original formulation.
In settings where actions are associated with preconditions, it is possible to apply a precondition relaxation , by which a subset of the preconditions of an action are ignored Sreedharan et al.
For example, for MDPs represented via a factored state space, each action A is associated with a set PRE specifying the required value of its random variables subset.
A precondition relaxation transform removes the restriction regarding these variables.
Another popular formalism for organizing the action space of an agent are options , which represent correlated, long-horizon sequences of actions Sutton, Precup, and Singh (1999); Abel et al.
(2020b).
Options are deﬁned via three components that indicate at which states the option can be executed, under which condition it terminates, and what is the policy to perform between these conditions.
Options are known to expedite learning and support generalization in RL (e.g., Konidaris and Barto (2007); Bagaria et al.
(2021a).
Such representations can support explanations, in particular for model-based option discovery approaches Bagaria et al.
(2021b), where the model itself can be used to reason about the options that are discovered and the agent’s behavior.
Transforms as Explanations
We view the explainability problem as comprised of three entities: an actor , who is an agent operating in the environment, an observer , who has some anticipation about the behavior of the actor that may not be met by the actual behavior, and an explainer , who wishes to clarify the discrepancy between the anticipated and actual behavior.
The input to a Reinforcement Learning Policy Explanation (RLPE) problem includes a description of the environment, a description of the behavior (policy) of an RL agent in the environment, a set of possible behaviors an observer expects the actor to follow, and a set of possible transforms that can be applied to the environment.
Deﬁnition
A Reinforcement Learning Policy Explanation (RLPE) model is deﬁned by the tuple
R = (cid:104) M, A, ˜ π, T (cid:105) , where:
• M is an MDP representing the environment, • A : M → Π represents the actor, which is associated with an RL algorithm that it uses to compute a policy π ∈ Π , • ˜ π is a set of possible anticipated partial policies deﬁned over M an observer expects the actor to follow and, • T : M → M is a ﬁnite set of transforms.
We assume the actor is a reward-maximizing RL agent.
The anticipated behavior is expressed as a partial policy describing what the observer expects the actor to do in a subset of its reachable states.
Clearly, the settings of interest here are those in which the actual policy that the actor follows differs from the anticipated policy.
We denote by T the universal set of transforms.
Each transform T ∈ T is associated with a mapping function for each of the MDP elements that it alters.
We let φ T and ψ T denote the state and action mapping functions, respectively (when the MDP element is not altered by the transform, the mapping represents the identity function).
When a sequence of transforms is applied, we refer to the composite state and action mapping that it induces and deﬁne it as follows.
Deﬁnition 6 (Composite State and Action Space Function)
Given a sequence (cid:126)T = (cid:104) T 1 , . . .
, T n (cid:105) ,
T i ∈ T , the composite state space function of (cid:126)T , is φ (cid:126)T ( s ) =
φ T n · , . . .
, · φ T 1 ( s ) .
The composite action space function is ψ (cid:126)T ( s ) =
ψ T n · , . . .
, · ψ T 1 ( s ) .
The explainer seeks a sequence of transforms that produce an environment where the actor follows a policy that corresponds to the observer’s expectations.
Formally, we seek a transformed environment where the actor’s policy satisﬁes the anticipated policy, i.e., for every state-action pair in the anticipated policy, the corresponding state in the transformed model is mapped to its corresponding action.
Given a partial policy π , we let S ( π ) represent the set of states for which the policy is deﬁned, and deﬁne policy satisfaction as follows.
Deﬁnition 7 (Policy satisfaction)
Given a partial policy π deﬁned over MDP M = (cid:104) S, s 0 , A, R, P, γ (cid:105) , a partial policy π (cid:48) deﬁned over MDP M (cid:48)
= (cid:104) S (cid:48) , s (cid:48) 0 , A (cid:48) , R (cid:48) , P (cid:48) , γ (cid:48) (cid:105) , a state mapping function φ : S (cid:55)→ S (cid:48) and an action mapping function ψ : A (cid:55)→ A (cid:48) , π (cid:48) satisﬁes π , denoted π (cid:48) |
= π , if for every s ∈ S ( π ) , φ ( s ) ∈ S ( π (cid:48) ) and ψ ( π ( s )) =
π (cid:48) ( φ ( s )) .
Clearly, for any two policies there exist state and action mappings that can be applied to cause any policy to satisfy another policy.
In order to produce valuable explanations we need meaningful transforms that are applied to the underlying MDP, which change it in a way that highlights the elements of the model that cause unexpected behaviors in the actor’s policy.
In addition, inspired by the notion of Minimal Sufﬁcient Explanation Juozapaitis et al.
(2019), we want to minimize the change that is applied to the environment.
Intuitively, the more similar the original and transformed
MDPs are, the better the explanation.
We therefore assume the input to an RLPE problem includes some distance metric d : M × M → R + between a pair of MDPs.
There are a variety of metrics for measuring the distance between two MDPs Song et al.
Ammar et al.
A straightforward way to measure the distance between two MDPs is to count the number of transforms that are applied to the original MDP to reach the new MDP, but this deﬁnition is meaningful only with atomic transforms , that change a single element of the MDP (e.g., changing the transition probabilities of a single action).
Another measure is the one suggested by Song et al.
Song et al.
(2016), where the distance between two MDPs M and M (cid:48) is calculated by computing the accumulated distance between every state in M and its corresponding state in M (cid:48) .
Wang et.
al.
Wang, Dong, and Shao (2019) use a representation of an MDP as a bipartite graph of state and action vertices and deﬁne state and action distances between two vertices recursively according to the similarity between their neighbors in the graph.
They show advantage of this method in quantifying structural similarities compared with the bisimulation approach of Ferns et.
al.
Ferns, Panangaden, and Precup (2004).
The method is designed for measuring similarities within the same MDP, but may be applicable in our context to measure distances between an original and a transformed MDP, as long as a mapping between the states and actions of the models are given.
The objective of the explainer is to ﬁnd a sequence of transforms that yield an MDP M (cid:48) such that actor’s policy in M (cid:48) satisﬁes ˜ π .
Among the sequences that meet this objective, we are interested in sequences that minimize the distance between the original and the transformed MDP.
Formally:
Deﬁnition
Given a RLPE model R and a metric function d : M × M → R + , a RLPE problem seeks a transform sequence (cid:126)T = (cid:104) T 1 , . . .
, T n (cid:105) , T i ∈ T s.t.
the policy π (cid:48) of the actor in (cid:126)T ( M ) satisﬁes ˜ π , i.e, π (cid:48) | =
˜ π , 2.
among the sequences that satisfy [1], (cid:126)T minimizes the distance d ( M, (cid:126)T ( M ))
Finding Explanations
In an RLPE setting, the explainer has access to a set of transforms, but does not know which transform sequence will produce meaningful explanations.
This means that the explainer may need to consider a large set of possible sequences.
Therefore, even though an optimal (i.e., minimum distance) sequence of transforms that satisﬁes the anticipated policy can be found using an exhaustive search over T , a naive approach is impractical as the number of transform combinations is exponential in | T | .
To address this computational challenge, we offer three approaches for streamlining the search process.
Inspired by the search for an optimal MDP redesign by Keren et al.
(2019), a basic approach is a Dijkstra-like search through the space of transform sequences.
We assume a successor generator (oracle) is available to provide the MDP that results from applying each transform.
The search graph is constructed in the following way.
The root node is the original environment.
Each edge (and successor node) appends a single transform to the sequence applied to the parent node, where the edge weight represents the distance between the adjacent MDPs according to the distance measure d .
For each explored node we examine whether the actor’s policy in the transformed MDP satisﬁes the anticipated policy.
The search continues until such a model is found, or until there are no more nodes to explore.
The result is a transform sequence that represents an explanation.
This approach is depicted in Figure 2, where the top of the ﬁgure depicts the search in the transform space, while the lower part depicts the MDPs corresponding to each transform sequence.
This approach is guaranteed to return an optimal (minimum distance) solution under the assumption that the distance is additive and monotonic w.r.t the transforms in T , in that a transform cannot decrease the distance between resulting MDP and the original one.
From a computational perspective, even though in the worst case, this approach covers all the possible sequences, in practice, it may ﬁnd solutions much quicker.
In addition, in cases where the transforms are independent, in that their order of application does not affect the result, it is possible to expedite the search by maintaining a closed list that avoids the re-computation of examined permutations.
The depth of the search can also be bounded by a predeﬁned ﬁxed number of transforms.
In spite of these computational improvements, the above solution requires recomputing an actor’s policy in the transformed environment for each node in the search graph.
One way to avoid this is by preserving the agent’s policy in the original environment and using it for bootstrapping the re-training of the agent in the transformed environment.
Another way to expedite the search is to group together a set of transforms and examine whether applying the set leads to a change in the actor’s policy.
If this compound transform does not change the actor’s policy, we avoid computing the values of the speciﬁc transforms.
The approach can potentially reduce the computational effort in settings in which aggregation can be done efﬁciently, e.g., when transforms have parameterized representations.
In our example, if allowing a taxi to move through walls does not change the actor’s policy, we avoid computing the value of individual transforms that allow movement through a single wall.
Empirical Evaluation
Our empirical evaluation is dedicated to examining the ability to produce meaningful explanations via MDP transforms, and to examine the efﬁciency of our suggested approaches for ﬁnding satisfying explanations.
Each RLPE setting includes a description of the underlying environment, the actual policy followed by the actor, and the policy anticipated by the observer.
We describe each of these components below, before describing our results.
7.1 Environments
We used three domains: Taxi, Apple Picking, and Frozen Lake.
The Taxi domain is an adaption of the domain suggested in Dietterich (2000) with an addition of a fuel constraint, and is described in Example 1.
The environment is deterministic and episodic.
Episodes end when the passenger arrives at her destination, the taxi runs out of fuel, or the agent completes 60 steps.
The Frozen Lake domain Brockman et al.
(2016) is a 8 × 8 gridworld representing a navigation task on an icy lake.
There are four actions for moving north, south, east, or west, and the agent’s state is its location.
The environment description, speciﬁes Start and Goal locations in the corners of the grid, as well as the locations of holes in the ice.
The reward is a constant -1 per step until termination, which occurs when the agent reaches the goal (+20 reward) or falls in a hole (-20 reward).
The Frozen Lake environment is stochastic: when the agent moves in direction D , there is a 50% chance of moving in direction D and a 50% chance of slipping on the ice and instead moving in a direction directly clockwise or counterclockwise to D .
The Apple Picking environment is a 5 × 5 gridworld representing a navigation task in an orchard.
There are ﬁve actions for moving north, south, east, or west, and picking an apple.
The environment state consists of the agent location and the status of the apples distributed in the grid in random locations.
Thorny walls scattered on the grid.
The reward for stepping into a thorny wall is -5, and there is a constant -1 per step until termination, which occurs when the agent picks all apples (+ reward) or the episode terminates.
The domain is stochastic: when moving near a thorny wall, the agent succeeds with probability 0.6 and falls into the thorny wall with probability 0.4.
7.2 Setup
Observer:
We consider an observer with partial knowledge about the environment.
In the taxi domain, for example, the observer may have access to the map and reward function, but may be unaware of the need for fuel for movement.
For all environments we assume that the observer anticipates that the actor follows an optimal policy in the observer’s partial model.
We compute the observers anticipated policy using a breadth-ﬁrst search, using a determinization for stochastic domains.
We keep track of multiple optimal paths that may exist, resulting in more than one anticipated policy.
Actor: We use the DQN, SARSA and CEM algorithms, from the keras-rl library 2 to represent the actor.
Note that we could use any other algorithm since our framework is agnostic to the method used by the actor.
We trained the agents for 600,000 episodes for the Taxi domain, and the Frozen Lake domain, and 1,000,000 episodes for the Apple picking domain, each with 60 maximum steps per episode.
Our experiments were run on a cluster using a mix of CPUs, with 4 cores using 16GB RAM.
Explainer:
For the explainer, we use the following transforms:
• most likely outcome : a special case of the single-outcome determinization by Yoon, Fern, and Givan (2007a).
• precondition relaxation as variant of the usage by Sreedharan et al.
(2020) based on STRIPS models Geffner and Bonet (2013).
• all outcome determinization suggested by Keller and Eyerich (2011b).
We implemented all three transforms as atomic transforms that each modify a single action.
For the most likely outcome, we create one transform for each action.
For the all outcome determinization, we create a transform for each action and stochastic outcome pair.
For the precondition relaxation transform we created a transform for each action and precondition pair.
We used three methods for searching for explanations, which were detailed in the previous section (for further details see the supplamentary materials).
The BASE approach is a Dijkstra search, PRE-TRAIN is a Dijkstra search in which we bootstrap the learning in the modiﬁed environment by using the policy on the original environment.
We also prune transforms that do not directly effect the anticipated policy.
The third method, CLUSTER , computes values of transform clusters.
Further details on the methods and domains are found in the supplementary materials.
BASE
PRE-TRAIN
CLUSTER
[Figure]
7.3 Results
To assess the ability to produce explanations using environment transforms, we measured the satisfaction rate of each transform in a given environment.
This measure is deﬁned as the fraction of the number of states for which the anticipated policy and actor policy agree among all states for which the anticipated policy is deﬁned, i.e., the number of states s ∈ S ( π ) for which φ ( s ) ∈ S ( π (cid:48) ) and ψ ( π ( s ))
=
π (cid:48) ( φ ( s )) .
We also measured the length of the explanation, which we used as our distance measure (that should be minimized).
In Figure 3 we examine the satisfaction rate achieved per transform in each domain.
As can be seen, in each domain there is at least one transform combination that fully satisﬁes the anticipated policy.
For the taxi domain the highest matching rate of 1.0 is reached when the precondition of the fuel for the actions down, right, pickup and drop-off are relaxed.
This result aligns with our taxi example where the observer wasn’t aware of the fuel limitation.
For the apple picking domain, the highest satisfaction rate is achieved by applying the all outcome determinization and the most likely outcome transforms.
For the Frozen Lake domain the all outcome determinization achieved the best results.
Table 3 compares the performance of the different search methods, showing the ratio of instances that were solved (denoted in the table as “sol”) and the average computation time in half hour intervals (denoted in the table as “time”) for the instances that were solved by all approaches.
The results show that, as expected, the exhaustive BASE method outperforms the two other methods in terms of ratio of instances solved.
However, it consumes up to 5 times more computation time.
In addition, although the PRE-TRAIN and CLUSTER methods prune transform sequences heuristically, the compromise in terms of solved instances is negligible.
Conclusion
We introduced a novel formulation of explainability of RL, and an approach to generating explanations using formal model transforms, which were previously mainly used for facilitating the computation of MDP policies.
In addition, we presented approaches for ﬁnding a sequence of transforms that represents an explanation to the actor’s behavior.
Our empirical evaluation on a set of standard RL benchmarks illustrates the ability of formal symbolic MDP transforms to produce explanations that can be interpreted by a human user, and the efﬁciency of our approaches for searching through the space of transform sequences.
There are various ways to extend our approach.
First, our empirical setup can be enriched to include anticipated policies as they would be expected by human users, and to evaluate the extent by which the generated explanations are meaningful to them.
Secondly, while this work uses a restrictive satisfaction relation that required a complete match between the anticipated policy and the actor’s behavior, it may be useful to use more ﬂexible qualitative evaluation metrics for satisfaction.
Finally, while the focus in this setting has been on single-agent domains, we plan to extend this approach to explain behaviors of agents in multi-agent domains and to add transforms that are relevant to both collaborative and adversarial multi-agent settings.