Explainable Reinforcement Learning via Model Transforms
rebuttal version
Anonymous Author(s)
AffiliationAddress email
Abstract
Understanding emerging behaviors of reinforcement learning (RL) agents maybe difficult since such agents are often trained in complex environments usinghighly complex decision making procedures.
This has given rise to a variety ofapproaches to explainability in RL that aim to reconcile discrepancies that mayarise between the behavior of an agent and the behavior that is anticipated byan observer.
Most recent approaches have relied either on domain knowledge,that may not always be available, on an analysis of the agent’s policy, or on ananalysis of specific elements of the underlying environment, typically modeled asa Markov Decision Process (MDP).
Our key claim is that even if the underlying
MDP is not fully known (e.g., the transition probabilities have not been accuratelylearned) or is not maintained by the agent (i.e., when using model-free methods),it can nevertheless be exploited to automatically generate explanations.
For thispurpose, we suggest using formal MDP abstractions and transforms, previouslyused in the literature for expediting the search for optimal policies, to automaticallyproduce explanations.
Since such transforms are typically based on a symbolicrepresentation of the environment, they may represent meaningful explanations forgaps between the anticipated and actual agent behavior.
We formally define thisproblem, suggest a class of transforms that can be used for explaining emergentbehaviors, and suggest methods that enable efficient search for an explanation.
Wedemonstrate the approach on a set of standard benchmarks.
Introduction
The performance-transparency trade-off is a major limitation of many artificial intelligence (AI)methods [26]: as the inner workings of an agent’s decision making procedure increases in complexity,it becomes more powerful but the agent’s decisions become harder to understand.
Accordingly,interest in explainable AI and the development of transparent, interpretable, AI models has increasedrapidly in recent years [5].
This trend is particularly prevalent in reinforcement learning (RL) anddeep reinforcement learning (DRL), where an agent autonomously learns how to operate in itsenvironment.
While RL has been successfully applied to solve many challenging tasks, includingtraffic control [4], robotic motion planning [18], and board games [27], it is increasingly challengingto explain the behavior of RL agents, especially when they do not operate as anticipated.
To allowhumans to collaborate effectively with RL-based AI systems and increase their usability, it is thereforeimportant to develop automated methods for reasoning about and explaining agent behaviors.
While there has been recent work on explainability of DRL (see [25] for a recent survey), most ofthese methods either rely on domain knowledge, which may not be available, or on a post-processanalysis of the parameterized policy learned by the agent (e.g., by reasoning about the structure of the underlying neural network [12]).
Moreover, most existing methods for explainability do not fully exploit the formal model that is assumed to represent the underlying environment, typically modeledas a Markov Decision Process (MDP) [6], and analyze instead one chosen element of the model (e.g.,the agent’s reward function [15]).
In this work we focus on RL settings in which the model of the underlying environment is partiallyknown, i.e., the state space and action space are specified via some symbolic representation, but thetransition probabilities and reward function are unknown.
This is a common assumption in many RLmethods, including both model-based methods and model free methods.
In both cases, the action andstate spaces are typically given, but the agent is not initially provided with the reward function andtransition probabilities and can either learn them explicitly (in model-based RL) or directly learn tooptimize its behavior (in model-free RL).
For example, in a robotic setting, the agent may have somerepresentation of the state features (e.g., the location of objects) and of the actions it can perform (e.g.,picking up an object), but does not know its reward function or the probabilities of action outcomes.
Our key claim is that even if the underlying model may not be fully known, it can nevertheless beused to automatically produce meaningful explanations for the agent’s behavior.
Specifically, wesuggest producing explanations by searching for a set of formal abstractions and transforms that areapplied to the MDP used to represent the underlying environment and that yield a behavior that isaligned with an observer’s expectations.
For this purpose we exploit the rich body of literature thatoffers a variety of MDP transforms [7, 19, 1] that manipulate different elements of the model by, forexample, ignoring the stochastic nature of the environment, ignoring some of the effects of actions,or considering irrelevant constraints [7].
While these methods have so far been used to expediteplanning and RL solutions, we use them to automatically produce explanations.
That is, while forplanning the benefit of using such transforms is in increasing solution efficiency, we use them toisolate features of the environment that cause an agent to deviate from a behavior that is anticipatedby an observer.
Formally, we consider an explainability setting, which we term Reinforcement Learning Policy
Explanation (RLPE), that is comprised of three entities.
The first entity, the actor , is an RL agentthat seeks to maximize its accumulated reward in the environment.
The second entity, the observer ,expects the actor to behave in some way and to follow a certain policy, which may differ from the oneactually adopted by the actor.
We refer to this policy as the anticipated policy , which specifies whichactions an observer expects the actor to perform in some set of states 1 .
Finally, the explainer hasaccess to the description of the environment, to the anticipated policy, and to a set of MDP transforms.
The explainer seeks a sequence of transforms to apply to the environment such that the actor’s policyin the transformed environment aligns with the observer’s anticipated policy.
Example 1 To demonstrate RLPE, consider Figure 1, which depicts a variation of the Taxi domain[10].
In this setting, the actor represents a taxi that operates in an environment with a single passenger.
The taxi can move in each of the four cardinal directions, and pick up and drop off the passenger.
The taxi incurs a small cost for each action it performs in the environment, and gains a high positive reward for dropping off the passenger at her destination.
There are walls in the environment that thetaxi cannot move through.
The observer has some partial view of the environment and knows whichactions the taxi can perform and how it can collect rewards.
With the information available to her andthe, possibly incorrect, assumptions she makes about the actor’s reasoning, the observer anticipatesthat the taxi will start its behavior by moving towards the passenger (so it can later pick her up anddrop her off at her destination).
This description of the anticipated behavior over a subset of thereachable states in the environment is the anticipated policy .
The prefix of this policy is depicted bythe green arrow in the figure.
However, the actual policy adopted by the actor, for which the prefix isrepresented by the red arrow, is to visit some other location before moving towards the passenger.
In order to explain the actor’s behavior, the explainer applies different action and state transformsto the environment.
The objective is to find a transformed model in which the actor follows theanticipated policy.
We note that our suggested approach can produce meaningful explanations only ifthe explainer uses transforms that are meaningful to the observer.
In our example, the explainer firstapplies an action abstraction that allows the taxi to move through walls and trains the actor inthe transformed environment.
Since the policy in the transformed model still does not match theanticipated policy, the explainer can infer that the reason for the discrepancy is not the fact that theobserver may be unaware of the walls in the environment, and therefore this transform would notrepresent a meaningful explanation.
As a second attempt, the explainer applies an abstraction thatrelaxes the constraint that a car needs enough fuel to be able to move, and allows the taxi to moveregardless of its fuel level.
After training, the actor’s policy in the transformed environment alignswith the anticipated policy.
This indicates the observer may not be aware of the fuel constraint, andtherefore does not expect the actor to drive towards the gas station before driving to the passenger.
This transform can therefore be used to explain the discrepancy between the anticipated and actualpolicies, and represents an explanation if this constraints can be conveyed to the observer.
Beyond this illustrative example, the ability to understand the “ anticipation gap ” is important inmany applications.
Examples include autonomous driving, where it is critical to know why a vehicledeviates from an anticipated course of action, medical applications, where it is crucial to explain whyan AI system recommends one treatment over another, and search and rescue missions, where a robotis moving in an unknown environment with observations that are different from those of its operator,and may behave in unpredictable ways.
While the translation of the transform sequence that reconciles the gap between the observer andactor to natural language is beyond the scope of this work, since the transforms manipulate theunderlying MDP model, they incorporate symbolic information about the environment, and cantherefore often be translated to intuitive explanations (e.g., notifying the observer about a missingprecondition in its model of some action).
Thus, our approach automatically generates explanationswithout compromising generality.
Moreover, while we used a single-agent setting to demonstrateour approach, the same ideas can be applied to collaborative multi-agent settings, where the set ofapplicable transforms can include, in addition to the transforms used also for single-agent settings,transforms that deal with the multi-agent aspects of the system (e.g., shared resource constraints).
A variety of approaches have been developed recently to provide meaningful post-hoc explanationsfor the behavior of RL agents.
A typical approach in many of these works is to focus on a particularelement of the model and investigate its effect on the agent’s behavior.
For example, in [15] thereward function is decomposed into an aggregation of meaningful reward types, and then actions areclassified according to their reward types.
In [20], human-designed features, such as the estimateddistance to the goal, are used to represent action-value functions.
In [14], human-user studies areused to extract saliency maps for RL agents, in order to evaluate the relevance of features with regardto mental models, trust, and user satisfaction, while [12] and [24] use saliency maps to producevisual explanations.
Similarly, [3] produce a summary of an agent’s behavior by extracting importanttrajectories from simulated behaviors of the agent.
Our approach supports arbitrary transforms and abstraction that can be applied to the environmentmodel without relying on a particular learning approach of the agent, and can be applied to bothsingle-agent and multi-agent settings.
This variety of transforms that can be used for generatingexplanations relies on the various methods suggested for expediting planning and the search for RLsolutions [19].
The work by [30] learns partial symbolic models of the environment and task, andidentifies missing preconditions to explain the behavior of an optimal planning agent in deterministic environments.
Our framework generalizes to stochastic environments with partially informed RLagents and to arbitrary transforms that can be applied.
The contributions of this work are threefold.
First, we present a novel use of formal model transformsand abstractions, formerly mainly used for planning, to produce explanations of RL agent behaviors.
Second, we present a formal definition of the Reinforcement Learning Policy Explanation (RLPE)problem and specify classes of state and action abstractions that can be used to produce explanations.
Finally, we empirically demonstrate our approach on a set of standard single agent and collaborativemulti-agent RL benchmarks.
Background
Reinforcement learning (RL) deals with learning policies for sequential decision making of agents thatoperate in an environment for which the dynamics is not fully known [31] .
A common assumption isthat the environment can be modelled as a Markov Decision Process (MDP) [6], which is a widelyused formalization for sequential decision-making in stochastic environments.
An MDP is typicallydefined as a tuple ⟨ S, s 0 , A, R, P, γ ⟩ , where S is a finite set of states, s 0 ∈ S is an initial state, A is afinite set of actions, R : S
×
A × S → R is a Markovian and stationary reward function that specifiesthe reward r ( s, a, s ′ ) that an agent gains from transitioning from state s to s ′ by the execution ofaction a , P : S ×
A → P [ S ] is a transition function denoting a probability distribution p ( s, a, s ′ )over next states s ′ when action a is executed at state s , and γ ∈ [0 , 1] is a discount factor, representingthe deprecation of agent rewards over time.
In this work we use factored MDPs [8], where the set ofstates is described via a set of random variables
X
= X 1 , . . .
, X n , and where each variable X i takeson values in some finite domain Dom ( X i ) .
A state is an assignment of a value X i ∈
Dom
( X i ) foreach variable X i .
To model the fully collaborative multi-agent settings we support in this work, weuse a Markov game [21], which generalizes the MDP model by including joint actions
A =
{ A i } ni =
1representing
the collection of action sets A i , for each of the n agents.
We will hereon refer to an
MDP as the model of the underlying environment, and highlight the considerations that are specificto a Markov game.
A solution to an RL problem is either a stochastic policy , indicated π : S → P [ A ] , representing amapping from states s ∈ S to a probability of taking an action a at that state, or a deterministic policy ,indicated π : S → A , mapping from states to a single action.
The agent’s objective is to find a policythat maximizes the expected accumulated reward.
There are a variety of approaches for solving RL problems [32, 31], that can be generally categorizedas either policy gradient methods (e.g., Stochastic Gradient Ascent) which learn a numerical preference for executing each action, value-based methods (e.g., Q-learning), which estimate the valuesof state-action pairs, and actor-critic methods, which combine the value and policy optimizationapproaches.
Another important distinction exists between model-based methods, where a predictivemodel is learned, and model-free methods, which learn a control policy directly.
We support thisvariety by assuming the algorithm that is used by the actor to compute its policy is part of our input.
Our requirement is that each state is characterized by the set of actions that are applicable at that state.
MDP Transforms
We use MDP transforms to explain the behaviors of RL agents.
Given a large set of possibletransforms that is given as input, an explanation is generated by searching for a set of transforms toapply to the environment’s model such that the actor’s behavior in the modified model aligns with theobserver’s expectations.
Since the transition from the original to the transformed environment is doneby manipulating the symbolic representation of the environment, the difference between the modelscan help the observer reason about the actor’s behavior, thus representing an explanation.
We formally define this process in the following sections, and devote this section to describingvarious transforms suggested in the literature for expediting planning and the search for optimal RLpolicies, and that we suggest using for explainability.
To account for different modifications that canbe applied, we define a transform as any mapping T : M (cid:55)→ M that can be applied to an MDP toproduce another MDP.
We use the term “transforms" rather than other common terms used in theliterature to avoid restricting our discussion to specific kinds of mappings.
Thus, transforms mayrepresent “abstractions" or “relaxations", that are typically used to represent mappings that facilitate planning, but also other mappings that may not provide such guarantees, including those that mayrender more complex environments.
Moreover, the transforms we support may modify any element ofthe MDP.
We provide some examples of transforms that can be applied, but our framework supportsany transform to produce explanations.
We start by defining transforms that modify the representationof the MDP’s state space.
Definition 1 (State mapping function)
A state-mapping function ϕ : S (cid:55)→ S ϕ maps each states ∈ S , into a state s ′ ∈ S ϕ .
The inverse image ϕ − 1 ( s ′ ) with s ′ ∈ S ϕ , is the set of states that map tos ′ under mapping function ϕ .
When changing the state space of an MDP, we need to account for the induced change to the otherelements of the model.
For this, we use a state weighting function that distributes the probabilitiesand rewards of the original MDP among the states in the transformed MDP.
Definition 2 (State weighting function)
[19] A state weighting function of a state mapping functionϕ is function w ϕ : S (cid:55)→ [0 , 1] where for every ¯ s ∈ S ϕ , (cid:80) s ∈ ϕ − 1 (¯ s ) w ( s ) = 1 .
Definition 3 (State-Space Transform) [19] Given a state mapping function ϕ and a state abstraction weighting function w ϕ , a state space transform T ϕ,w maps an MDP M = ⟨ S, s 0 , A, R, P, γ ⟩ to
T ( M ) =
⟨ ¯ S, ¯ s 0 ,
A, ¯ R, ¯ P, γ ⟩ where for every action a :
• ¯ S = S ϕ
• ¯ s 0 = ϕ
( s 0 )
• ¯ R (¯ s, a ) =
(cid:80) s
∈ w − 1 (¯ s ) w ( s ) R ( s, a )
• ¯ P (¯ s, a, ¯ s ′ )
= (cid:80) s ∈ ϕ − 1 (¯ s ) (cid:80)
s
′ ∈ ϕ − 1 (¯ s ′ )
w ( s ) P ( s, a, s ′
State-space transforms can, for example, group states.
In factored representations, this can be easilyimplemented by ignoring a subset of the state features.
In Example 1, a state-space transform canignore the fuel level, grouping states that share the same taxi and passenger locations.
Another family of transform changes the action space.
Definition 4 (Action mapping function)
An action mapping function ψ : A (cid:55)→
A ψ maps everyaction in A to an action in A ψ .
The inverse image ψ − 1 ( a ′ ) for a ′ ∈
A ψ , is the set of actions thatmap to a ′ under mapping function ψ .
Various action space transforms have been suggested in the literature for planning with MDPs [23, 2].
Since such transforms inherently bear a symbolic meaning with regards to the environment and agent,a sequence of transforms that yields the anticipated policy represents the gap between the actor’sbehavior and the expectations of the observer, and therefore can be used as an explanation.
We notethat the focus here is on automatically finding transforms that bridge this gap, and since they arebased on symbolic manipulations of the environment we assume their mapping to natural language isrelatively straightforward (but is beyond the scope of this work).
For example, in cases where the transition function of the MDP is assumed to be (partially) specified,it is possible to apply the single-outcome determinization , where all outcomes of an action areremoved (associated with zero probability) except for one, e.g., the most likely or desired outcome.
Similarly, the all outcome determinization allows a planner to choose a desired outcome [34].
Thistransform is typically implemented by creating a separate deterministic action for each possibleoutcome of the original formulation.
If such transforms yield the anticipated policy, it implies thatthe observer may not be aware of the alternative outcomes of an action or of the stochastic nature ofthe environment.
In settings where actions are associated with preconditions, it is possible to apply aprecondition relaxation , by which a subset of the preconditions of an action are ignored [30].
Forexample, for MDPs represented via a factored state space, each action a is associated with a setpre ( a ) specifying the required value of a subset of its random variables.
A precondition relaxationtransform removes the restriction regarding these variables.
Similarly, it is possible to ignore someof an action’s effects, for example by applying a delete relaxation and ignoring an actions’ effecton Boolean variables that are set to false [13].
A transform may also consider adding preconditions that may considered by the observer by mistake.
If such transforms produce the anticipated policy, aplausible explanation is that the observer is not aware of the preconditions or effects of actions, suchas in the setting we describe in Example 1.
The transforms mentioned above are applicable to the collaborative multi-agent settings we support inthis work.
In addition, for such settings we can also apply transforms that are specific to multi-agentsettings, such as transforms that allow collisions or that change the communication range in themodel.
In a multi-agent extension of our taxi example, an observer may not be aware that taxis cannotoccupy the same cell - a discrepancy that can be explained by applying a transform that ignores theconstraint (precondition) that a cell needs to be empty for an agent to move into it.
Transforms as Explanations
We formalize the explainability problem as comprised of three entities: an actor , who is an agentoperating in the environment, an observer , who has some anticipation about the behavior of theactor that may not be met by the actual behavior, and an explainer , who wishes to clarify thediscrepancy between the anticipated and actual behaviors.
The input to a Reinforcement Learning
Policy Explanation (RLPE) problem includes a description of the environment, a description of thebehavior (policy) of an RL agent in the environment, a set of possible behaviors an observer expectsthe actor to follow and a set of possible transforms that can be applied to the environment.
Definition
A Reinforcement Learning Policy Explanation (RLPE) model is defined as R =⟨ M, A, ˜ π, T ⟩ , where
• M is an MDP representing the environment,
• A : M → Π is the actor, which is associated with an RL algorithm that it uses to compute a policy π ∈ Π ,
• ˜ π is the anticipated policy the observer expects the actor to follow, and
• T : M → M is a finite set of transforms.
We assume the actor is a reward-maximizing RL agent 3 .
The anticipated behavior of the observer describes what the observer expects the actor to do in a subset of its reachable states 4 .
Since we do notrequire the anticipated policy to be defined over all states, we refer to it as a partial policy and allowsituations in which the observer only specifies its anticipated behavior over selected states.
Clearly,the settings of interest here are those in which the actual policy that the actor follows differs fromthe anticipated policy.
We denote by T the universal set of transforms.
Each transform T ∈ T isassociated with a mapping function for each of the MDP elements that it alters.
We let ϕ T and ψ Tdenote the state and action mapping functions, respectively (when the MDP element is not alteredby the transform, the mapping represents the identity function).
When a sequence of transforms isapplied, we refer to the composite state and action mapping that it induces and define it as follows.
Definition 6 (Composite State and Action Space Function)
Given a sequence ⃗T = ⟨ T 1 , . . .
, T n ⟩ ,
T i ∈ T , the composite state space function of ⃗T , is ϕ ⃗T ( s ) =
ϕ T n · , . .
, · ϕ T 1 ( s ) .
The compositeaction space function is ψ ⃗T ( s )
=
ψ T n · , . . .
, · ψ T 1 ( s ) .
The explainer seeks a sequence of transforms that produce an environment where the actor follows apolicy that corresponds to the observer’s expectations.
Formally, we seek a transformed environmentwhere the actor’s policy satisfies the anticipated policy, i.e., for every state-action pair in the anticipatedpolicy, the corresponding state in the transformed model is mapped to its corresponding action.
Givena policy π , we let S ( π ) represent the set of states for which the policy is defined, and define policysatisfaction as follows.
Definition 7 (Policy satisfaction) Given a partial policy π defined over MDP M =⟨ S, s 0 , A, R, P, γ ⟩ , a partial policy π ′ defined over MDP M ′ = ⟨ S ′ , s ′ 0 , A ′ , R ′ , P ′ , γ ′ ⟩ , a state mapping function ϕ : S (cid:55)→ S ′ and an action mapping function ψ : A (cid:55)→ A ′ , π ′ satisfies π , denotedπ ′ |
= π , if for every s ∈ S ( π ) , ϕ ( s ) ∈ S ( π ′ ) and ψ ( π ( s ))
=
π ′ ( ϕ ( s )) .
Clearly, for any two policies there exist state and action mappings that can be applied to cause anypolicy to satisfy another policy.
In order to produce valuable explanations the input needs to includesmeaningful transforms to be applied to the underlying MDP, i.e., transforms which change it in away that highlights the elements of the model that cause unexpected behaviors in the actor’s policy.
In addition, inspired by the notion of Minimal Sufficient Explanation [15], we want to minimize thechange that is applied to the environment.
Intuitively, the more similar the original and transformed MDPs are, the better the explanation.
We therefore assume the input to an RLPE problem includessome distance metric d : M × M → R + between a pair of MDPs, for example by counting thenumber of atomic changes to the elements of the MDP [29] (see the appendix for a description ofsome relevant distance metrics from the literature).
The objective of the explainer is to find a sequence of transforms that yield an MDP M ′ such that theactor’s policy in M ′ satisfies ˜ π .
Among the sequences that meet this objective, we are interested insequences that minimize the distance between the original and the transformed MDP.
Formally:
Definition
Given a RLPE model R and a metric function d : M × M → R + , an RLPE problemseeks a transform sequence ⃗ T = ⟨ T 1 , . .
, T n ⟩ , T i ∈
T s.t.1.
the actor’s policy π ′ in ⃗T ( M ) satisfies ˜ π , i.e, π ′ | =
˜ π ,2.
among the sequences that satisfy [1], ⃗T minimizes the distance d ( M, ⃗T ( M )) .
Finding Explanations
In an RLPE setting, the explainer has access to a set of transforms, but does not know a-prioriwhich transform sequence will produce meaningful explanations in a given setting.
This means thatthe explainer may need to consider a large set of possible sequences.
Therefore, even though anoptimal (i.e., minimum distance) sequence of transforms that satisfies the anticipated policy can befound using an exhaustive search over T , a naive approach is impractical as the number of transformcombinations is exponential in | T | .
To address this computational challenge, we offer several approaches for expediting the search.
Inspired by the search for an optimal MDP redesign in [17], a basic approach is a Dijkstra-like searchthrough the space of transform sequences.
Assuming a successor generator is available to provide the
MDP that results from applying each transform, the search graph is constructed in the following way.
The root node is the original environment.
Each edge (and successor node) appends a single transformto the sequence applied to the parent node, where the edge weight represents the distance betweenthe adjacent MDPs according to the distance measure d .
For each explored node we examine whetherthe actor’s policy in the transformed MDP satisfies the anticipated policy.
The search continues untilsuch a model is found, or until there are no more nodes to explore.
The result is a transform sequencethat represents an explanation.
This approach is depicted in Figure 2, where the top of the figuredepicts the search in the transform space, while the lower part depicts the MDPs corresponding toeach sequence.
This approach is guaranteed to return an optimal (minimum distance) solution under the assumptionthat the distance is additive and monotonic with respect to the transforms in T , in that a transformcannot decrease the distance between the resulting MDP and the original one.
From a computationalperspective, even though in the worst case this approach covers all the possible sequences, in practiceit may find solutions much quicker.
In addition, in cases where the transforms are independent , in thattheir order of application does not affect the result, it is possible to expedite the search by maintaininga closed list that avoids the re-computation of examined permutations.
The depth of the search canalso be bounded by a predefined fixed number of transforms.
In spite of these computational improvements, the above solutions require recomputing an actor’spolicy in the transformed environment for each explored node.
One way to avoid this is by preservingthe agent’s policy in the original environment and using it for bootstrapping the re-training inthe transformed environment.
Another way to expedite the search is to group together a set oftransforms and examine whether applying the set leads to a change in the actor’s policy.
If this compound transform does not change the actor’s policy, we avoid computing the values of the specifictransforms.
This approach is inspired by pattern database (PDB) heuristic approaches [11], as wellas the relaxed modification heuristic of [17]. Even though this heuristic approach compromisesoptimality, it can potentially reduce the computational effort in settings in which aggregation canbe done efficiently, such as when transforms have parameterized representations.
In our example, ifallowing a taxi to move through (all) walls does not change the actor’s policy, we avoid computingthe value of a single transform of a single wall.
Finally, we examine the efficiency of performing afocused policy update : when applying a transform, instead of updating the policy for all states, westart by only updating the policy of states that are directly affected by the transform, and then followthe propagated effect of this change.
For example, when removing a wall in the taxi domain, we startby updating the policy of states that are near the wall, and iterativly follow the propagated effect ofthis change to adjacent cells.
Empirical Evaluation
Our empirical evaluation is dedicated to examining the ability to produce meaningful explanationsvia MDP transforms, and to examine the empirical efficiency of our suggested approaches for findingsatisfying explanations.
Each RLPE setting includes a description of the underlying environment, theactual policy followed by the actor, and the anticipated policy.
We describe each component below,before describing our results 5 .
Environments: We conducted experiments with 12 different environments, including both deterministic and stochastic domains and single and multi-agent domains (see Figure 3).
Among thesingle agent domains, 3 are adaptations of environments suggested by OpenAI [9]: Frozen Lake ,the Taxi domain [10] from Example 1, which we extend with a fuel constraint, and Stochastic Apple
Picking, in which agents need to collect apples but have a probability of slipping.
We also used seven
PDDLGym domains [28]: Sokoban, Blocks World, Towers of Hanoi, Snake, Rearrangement, Triangle
Tireworld and Exploding Blocks.
The PDDLGym framework aligns with the OpenAI Gym interfacewhile allowing the user to provide a model-based relational representation of the environments.
Wenote that this representation is not available to the actor, which operates using standard RL algorithmswithout access to the model (e.g., using model-free methods).
Our multi-agent domains includedextensions of the Sokoban and Taxi domain to collaborative (shared reward) multi-agent settings.
Observer:
We consider a partially informed observer which has access to a subset of the environmentfeatures.
In the taxi domain, for example, the observer may be unaware of the fuel constraint.
Forall environments we assume the observer anticipates that the actor follows a policy that is optimalw.r.t.
the observer’s (possibly incomplete or inaccurate) model.
Optimal plans are produced using A*implemented using [22].
Actor: We use the DQN, SARSA and CEM from the keras-rl library 6 and Q-LEARNING [33] torepresent the actors (note that our framework can be used with any RL algorithm since it is agnosticto the method used by the actor).
We trained agents for 600 K - 1 , 000 K episodes, with a maximumof 60 steps per episode.
Experiments were run on a cluster using 6 CPUs, with 4 cores using 16GB
RAM
Explainer: We used 5 paramterized transform types: state space reduction [34], likely outcomerelaxation [34], precondition relaxation [30], all outcome determinization (for stochastic domains)[16], and delete relaxation [13].
Grounding is performed automatically for each transform, for theenvironments in which it is applicable.
Each grounded transform modifies a single action or statevariable.
We used three methods for searching for explanations which were described in the previous section.
BASE is a Dijkstra search.
PRE-TRAIN is a Dijkstra search using the learned policy in the originalenvironment to bootstrap the learning in the modified environment, and a focused policy update toavoid iteratively updating the entire policy.
PRE+CLUSTER extends PRE-TRAIN by computingvalues of transform sets and using them to prune useless transforms.
Results: To assess the ability to produce explanations using environment transforms, we measuredthe satisfaction ratio of each transform in a given environment.
This measure is defined as the fractionof the number of states for which the anticipated policy and actor policy agree among all states forwhich the anticipated policy is defined, i.e., the number of states s ∈ S ( π ) for which ϕ ( s ) ∈ S ( π ′ )and ψ ( π ( s ))
=
π ′ ( ϕ ( s )) .
We also measured the length of the explanation (i.e. the number of atomictransforms that were applied to the MDP), which we used as our distance measure (and that shouldbe minimized).
Figure 4 presents a scatter plot with the results for the different methods for the single agent domains(due to space limitations, additional results and results for the multi-agent domains can be found inthe appendix).
It represents for each domain the time it takes (in hours) to find an explanation (x axis)and the satisfaction ratio (y axis) for DQN (the results for the other approaches and measures are inthe appendix).
Results show that while BASE, achieves the highest satisfaction ratio (which is to beexpected from an exhaustive) approach, its computation time is much higher, with up to nine timeshigher computation time then the best approach for Tireworld.
In contrast, PRE+CLUSTER outperforms all other methods in terms of computation time.
The approach compromises satisfaction ratio,but with 84% success in the worst case and with an average variance of 0.03 over all environments).
Conclusion
We introduced a novel formulation for explainability in RL, and a novel approach for generatingexplanations using formal model transforms, which were previously primarily used for planning.
Ourempirical evaluation on a set of single and multi-agent RL benchmarks illustrates the efficiency ofour approaches for finding explanations among large sets of transforms.
Possible extensions of our work include integrating human users or models of human reasoninginto the process of generating anticipated policies and in the process of evaluating the quality of theexplanations generated by our methods.
In addition, while this work uses a restrictive satisfactionrelation that requires a full match between the anticipated policy and the actor’s behavior in discretedomains, it may be useful to use more flexible qualitative evaluation metrics for satisfaction andto account for continuous domains.
Finally, our current account of multi-agent settings focuses on fully collaborative settings.
As a next step we intend to extend our multi-agent account to adversarialdomains.