Explaining a Reinforcement Learning Agent via Prototyping
Abstract
While deep reinforcement learning has proven to be successful in solving controltasks, the “black-box” nature of an agent has received increasing concerns.
Wepropose a prototype-based post-hoc policy explainer , ProtoX, that explains a blackbox agent by prototyping the agent’s behaviors into scenarios, each representedby a prototypical state.
When learning prototypes, ProtoX considers both visualsimilarity and scenario similarity.
The latter is unique to the reinforcement learningcontext, since it explains why the same action is taken in visually different states.
To teach ProtoX about visual similarity, we pre-train an encoder using contrastivelearning via self-supervised learning to recognize states as similar if they occurclose together in time and receive the same action from the black-box agent.
Wethen add an isometry layer to allow ProtoX to adapt scenario similarity to thedownstream task.
ProtoX is trained via imitation learning using behavior cloning,and thus requires no access to the environment or agent.
In addition to explanationfidelity, we design different prototype shaping terms in the objective function toencourage better interpretability.
We conduct various experiments to test ProtoX.
Results show that ProtoX achieved high fidelity to the original black-box agentwhile providing meaningful and understandable explanations.
Introduction
Deep reinforcement learning (DRL) has been widely used to solve various control problems such asautonomous driving [1], gaming [2] etc.
Despite their superior performance, DRL agents are oftencriticized for being “black-boxes” in nature and unable to provide human-understandable explanationsfor their actions.
Popular solutions to interpreting a black-box agent include attention-based methods[3] and saliency maps [4, 5, 6].
These explanations, however, only inform users of where the agent“looks at” but do not provide a more specific explanation for why an agent takes this action in agiven state .
Some recent methods answer this question by formulating a classification problem andbuild a decision tree from state information to predict an action [7].
However, tree-based models facethe limitation of feature representation, that they can only work with meaningful features in order tomake sense to humans.
They do not naturally apply to unstructured data such as images and textsunless human-understandable information is manually extracted.
This paper aims to explain a black-box agent that works with unstructured data.
To make theexplanations understandable, we make use of the concept of prototypes from recent DNN modelsfor image classification [8, 9] and text classification [10].
A prototype is an instance (e.g. an image)from data that is representative of a prominent feature for a certain class.
For example, the redbellied woodpecker class could have a prototype for its red head and a prototype for its wings witha zebra-like pattern, etc.
Then, when there is a new input, the input can be compared against allprototypes trained from data.
If the input also has a red head and zebra-like pattern on the wings, theweighted sum of similarities in the red-bellied woodpecker class will be high, and thus the input will be predicted to be in this class
In this paper, we propose a policy explainercalled ProtoX, which explains an actionby relating an input state to a set of prototypical behaviors learned from the demonstrations from the agent.
Figure 1 showsan example of ProtoX explanation for anagent playing the SuperMario game.
ProtoX learns a set of prototypical states duringtraining, which are ofthe agent’s behavior.
In summary, ProtoX prototypes an agent’sbehavior into a small set of representativebehavior patterns, each represented by aprototype, and explains the agent’s actionbased on these basic patterns.
Therefore, akey determinant of the success of ProtoX isthe prototypes.
An important considerationwhen computing the similarity between aninput and a prototype is the visual similarity.
Our definition of similarity must appropriately balance the aspects of visual andcontextual similarity.
Otherwise, e.g. if twostates look entirely different, high similarity between them will confuse human usersand discredit the explanation.
In ProtoX, we design a self-supervised training step for the encoderto learn visual similarity. Usually, images that are closer to each other in time and receive the sameactions are more visually similar than those far away or receiving different actions.
Thus, we createa siamese net [11, 12] that utilizes a modified form of the time-contrastive loss.
Our pre-trainingroutine yields encoders whose feature spaces cluster together visually similar states which occurclosely in time and have the same corresponding actions.
In the supervised learning setting, visual similarity would be the only consideration, given the natureof image classification tasks.
However, in a reinforcement learning setting, two visually similar statescould receive different actions (such as the flip points we define later), and two visually dissimilarstates could represent a similar “scenario” that requires the same action.
Therefore, the similaritiesneed also need to take into account scenario similarity.
Here, a scenario represents a collection ofstates in which an agent performed the same action for similar reasons.
For example, in Figure 1, anaction RIGHT+JUMP has a high similarity score to prototypes that represent different scenarios forjumping, which could be that Mario encounters a pipe (the 1st prototype), a hole (the 3rd prototype),or preparing to hit a question mark brick (the 2nd prototype).
In fact, the scenario similarity bringsmore value to the explanation, since it helps users to think about what is common between twovisually dissimilar states that leads to the same action.
To allow the feature representations toincorporate scenario similarity, we add a near-isometry layer after the encoder, which applies a lineartransformation of the output of encoder to allow some states to move closer to each other.
ProtoX is trained end-to-end via imitation learning, by learning from the demonstrations of theblack-box agent to be explained.
We choose behavioral cloning [13] in this paper because it requiresminimal information from both the agent and the environment: only demonstrations from the agentand no access to the environment or the agent.
Therefore, it is most applicable in practice.
Manyother imitation methods, such as GAIL-based methods and inverse reinforcement learning methods,require more information from the environment or/and the agent, such as knowing the transitionfunctions, executing a policy during training, etc.
We summarize the contributions of ProtoX. First, ProtoX is the first to explain the actions of areinforcement learning agent via prototypes. Second, we design an encoder to preserve visualsimilarity for better interpretability via self-supervised learning and an isometry layer.
It allows
ProtoX to adapt to both visual and contextual notions of similarity. Third, while the existinginterpretable baseline with unstructured data. Lastly, ProtoX only needs to learn from demonstrations and does not need toaccess the environment, which makes it easier to use in practice.
Related Work
RL interpretation can be categorized into two main groups: inherently interpretable RL modelsand post-hoc explanation approaches.
Inherently interpretable RL models generate policies that areintrinsically understandable, such as Neurally Directed Program Search [14], fuzzy RL policies [15],hierarchical policies [16, 17, 18], or extract time step importance within episodes as strategy-levelexplanations [19].
Post-hoc methods, like the proposed ProtoX, aim to explain a given black-boxagent to help users understand the agent’s decision-making process towards an action.
The goal is notgood task performance but high fidelity in explaining a given black-box agent, regardless of the ownperformance of the agent.
Based on the type of data they work with, post-hoc methods can also be split into two areas.
The firstclass of methods only work with structured data, such as genetic programming [20], VIPER[7] andother rule-based methods [21, 22, 23], expected consequences [24], causal lens [25], complementary
RL [26], etc.
These methods often adopt existing forms of interpretable models from supervisedlearning, such as rules [7, 26].
However, since the explanations need to directly use state features,they do not naturally work with unstructured data.
To learn video games from pixels, they requiremeaningful features to be extracted manually .
The other type of post-hoc methods work withstructured data.
Examples include saliency-based methods [27, 28, 29, 26, 4], reward decomposition[30], and interestingness elements [31].
The explanations, however, are either only at the global levelor too abstract, such as saliency maps that only inform users of where the agent “looks at”, but do notdirectly explain the policy.
Another stream of work that aims to provide some interpretability stem from the family of imitationlearning.
For example, inverse reinforcement learning (IRL) methods [32, 33, 34] aim at learninga reward function under which the observed demonstrations of an agent is near-optimal.
Theinterpretability, however, is only achieved through the learned reward function with a pre-definedstructure (e.g., linear combination of state features) rather than understanding the policy itself.
Methods based on generative adversarial imitation learning (GAIL) [35] such as xGAIL [36] andinfoGAIL [37] provide indirect explanations to the learned policy in terms of global and local featureimportance or latent factors that govern the policy generation.
However, none of them provide a directexplanation of the expert’s actions under each state that are intuitive to human.
More importantly,all of the above methods require intensive interactions with the environment or the agent duringtraining (such as executing a policy), which are often not accessible in most practical situations.
Onthe other hand, our proposed ProtoX model is completely offline and requires no environment oragent interaction.
It also needs no additional information from the environment such as the transitionfunction, reward function, etc.
Prototype-based Policy Explainer
We consider a discrete-time partially observable Markov decision processes (POMDP), whichinclude video games such as the OpenAI Gym classic control games[38].
We denote the observation (e.g., frame of video game) at time t by o t and the C latest observations at time t bys t =
( o t , o t − 1 , . . .
, o t − C +1 ) .
We call s t the state at time t and denote the set of all states across alltimes as a state space S .
1 Let A be the action space of the POMDP and we consider a policy thatchooses an action a ∈ A at time t based on the C latest observations, namely, a mapping π : S → A .
Given a policy by an expert π ∗ : S → A , our goal is to build a prototype-based policy explainer thatis parameterized by ϕ , denoted as π ϕ : S → A , called ProtoX, such that π ϕ ( s ) =
π ∗ ( s ) , ∀ s .
Wetrain π ϕ via behavioral cloning, which proceeds with two steps.
First, we apply π ∗ to the POMDP togenerate a set of state-action trajectories and decompose all trajectories into a dataset of the expert’sactions at different states, denoted by D =
{ ( s i , π ∗ ( s i ))
i } .
Next, we train a classifier to predictactions from states.
We present the architecture of ProtoX, which consists of four main components, an encoder, anear-isometry layer, a prototype layer, and a fully connected layer.
See Figure 2 for an illustration ofthe architecture.
Below, we describe its main components, and then formulate the learning objectivewhich considers both the fidelity and the interpretability.
3.1 Feature Representations Preserving Visual and Scenario Similarity
A key determinant of the success of ProtoX is that the feature representation of the states that canrespect the visual similarity and reflect scenario similarity.
Therefore, ProtoX needs to recognizetask-relevant features from the images to benefit the downstream task, as well as understanding whatis considered “visually similar”.
However, we do not have labeled data that supervises an encoder todetermine which images are similar and which are not.
Therefore, we use a self-supervised learningapproach.
We construct an objective via contrastive learning and train an encoder f θ parameterizedby θ using a siamese variational auto-encoder (VAE) [39].
The contrastive portion of our objective issimilar to the time-contrastive triplet loss from [11], based on the idea that images that are closer toeach other in time are more likely to be visually similar to each other.
Our loss function differs slightlyin that it uses a time-contrastive quadruplet loss.
We use a quadruplet loss so that the encoder learnsthat states are the most similar when they occur close in time and are associated with the same action.
Thus, this encoder also incorporates functional and temporal similarity in states’ representations.
∗ .e. , π ∗ ( s t ) =
π ∗ ( s + t ) .
We constructtwo types of negative (dissimilar) states.
Wefind one negative state s − t among states thatare within the same temporal neighborhoodbut receive a different action than the anchor.
We find a second negative state s −− t amongstates that are outside the temporal window.
Since s −− t is far away from s t in time, it ishighly likely they look dissimilar.
Thus, aquadruplet ( s t , s + t , s − t , s −− t ) ∈ S 4 is formulated.
See Figures 3 for an illustration of the idea in Super Mario.
The quadruplet loss is then defined on ( s t , s + t , s − t , s −− t ) as
L quadruplet
[Equation]
m 1 and m 2 are referred to as margins.
m 1 is used so that, by minimizing L quadruplet , f θ views anchorsas at least m 1 closer to positives than to the first type of negatives.
The use of m 2 is to ensure that the two types of negatives are pushed at least m 2 away from each other.
The full loss function for thesiamese VAE is the quadruplet loss plus the conventional VAE loss function.
Pseudocode and furtherdetails for our pre-training algorithm can be found in the supplementary material.
Once f θ is trained, it will be frozen for the down-stream training, such that the property discussedabove is preserved.
Output of the Encoder The encoder is able to map visuallysimilar states closer in their encodings.
In general, we find thatthe most similar states to a given state are those that almost lookidentical.
The less similar states are generally those that could occurin a small window of time before or after the given state.
We showthe efficacy of the encoder by aggregating the most similar 30 statestogether, i.e., the 30 states with the smallest || f θ ( x ) − f θ ( x ′ )
||
See Figure 4 for an example that shows what states the pre-trainedencoder views as similar in Super Mario Bros. level 1-1.
Thisexample shows that the states similar to the input are those where
Mario is in the process of jumping over a hole in the ground.
Learning Scenario Similarity Note that the training of f θ doesnot involve the downstream tasks, thus the feature representations,while respecting the visual similarity, may not consider the taskrelevance, i.e., the scenario similarity, which is task-specific andagent-specific.
Similar failure modes in contrastive learning areexplored in [40].
Therefore, to overcome the possibility of thef θ failing to be useful to the downstream task, we add a linearlayer A : R C ′
× H ′ × W ′ → R C ′ H ′ W ′ after the encoder.
Since A islearned during training, it will further adapt the representations ofthe input states to the downstream RL task.
Howsever, we alsowant to make sure this linear transformation of A does not undo
ProtoX’s ability to understand visual similarity.
Thus, A needs tobe nearly isometric, such that, via[41], if (cid:13)(cid:13)
A ⊤ A − I (cid:13)(cid:13) 2 2 ≤ δ (whereδ > 0) , then for all data x, x ′ , ∥
Af θ ( x ) − Af θ ( x ′ ) ∥ 22 ≤ (1 + δ ) ∥ f θ ( x ) − f θ ( x ′ ) ∥ 22 .
Therefore, wewill include a penalty term in the objective such that the isometry layer allows some evolution of theencoder
Af θ while still retaining the way f θ learned to separate dissimilar states during pre-training.
[Equation]
where I is the identity mapping of the appropriate dimension.
The t-SNE plots in Figure 2 explain why the isometry layer benefits the downstream task.
We showthe 30 states most similar to the same input in Figure 4, this time after the isometry layer, i.e., the 30states with the smallest ||
Af θ ( x ) −
Af θ ( x ′ )
||
In contrast to Figure 4, in the example in Figure 5,
ProtoX identifies scenario similarity, as it considers jumping over a hole is similar to jumping over agreen pipe.
For more analysis of the isometry layer, please see the supplementary material.
3.2 Learning Prototypes
We then add a prototype layer after the isometry layer that automatically learns K prototypes for eachaction in A , represented by { p ak } a ∈A k =1 , ··· ,K .
Note that the prototypes are learned via an end-to-endtraining.
Each prototype has a shape of 1 × C ′ H ′ W ′ , which is the latent representation of an actualimage from training data.
Given an input state x , ProtoX computes the distances ∥
Af θ ( x ) − p ak ∥ fork = 1 , . .
, K and a ∈ A .
It uses these distances to compute similarity scores between the image andprototypes, following the function from [42]:
[Equation]
where β controls the concentration of similarity scores and is set to 0.05 in all our experiments.
The similarities are then fed into a fully connected layer with a weight matrix W ∈ R |A| K ×|A| with
W = ( w aa ′ ,k ) a ∈A k
=1 ,...,K,a ′ ∈A .
The subscript represents the prototypes and the superscript represents the output actions, and there are |A| K prototypes in total.
The linear layer outputs |A| evidencescores, each of them linearly combines the contributions from all prototypes:
[Equation]
The linear model is chosen for its ease of explanations.
It could be replaced by other interpretablemodels like decision trees, following the idea in [9].
The evidence score is computed for every actionand the final prediction (action) is made as argmax a ∈A e a ( x, W ) .
Given this decision rule, we nextdesign an objective function that encourages the model to associate scenarios with an appropriateaction to perform.
3.3 Objective Function
Since π ϕ is a post-hoc explainer for an expert π ∗ , we consider two desiderata when training π ϕ :
1)fidelity of imitation, and 2) interpretability of the prototypes.
We construct the following terms inan objective function for the two purposes.
Fidelity A cross-entropy term, denoted as 1 n (cid:80) ni
=1 L CE ( y i , ( e a ( x i , W )) a ∈A ) , is included to makesure ProtoX can reproduce the actions of the black-box expert as accurately as possible, which reflectsthe fidelity in the explanation.
Here, e a is defined in (4).
Separation and Clustering Terms
The terms Clst (Clustering) and Sep (Separation) are takenfrom the ProtoPNet [8].
The Clst term encourages prototypes to be close to the encodings of traininginstances that have the same action label as the prototypes.
The Sep term encourages prototypesto be far from encoded training instances with different action labels as the prototypes.
This termenforces consistency with our time-contrastive loss, since both state that image representations shouldbe dissimilar if their corresponding actions differ.
The Clst and Sep terms together encourage themodel to learn prototype scenarios such that similar scenarios have an associated appropriate actionand dissimilar prototype scenarios have different appropriate actions. Sep
[Equation]
Representability term The Rep (representability) term encourages prototypes to be similar to someencoded training instances, to insure the representability of the prototypes.
Because our pre-trainingroutine already ensures that encodings f θ ( s t ) are meaningful representations, we use this Rep termto further ensure that the prototypes are representative of the data.
By continuously ensuring thatprototypes are close to actual encodings of training data, we justify the success of prototype projectionsteps that will be explained later.
This term is inspired by the Evidence term of [42].
[Equation]
Isometry Term As discussed previously, the isometry term in formula (2) allows some evolutionof the encoder
Af θ while still retaining the way f θ learned to separate dissimilar states.
Therefore, the objective function is
[Equation]
where λ i ≥ 0 , i = 1 , . .
, are weighting parameters subject to tuning.
The first term encourageshigh fidelity and the last four terms are for prototype shaping.
3.4 Downstream Training Routine
We train our model with the standard behavior cloning (BC) algorithm, but with the aforementionedsurrogate loss L .
This training routine is essentially inspired by that of the related ProtoPNet[8].
Prototype Projection
The standard image classification training pipeline is augmented with aprototype projection step every 25 epochs.
During this step, the parameter data of each prototype in ProtoX is replaced with the encoding of the training datum it is most similar to, i.e.,
[Equation]
, where
[Equation]
Then, the A layer and the prototype vector are frozen, and the fully connected layer is trained byitself for 20 epochs.
The representability term in the objective ensures the success of the projectionstep since each prototype is already close to at least one training instance, thus projecting them doesnot cause a significant change in the loss function.
Prototype Merging Once a ProtoX model is trained, and we project prototypes into the spaceof encodings of real states, we often find that there is a significant amount of repetition amongthe prototypes.
To eliminate the redundant prototypes, we create a new prototype layer consistingonly of the unique prototypes.
We then also create a new fully connected layer whose weightscapture the redundancy present in the previous prototype layer.
Supposing that prototype i = i 1is the same as prototypes i 2 , ..., i k after projection, the connection between prototype i and actionl in the fully connected layer is w il = (cid:80) lj
=1 w i j l .
We find that this prototype merging procedureeffectively reduced the number of prototypes while retaining the performance of the network.
Thisalso means when we build ProtoX, we can over parameterize it by allowing it to learn a large numberof prototypes, but they will eventually be projected to a small number of unique prototypes.
Experiments
We evaluate ProtoX on four Atari game environments from OpenAI Gym and compare the resultswith two black-box and one interpretable baseline.
We first simulate a near optimal black-box agentand then use different methods to imitate the black-box.
We then evaluate the fidelity of differentmethods, measured as the accuracy of their predictions compared to the agent.
In addition, weevaluate the sensitivity of ProtoX on flip points, to test whether ProtoX can capture important timesteps in a reinforcement learning task.
Then we investigate the interpretability of the explanationsgenerated by ProtoX. Finally, we simulate a bad agent and uses ProtoX to diagnose its bad behaviors.
4.1 Experimental Setup
We use four classic control tasks from OpenAI Gym, namely, Pong, Seaquest and two levels from
Super Mario Bros[43].
For each game, we generate a near-perfect black-box agent, denoted as π ∗ .
Pong and Seaquest use PPO[44] models from the stable-baselines3[45] package.
The experts for
Super Mario are also PPO models from [46].
Then we train ProtoX and other methods to imitate eachagent.
See the Appendix for hyperparameter settings and further details on our experimental design.
We compare ProtoX with three baselines, namely, 1) VIPER[7] which builds a decision tree modelto output policies.
Since VIPER only works with structured data and does not directly work withimages, we extract features from the Pong and Seaquest games.
Due to the complexity of Super
Mario, we do not benchmark VIPER on the Super Mario games. 2) a GAIL-based method that learnsfrom observational data generated by an agent, called GAIfO[47].
GAIfO is a black-box model.
Notethat unlike ProtoX, GAIfO still interacts with the environment.
3) ResNet-BC, which follows thesame training procedure as ProtoX except using a ResNet18 model to work with the game frames.
ResNet-BC is a black-box model that does not explain an agent.
We run ResNet-BC to evaluatehow much fidelity is lost due to using prototypes; Both ProtoX and ResNet-BC are trained with thebehavior cloning algorithm using 30,000 state-action pairs obtained via an expert trained with PPO.
4.2 Fidelity Evaluation
We first evaluate the fidelity of each method, i.e., how often each method agrees with the PPO agent π ∗ .
For each game, we let the agent generate a test set of 10 , 000 state-action pairs D test =
{ s i , π ∗ ( s i ) } i .
We then apply each method to the states and record their predicted actions { π ( s i ) } i , which iscompared against { π ∗ ( s i ) } i to compute the accuracy, i.e., (cid:80) 1 ( π ∗ ( s i )=
π ( s i ))
|D test | recorded in Table 1.
[Table]
ProtoX achieved much better fidelity than theexisting interpretable method, VIPER, since
VIPER is a decision tree-based model and doesnot allow image state inputs.
We set the maximum tree depth to 10 for both games, whichwill produce a tree with thousands of nodes forboth games.
Increasing the maximum tree depthwill slightly increase the fidelity by less than3 percent, but it will significantly increase thenumber of nodes in the tree.
Yet VIPER stillperforms much worse than ProtoX. In addition, as numerical features (coordinates of a ball, or asubmarine, etc) are more difficult to understand than images in video games.
GAIfO also performspoorly.
GAIL based methods are typically only applied to low dimensional problems like jointdynamics [35, 47].
When applied to high-dimensional Atari problems, GAIL has been shown toachieve poor performance [48].
We arrived at a similar conclusion with our experiments.
The comparison of ProtoX and ResNet-BC, even though ResNet-BC is not interpretable, shows
[Table]
that the interpretability design in ProtoX has small negative impacton the fidelity, a 3% drop on average.
When training ProtoX, wepurposely over-parameterize the network by using more prototypesthan should be necessary.
For example, when training ProtoX for
Super Mario Bros.
World 1-1, we use 350 prototypes initially.
However, after the prototype projection and merging step, we are ableto significantly reduce the number of prototypes in a trained ProtoXmodel, thus making each explanation low in complexity.
In the caseof Super Mario Bros, we can reduce the number of prototypes from350 to only 17 while retaining relatively high fidelity.
We report thenumber of prototypes for ProtoX models in Table 2.
The small number of prototypes ensure that theexplanations are easy to follow.
4.3 Sensitivity to Flip Points
For a reinforcement learning agent, usually the most important explanations are for time steps where
[Table]
the action changes from the previous action.
This isbecause there’s often no significant pixel change intwo consecutive game frames, and thus a change inthe action would be worth investigating.
Therefore,if a state receives a different action from its previousstate, we call it a flip point .
The set of flip points canbe represented as F = { s t ∈ S| π ∗ ( s t )
̸
= π ∗ ( s t − 1 ) } .
We show an example of flip point in Figure 6.
Attime step t − 1 , the agent presses RIGHT + SPEED to run to the right.
In the next time step, theagent presses RIGHT + JUMP to begin jumping over the green pipe.
We evaluate the sensitivity of
ProtoX to flip points, i.e., when the black-box agent changes its action, can ProtoX catch up?
Wedefine sensitivity as the percentage of successful flips, i.e., sensitivity ( π ϕ ) =
(cid:80) x ∈ F 1 ( π ∗ ( x )= π ϕ ( x )) |F| .
We collect a total of 10,000 flip points from the test set and report the fidelity of ProtoX on this set in
Table 3.
This shows that even though flip points are visually similar, ProtoX realizes that they differand is able to reproduce the agent’s action.
step t − 1 with a prototype of running to the , unimpeded by any obstacle.
In contrast, ProtoX recognizes the jumping action in time step t as being similar to another scenario where the expert agent jumped over a green pipe.
The first prototype is for the action “FIRE”, which means the submarine tries to shoot enemies to gain points.
The prototype represents a situation where the player has shot a missile to the right at an enemy coming in from the right screen edge.
The second prototype is for the action “UP”, which means the submarine will move up for oxygen.
This prototype represents a scenario where the submarine has saved divers and there are no enemies in the front and so it is safe to go up.
4.4 Prototype Interpretation
To investigate the interpretability of ProtoX, we show an example of the prototypes in a ProtoX modellearned from Seaquest.
The game is an underwater shooter in which the player controls a submarine.
Its three goals are 1) to shoot enemies, 2) to rescue divers, and 3) to surface for oxygen when needed.
In Figure 7, we show 2 examples of prototypes.
To understand what scenario each prototyperepresents, we overlay 60 most similar states by plotting the full-color representations of each state with low transparency over one another. Thus, we can faintly what each scenario represents.
Forexample, for the first prototype in Figure 7, the submarine is located in several places near the bottomcenter of the screen.
But the common feature in this scenario is the enemies to the right of thesubmarine.
The common feature for the second prototype is that the submarine is in the same generalposition as in the prototype, but with varying air tank level and varying number of rescued divers.
Diagnosing Bad Behaviors
The explanations provided by ProtoX can also be used to diagnosewhy an agent makes mistakes.
To simulate this use case, we train a bad agent that never jumps for the Super Mario game, and consequentially is always killed by the first goomba it encounters.
We thentrain a ProtoX on trajectory data from this agent to see whether ProtoX can explain why the agentfails to jump to avoid the goomba.
Figure 8 (a) shows that Mario sprinted to the right into the goomba. Figure 8:
For the bad agent, ProtoX views the prototype in b) as similar without consideration of any strategic elements.
In particular, only the sky seems to matter.
In contrast, ProtoX for the good agent selects a different prototype (c) and seems to view the immediate area of the Mario hitting the brick to kill the goomba as important, which explains why the good agent chooses action “JUMP”: he needs to jump to kill the goomba.
ProtoX explains this by relating thisstate to similar prototypes and weshow the most similar prototype in
Figure 8 (b), which is a state in thebeginning of the game when Mariocould sprint to the right because thereare no obstacles. To understand whythe bad agent thinks the input is similar to the prototype, we create an importance map over the bad agent’s prototype. The importance map is generated by masking the prototype withpatches and evaluate the change insimilarity between the masked prototype and the input state. A biggerchange indicates higher importance.
Then, to highlight the most importantarea, we keep only the top 95% of pixels with the highest values.
The restare set to 0.
See the supplementary material for how to generate the importance plot. The importancemap is overlaid on the prototype in Figure 8 (b). We deduce from the importance plot that the badbehavior is due to the agent only looks at the blue sky to determine the similarity. To contrast with the bad agent, we then feed the same input state to another ProtoX trained on agood agent.
The new ProtoX identifies a different prototype for the input, shown in Figure 8 (c).
Inaddition, the importance plot shows that new ProtoX looks at the strategic elements of the games.
Specifically, the importance highlights Mario jumping to hit a brick to kill the goomba.
ProtoX thinksthis is similar to the input state, where Mario should also jump to kill the goomba.
Conclusion
We propose a prototype-based policy explainer, ProtoX, for explaining black-box reinforcementlearning agents.
ProtoX explains an agent by prototyping its behavior into different scenarios, eachrepresented by a prototype.
Then ProtoX explains an action by comparing an input with prototypes,computes the similarity scores, and then calculates accumulated evidence for each action.
ProtoXtakes into account both visual similarity and scenario similarity.
We use self-supervised training andcontrastive learning to learn the notion of visual similarity from expert demonstrations, and allow
ProtoX to learn scenario similarity by using an isometry layer.
Results show that ProtoX achievedhigh fidelity compared to VIPER, a black-box model with similar network architecture, and also a
GAIfO model whose policy network had a similar architecture.
Discussion of Limitations
The interpretation of the prototypes requires basic understanding of thetask.
For example, to interpret the prototype for UP in Figure 7, users need to know the basic logic forplaying the game, thus can comprehend why the submarine goes up in that scenario.
In addition, wehave also noticed in the Seaquest game that the expert anticipates the arrival of enemies and shoots atthem before they even enter the screen.
Thus, ProtoX learns prototypes showing the expert shooting [Figure] at nothing.
ProtoX’s mode of interpretation does not inform the users of such anticipatory behavior.
One would not be able to understand such prototypes without a basic understanding of the game.
Also, ProtoX may learn similar and redundant prototypes.
Some pruning techniques could be createdto further improve the quality of prototypes.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See the Section
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5 Conclusion
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] N/A
(b) Did you include complete proofs of all theoretical results?
[N/A] N/A
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We include the
code, data, and necessary instructions in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] All training details are provided in the supplementary material.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No] Experiments were performed using a single seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] All experiments were done on a
[Table] system with an RTX 3060Ti GPU, AMD Ryzen 7 3700X 8-Core Processor, with 32GB [Table]
RAM
[Table]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the models
and libraries we used.
[Table]
(b) Did you mention the license of the assets?
[Yes]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
We did not use such data.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
We did not use such data.
All data are from
[Table]
OpenAI Gym
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]