Are Gradient-based Saliency Maps Useful in Deep Reinforcement Learning?
Abstract
Deep Reinforcement Learning (DRL) connects the classic Reinforcement Learning algorithms with Deep Neural Networks.
A problem in DRL is that CNNs are black-boxes and it is hard to understand the decision-making process of agents.
In order to be able to use RL agents in highly dangerous environments for humans and machines, the developer needs a debugging tool to assure that the agent does what is expected.
Currently, rewards are primarily used to interpret how well an agent is learning.
However, this can lead to deceptive conclusions if the agent receives more rewards by memorizing a policy and not learning to respond to the environment.
In this work, it is shown that this problem can be recognized with the help of gradient visualization techniques.
This work brings some of the best-known visualization methods from the ﬁeld of image classiﬁcation to the area of Deep Reinforcement Learning.
Furthermore, two new visualization techniques have been developed, one of which provides particularly good results.
It is being proven to what extent the algorithms can be used in the area of Reinforcement learning.
Also, the question arises on how well the DRL algorithms can be visualized across different environments with varying visualization techniques.
Introduction
Due to the success achieved in recent years by Deep Reinforcement Learning [7] [2], Industrial applications are becoming increasingly tangible [13].
Research is being conducted on 3D map reconstruction for autonomous cars where DRL can be one of the solutions [18] and also in the ﬁeld of AUVs [3] as well as many other applications that interact with the real world.
Once DRL algorithms are implemented on physical systems and interact with the real world, these systems can be dangerous for themselves and humans.
For this reason, debugging tools are needed to understand why the agent behaves that way and whether the agent is making the right decision for the correct reason and not making a right decision for the wrong reason [11].
Deep reinforcement learning algorithms are nowadays interpreted and measured by the rewards the agents can get.
For this reason these agents are called black box algorithms and are criticized.
This makes them difﬁcult to use in critical real-world applications.
This paper reveals that visualization techniques are a powerful debugging tool, that provides much more information than interpreting rewards.
With the help of Guided Backpropagation it is even possible to locate the error in a certain layer or stream in a neural network.
Moreover it is shown that Grad-Cam methods can deliver results very early in training in case of very badly trained networks.
This is a big advantage especially for off-policy algorithms that take a long time to explore at the beginning of the learning process.
With off-policy algorithms it can take a few days until the rewards start to increase so that the developer can determine if the neural network is learning at all [14].
Contributions.
For DRL, the claim made by Adebayo et al.
[1], that Guided Backpropagation does not visualize the desired regions, but through partial input recovery it works as a kind of edge detector, is shown not to be accurate.
Also, the claim that gradient methods can be difﬁcult to interpret, because, when answering the question "What perturbation to the input increases a particular output?", gradient methods can choose perturbations which lack physical meaning [5], is shown not always to be an issue.
Furthermore two new visualization techniques were developed, one of which provides particularly good results.
These were compared and analysed with 4 other popular visualization techniques.
Their advantages and disadvantages are discussed and different ﬁelds of application for the respective visualizations are suggested.
Related Work
In this section, previous work is presented and discussed.
As already mentioned, there is currently not a lot of work dealing with the topic of visualization in the area of deep reinforcement learning algorithms.
First two works from the ﬁeld of deep reinforcement learning are presented and then and then visualization techniques from the ﬁeld of image processing.
Semi Aggregated Markov Decision Processes.
The authors which introduced Semi Aggregated
Markov Decision Processes [17] used the Atari 2600 environments as interpretable testbeds, they developed a method of approximating the behavior of deep RL policies via Semi Aggregated Markov
Decision Processes (SAMDPs).
They used the more interpretable SAMDPs to gain insights about the higher-level temporal structure of the policy.
From a user perspective, an issue with the explanations is that they emphasize t-SNE clusters and state-action statistics which are uninformative to those without a machine learning background.
Perturbation-based saliency methods.
Another recently published work shows Perturbation-based saliency methods for the visualization of learned policies by Greydanus et al.
Their approach is to answer the question, "How much does removing information from the region around location change the policy?".
The authors deﬁned a saliency metric for image location as:
[Equation]
The difference π u ( I 1: t ) − π u ( I (cid:48) 1: t ) can be interpreted as a ﬁnite differences approximation of the directional gradient ∇ ˆ v π u ( I 1: t ) where the directional unit vector ˆ v denotes the gradient in the direction of I (cid:48) 1: t .
In other words, is looking at how important were these pixels for the policy.
By checking how strong the policy changes after removing some informations from the image.
They use the same approach to construct Saliency Maps for the value estimate V π too.
Greydanus et al.
[5] claims that gradient-based saliency methods do not yield well interpretable results.
When answering the question "What perturbation to the input increases a particular output?", gradient methods can choose perturbations which lack physical meaning.
Gradients.
The basic idea of backpropagation-based visualizations is to highlight relevant pixels by propagating the network output back to the input image space.
The intensity changes of these pixels which have the most signiﬁcant impact on network decisions.
Despite its simplicity, the results of saliency map are normally very noisy which makes the interpretation difﬁcult.
Guided Backpropagation.
The idea behind guided backpropagation is that neurons act like detectors of particular image features.
It is interesting in what image features the neuron detects and not in what kind of features it doesn’t detect.
That means when propagating the gradient, the ReLu function set all the negative gradients to zero.
[8] With other ways we only backpropagate positive error signals and we also restrict to only positive inputs.
Gradient-weighted Class Activation Mapping.
One of the problems when it comes to using CAM is that a modern neural network not only consists of convolution layers but of different layers such as LSTM, MaxPooling, etc.
GradCAM is an extension of CAM and it is broadly applicable to any
CNN-based architectures.
In order to obtain the class-discriminative localization map Grad-CAM
L c Grad − CAM ∈ R u
× v of width u and height v for any class c .
First, the gradients y c of class c are calculated up to the desired convolution feature map activations A k .
An average pooling over the width and height dimensions is then carried out [9].
[Equation]
Weight α ck represents a partial linearization of the deep network downstream from A , and captures the "importance" of feature map k for a target class c .
On the end it performs a weighted combination of forward activation maps with a ReLU fuction.
[Equation]
The result is a heatmap of the same size as the convolutional feature map.
This feature map must then be enlarged to the size of the original image and place it over the image to get the ﬁnal result.
If Grad-Cam is multiplied by Guided Backpropagation with the Hadamard product we get Guided
Grad-Cam [9].
Gradient Methods.
Adebayo et al.
[1] deals with the informative value of visualization techniques.
In this work, various visualization techniques were compared and checked whether the techniques actually depend on the model, the training data or whether it partially reconstructs the image.
In other words, their work claims that Guided Backpropagation and Guided GradCam act like an edge detector, which means that they show the desired positions without showing the learned model.
In their work they randomize the weights of a model starting from the top layer, successively, all the way to the bottom layer.
This procedure destroys the learned weights from the top layers to the bottom ones.
Their results indicate that Guided backpropagation and Guided GradCam do not visualize the learned model, but instead partially reconstruct the image.
They interpret their ﬁndings through an analogy with edge detection in images, a technique that requires neither training data nor model.
Laplacian Operator in Image Processing .
To understand one of the results we will derive the kernel of the Laplacian ﬁlter which is used for edge detection.
One approach for the design of directionally invariant high-pass ﬁlters for image processing (also called edge detectors) is to use second-order derivation operators, e.g. the Laplace operator [16].
For continuous functions, the
Laplace operator is deﬁned by:
[Equation]
We approximate the partial derivatives by difference equations and thus obtain :
[Equation]
Thus with ∆ x, ∆ y = 1 (except for the sign) the mask for the Laplace operator is :
[Equation]
Numerous other approximations of the Lapace operator are possible.
Examples are the following masks (their transfer functions have the form known from the Gaussian and binomial distributions)[16]
L 2 = (cid:34)− 1 −− −− 1 − (cid:35)
L 3 = (cid:34)− − − − (cid:35) L 4 = (cid:34) − 1 − 2 −− −− 1 − 2 −(cid:35)
Experimental Setup
Experiments were performed in two different environments a simple (Breakout-v0) and a complex one (Seaquest-v0) from OpenAI Gym [4].
Four different agents were implemented.
DDDQN (4 frames as input) [10], Splitted Attention DDDRQN (with LSTM), and two on Policy gradient algorithms
A3C (3 frames as input) [6] and an A3C Agent with LSTM.
The Splitted Attention DDDRQN was by far the best trained agent and receives most of the points (Seaquest-v0 9521 Points) that is why most of the results are referring to this agent [12].
In order to better examine the visualization methods and to enable better interpretation, the original frames of the games are placed over the gradients with a 50% opacity.
In this way it is possible to assign the gradients to the features and to better interpret the results.
With the off policy algorithms not only the output is visualized but also the Q-value and the Advantage stream.
To proof the results of Ziyu Wang et al.
In order to maintain comparability between the visualization techniques, the same state was always visualized in this work.
This also applies to the visualization of the actor and critic agents, as well as the visualization of the value and advantage streams.
Two new visualization techniques have been developed and the following visualization methods have been implemented.
Some features of the implementation are discussed below.
Gradient and Guided Backpropagation
Compared to image processing, where the gradients of the guided backpropagation or gradient method are normalized over the image, the gradients in a video can also be normalized over the entire video.
For this reason, both have been implemented and tested.
First, the gradients output were normalized for each state.
Then the gradient and guided back propagation method around the whole video was normalized and tested.
In the visualizations, the gradients were checked in relation to different output layers.
In particular, this work examines the advantage and the value stream in the off-policy algorithms in more detail, as well as the last layer that delivered the best results.
In the actor critic methods both outputs of the NN were examined.
Grad-Cam and Guided Grad-Cam.
With the Grad-Cam and Guided Grad-Cam method, the visualization can be applied to different convolutional layers.
An example of what the differences look like is also shown.
The best results were achieved on the ﬁrst convolutional layer across all agents and across all environments.
For this reason, the results of the Grad-Cam methods are usually applied to the ﬁrst convolutional layer.
G1Grad-Cam and G2Grad-Cam
Two new visualization techniques were developed and will be presented in this work, which are also compared.
Both visualization techniques are a further development of Grad-Cam and Guided Grad-Cam.
The idea behind it is the same as for guided back propagation, that neurons act like detectors of particular image features.
And we are interested in what image features the neuron detects and not in what kind of features it does not detect.
That means when propagating the gradient, we set all the negative gradients to 0.
In the further course,
Grad-Cam with the Guided Model is called G1Grad-Cam and GradCam with the Guided Model multiplied by Guided backpropagation is called G2Grad-Cam.
Experimental Results
Since a state usually contains several frames, the results of the gradient methods (unless other-wise stated) are related to the last frame in the sequence.
With the Grad-Cam methods the results (unless otherwise stated) relate to the ﬁrst convolutional layer.
Furthermore, the word stable is deﬁned in the context of visualization techniques in this paper as follows: gradients or visual highlights that can be seen on most frames in a video.
Gradients or visual highlights that can be seen on every 4th, 5th (or even less) frame in a video are called not stable .
First we discuss the visualization techniques, then we look closer at the class discriminative visualization algorithms and ﬁnally we make a hypothesis about the importance of negative gradients in backpropagation algorithms.
Detailed results are all provided in the appendix (Figures 2 to 78).
4.1 Visualization Algorithms
Gradient.
Gradients method delivers a lot of noise which often mixed with the important features.
It is only very poorly suited for debugging neural networks in the deep reinforcement learning area.
We get similar results with the agent in all environments.
This can be seen in Figures: 2, 13, 14, 21, 29, 31, and 50.
Guided Backpropagation.
Guided backpropagation delivers the best and most stable results under all environments and among all agents.
Also, that it shows negative gradients is a very good indication of how well an agent is trained.
It can be seen that the better the agent has been trained, the stronger the transition between negative and positive gradients.
Another interesting observation that guided back propagation (compared to the Grad-Cam methods) did not visualize the agent so well during the breakout game.
Even in the DDDQN network, where the agent was well trained, only weak gradients could be recognized on the agent himself on Figures 19 and 3.
In contrast, on Seaquest, which is a much more complex environment and the agent was not well trained, very strong gradients were displayed on the agent itself, as Figures 13 and 14 show.
Another comparison that has been made in this work is how the visualization of the guided backpropagation algorithm changes with respect to normalization.
In other words, how does the visualization change when the normalization of the gradients is carried out over a frame compared to when the normalization was created over the entire video.
The results showed that when the frame was normalized, the gradients in the video ﬂashed more strongly.
With normalization across the entire video, the transitions from the frames were smoother.
In this way, the less known / visited states can be recognized because the gradients on which are weaker to recognize.
This method is also the most suitable for visualizing the differences between the advantage and the value stream in the off-policy algorithms.
Guided back propagation was able to identify most of the features in any environment and these were also very stable across the video.
During the development of the split attention DDDQN agent, the guided back propagation method for debugging was also used.
An interesting ﬁnding that was made during development was that although the neural network had several errors, results were still very good (over 3000-3500 points, similar results were achieved by the normal attention DDDRQN network).
Only after the Advantage and
Value Stream was visualized did it become apparent that the network only learned on the Advantage side and that the gradients on the value side were very strong but chaotic.
Based on the points, the errors that were made during programming would not have been noticed.
Altogether two errors could be discovered and they could even be localized in the value stream.
One mistake was incorrectly linking the layer and another when merging the two streams.
When developing the A3C with LSTM network, guided back propagation was also used to debug the network.
Before a working A3C with LSTM was developed in this work, attempts were ﬁrst made to train the agent with bidirectional layers.
This was unsuccessful.
In the visualization, no gradients were displayed when the gradients from the output layer to the input layer were calculated (the output gradients during printing were also 0).
Then the network with guided backpropagation was examined more precisely with this visualization technique, because in contrast to the Grad-Cam methods, the gradient methods calculate the gradients from each individual layer to the input, this means that the individual layers can be examined.
It turned out that the agent started to learn in the ﬁrst three layers and no longer from the bidirectional layers.
When these layers were replaced by LSTMs, the agent immediately began to learn even in the higher layers.
Grad-Cam.
This method also showed stable results (based on the ﬁrst convolutional layer), even if the results were often issued in inventory, especially with less well-trained agents, as seen in Figure 46.
Looking at the higher layers, some features could also be visualized, but here the visualization was inverted more often, as shown in Figure 41, and the speciﬁed position of the features in the visualization algorithm was less precise, shown in Figures 33 and 37.
One advantage over the guided backpropagation method which is noticeable, is that with less well trained agents, the visualization (even if mostly often inventoried) was able to visualize the features faster and without interference.
However the G1Grad-Cam could beat these results. Grad-Cam was able to achieve better results in the Breakout environment (see Figure: 4), where the agent could be visualized relatively poorly with guided backpropagation.
Another advantage of the grad cam method (at least for neural networks without LSTM) compared to guided backpropagation is that, as can be seen in Figure 4, the time ﬂow can be visualized.
Means how important the previous frames were.
Due to the negative gradients that exist in guided backpropagation, a superimposition or averaging of the gradients from all input frames would not provide a similar result, since the gradients would cancel each other out with guided backpropagation.
Since the Grad-Cam has a ReLU function, only positive results are displayed.
However, this has the disadvantage that we cannot differentiate as well if an agent is well trained.
The visualization differences are minimal as can be seen in the Actor Critic between Figures 69 and 70 and different between the Splitted Attention DDDRQN (See Figure 23) and the DDDQN Agent .
However, the results with Grad-Cam are better with poorly trained agents and have less disruption than with guided backpropagation.
One reason for this could be that the grad-cam carried an average pooling across the height and width of the dimension are carried out.
With average pooling, the tendencies could be displayed better and there would be fewer disturbances due to averaging.
This could be the reason why by Grad-Cam by weakly trained networks achieve better results than with guided backpropagation.
A disadvantage compared to guided back propagation, which can be seen in all agents is that not all features can be visualized.
By the breakout environment the area of the image where the agent breaks through the last line and receives many points could be not visualized (see Figure 3 and Figure 4).
In Seaquest, Grad-Cam was unable to visualize the oxygen bar, nor the divers collected.
Guided Grad-Cam and G2Grad-Cam.
The combination of guided back propagation and one of the
Grad-Cam methods has proven to be very unstable.
One reason is that not all features are displayed in the Grad-Cam method, as has already been mentioned.
But the main reason is that the results are shown often inverted by the Grad-Cam method.
Due to the inversion, the areas with the features have a value of 0, which means that the multiplication with the guided back propagation also results in 0.
Basically, with these two methods, no additional knowledge can be obtained that has already been obtained with guided back propagation or Grad-cam.
G1Grad-Cam
The G1Grad-cam method gave much better results for the DDDQN agent in the game breakout and also the time ﬂow than with the Grad-Cam method, as shown in Figure 6 (G1Grad-Cam) versus Figure 4 (Grad-Cam).
Interestingly, the G1Grad-Cam method has problems visualizing the features in game Seaquest.
The G1Grad-Cam method shows less inverted results than the Grad-Cam method.
However, especially with less well-trained agents, the results are not stable or not exactly in position.
Basically, the method only convinced the DDDQN agent in the game Breakout.
Here G1grad-Cam was able to display the ball and the agent as well as their past positions very clearly than all other visualization techniques.
However, the upper breakthrough area could not be visualized as the guided back propagation method [-did.
-]did.
An interesting aspect of the poorly trained splitted attention agent is that in Seaquest the G1Grad-Cam method gives the best interpretable results over all visualization methods, as shown in Figure 47 (in contrast to the well trained one, where it gives no results).
The neural network was saved after 325 episodes (for comparison, the well-trained neural network was saved after 5300 episodes).
After 325 episodes the neural network has done about 250 000 steps and 75% of them have chosen a random action (because of exploration).
After this short period of time, the G1Grad [Table]
For comparison: After 1500 episodes it can be determined via the rewards that the neural network is learning.
This means that with the G1Grad-Cam method it is possible to recognize up to 5 times faster if the neural network is learning at all.
This makes it a helpful debugging tool.
4.2 Action Discriminative Visualization Algorithms
The fact that the agent has learned to recognize features does not mean that the agent has learned how to behave.
The agent recognizes e.g. the oxygen bar and understand when it ends that the episode will end, but it has not learned what to do about it.
It is similar with the ﬁshes that the agent recognizes them but does not necessarily mean that the agent will avoid the ﬁsh or knows what to do with it.
For this reason, action discriminative algorithms were examined more closely.
The investigations did not give any indication that Grad-Cam and G1Grad-Cam, which are action discriminative algorithms, have a visible visual relationship between the state and the action taken.
The expectation that Grad-Cam would emphasize a particular ﬁsh because the agent would swim away from a ﬁsh or target a ﬁsh, could not be conﬁrmed.
4.3 Hypothesis about Negative Gradients in Guided Backpropagation and their Meaning
Apparently negative gradients also play a role.
With the DDDQN agent, the negative gradients were time-dependent.
The newest frame has the most positive and the oldest most negative gradients and the negative gradients are always shown where the ball was or will be.
One reason for this could be the different architecture of the neural network which only calculates the error of the selected action but not the error of the non-selected actions compared to the other developed CNN.
With the other agents, it can be observed that the negative gradients did not form in a time-dependent but position-dependent manner.
The strong negative gradients have formed around the features.
It can also be seen that the stronger the positive and negative gradients, the better the agent.
A possible explanation could be that with guided backpropagation the gradients resemble a matrix or the kernel of an edge detector.
And the Laplacian Operator has strong positive and negative values in the matrix to detect edges.
Backpropagation could create a matrix that has learned to better identify features with strong negative and positive values, just like with a Laplace operator.
Figure 1 shows that strong positive gradients are followed by strongly negative gradients and some small positve again (compare in Equations 7 and 8, Equation L 1 and L 3 ).
The Laplace operator has a similar structure.
The difference is that the Laplace operator detects edges and not more complex features.
Guided backpropagation would create a kind of
Laplace matrix for features based on this hypothesis.
This could be a possible explanation for the role of negative gradients.
Conclusions and Future Work
Visualisation techniques .
First of all, the assumptions made in the paper "Visualizing and Understanding Atari Agents" [5], that guided backpropagation cannot be used for visualization techniques and the assertion made in the paper "Sanity Checks for Saliency Maps" [1], that guided backpropagation and guided Grad-Cam (at least in image processing) do not visualize the learned model but work similarly to an edge detector do not seem apply to deep reinforcement learning policies.
Visualization methods are very well suited as an additional, if not main debugging tool.
Especially
Guided Backpropagation was used during the development of Splitted Attention DDDQN with bidirectional LSTM agents and the A3C with LSTM agent to detect various errors in the neural network.
Not only can errors be detected but also an assignment could be made in which stream the error was located and which layer caused problems.
This kind of debugging is not possible with the usual reward evaluation.
Especially since the faulty neural network could achieve better results than the original one despite the errors it contains, these errors would not have been noticed.
Furthermore, it was also shown with two agents (Figure 68 and Figure 16) that Guided backpropagation can be used to detect if the agent has learned to avoid an area.
The agent obtained a high reward, but this is not the expected behavior.
The agent has learned to stay in the water at a certain height.
The agent always shoots to the left or to the right if something comes up there.
This is an important insight because it means that if the environment changes the agent will probably not be able to cope with these changes as well as an agent that has a lower score, but swims speciﬁcally towards the ﬁshes.
Here a simple reward function could even lead to misinterpretations.
Guided backpropagation could show the different strength of the gradients on the ﬁsh that swam at different heights, which could lead to the conclusion that the agent does not visit certain areas of the environment.
The Grad-Cam and especially the G1Grad-Cam method achieves very early results.
This is useful during the training process of the neural network.
For example, if an agent has to be trained fordays in a HPC cluster, G1Grad-Cam can quickly provide initial information after a few minutes / hours as to whether the agent recognizes any features at all or whether there are indications of errors in the neural network.
The rewards that are currently used as the main debugging tool are hardly usable at the beginning, especially with off-policy algorithms.
Because the agent is exploring at the beginning, the rewards are random, but the neural network learns to recognize the features even if it does not yet know what they mean or how the agent has to behave.
This is a big time advantage over the Reward function that took 2 days with the Split Attention Agent to show the ﬁrst signs that the Rewards are increasing.
The error does not necessarily have to be in the neural network, but also a faulty implementations can be detected in this way (during this work wrong stacking of the frames could be visualized on the neural network and the error could be corrected by the DDDQN agent).
The G1Grad-Cam and Grad-Cam method can save time when designing neural networks during debugging compared to a pure interpretation by the Reward function.
However, no evidence was found that using G1/Grad-Cam methods which are action discriminative methods, can visualize the difference in importance between the features, that has the biggest impact on the action (agent’s decision).
No feature was highlighted that is more important for the decision of an action.
Guided backpropagation was able to recognize most of the features.
All features detected by the
Grad-Cam methods and more.
Even though the gradients are sometimes hardly visible and only after long training the DDDQN agent could visualize itself in the game Breakout.
This method is especially suitable to evaluate the agent at the end (Does the agent recognize all important features?
Are there differences in intensity between the same features in different states?
How strong are the gradients on the features?) or for debugging purposes.
Guided backpropagation is better suited for debugging than the Grade-Cam method, because individual streams could be examined and individual layers.
In this way the error can be identiﬁed more quickly under certain circumstances (as in the development of the Split Attention Agent).
Negative gradients.
A hypothesis about possible explanation for the importance of negative gradients that forms around features were put forward in this work.
The explanation was compared to a
Laplacian ﬁlter which revealed the edges.
However, edges were not revealed here, but entire features it was concluded that through backpropagation a kind of Laplacian kernel feature detector was visualized.
Where the negative gradients would be produced by a second derivative to better detect the features.
Future work .
After having shown in this work that visualization techniques are a basic important tool for debugging neural networks, there are still some important questions left.
First of all the features that are important for the agent could be visualized.
What could not be shown in this paper is that in action discriminative methods, the features that have a greater importance for the selected action are more strongly emphasized.
A possible reason for this could be that by averaging all feature maps of the Grad-Cam methods the differences between the normal features and the features that directly inﬂuenced the agent’s decision were very small.
Here, a special implementation of
Guided backpropagation might help to better identify the differences, by reprogramming the Guided backpropagation to an action discriminative algorithm.
This would give a better understanding of the advantages shown in this work, that guided backpropagation can better evaluate the intensity of how well a neural network recognizes the feature.
In combination with a modiﬁed action discriminative
Guided backpropagation algorithm, it might be possible to get more meaningful results if it is possible to visualize the feature that had the greatest inﬂuence on the current action.
A Detailed List of Experiments
[Table]
In addition, there are the different settings of the visualization techniques, e.g. which layer or stream was visualized with gradient methods or up to which convolutional layers were the Grad-Cam method applied:
[Table]
The past frames in the input of the neural network for the guided backpropagation algorithm are also examined:
[Table]
An other question also arises: When does the neural network show ﬁrst signs of feature recognition?
Is it possible to ﬁnd out faster if the neural network or the agent in general has been programmed correctly?
Than about the interpretation of the reward function, which is only possible after a long time because of the exploration of the agent in off-policy algorithms
[Table]
For this reason
[Table]
a neural network that was after only 325 episodes / 250 000 steps saved, is also tested (for comparison: the well trained agent had trained over 5300 episodes and about 13 000 000 steps):
[Table]
B Detailed Experimental Results
The visualized results are presented in red or green color.
Meanwhile red is used for negative gradients, green has been used for positive ones.
Since the Grad-Cam methods have a ReLu function, the results are always positive.
All Images are taken at the same time for better comparison.
B.1 DDDQN: Breakout-v
to the
Grad-Cam method we can see much more stable visualized features.
We can see clearly the old positions of the ball and when the agent moves the old position of the Agent because we have four frames as input.
This is different to the Gradient methods where we cant ovalep the positive and negative gradients, thats why we are using the gradient methods on the last frame.
B.2 DDDQN: Seaquest-v
Figure stream).
In this picture we can see that the NN envelops more gradients around ﬁsh C than on ﬁsh B or A which have hardly any gradients.
This is because the agent has learned that if it stays on the height between the red lines and only shoots to the left or right when something comes up it survives longer and pays no attention to the rest of the environment.
The agent has learned his behaviour by heart and would not be able to react as well as an agent who makes less points but reacts better to the environment.
If the environment were to change slightly (e.g. the ﬁsh would suddenly swim from the top right to the bottom left) the agent could hardly react to it because it ignores everything that is not in the area between the lines.
stream).
In the Value Stream there is a small difference to the advatage stream.
We see that the agent has no oxygen and has to appear.
The DDDQN agent has never learned how to surfaced in seaquest, however he realises that the oxygen bar is an important feature that is related to the end of the episode, but the agent has not learned how to do what.
And since the agent has learned to leave the area between the lines, he will not be able to learn how to do it.
[Figure]
The agent has learned to pay more attention to the collected divers in the Value
Stream.
Here he has collected all 6 divers, which means that when he shows up he will get extra reward.
Furthermore you can see that the ﬁsh in the value stream has more gradients than in the advantage stream
When we visualise the second layer with the very well trained NN we always see no connections that we can interpret but not as well as we saw with the ﬁrst layer not show well interpretable results the agent still plays randomly and does not know what these features mean, he begins to understand that they have an inﬂuence on the decisions of the agent
There is little difference between the gradient method and Guided backpropagation
Although the agent still not begins to understand that they have an inﬂu ence on the decisions of the agent.
There is little difference between the gradient method and Guided backpropagation
The Grad-Cam method gives clearly better and more interpretable results than the gradient methods.
The
G1Grad- method gives the best results we can on the ﬁsh and the agent and also on the number of lives of the agent see some highlights.
[Figure]
Gredients so small that you can’t see them without a strong multiplication factor.
[Figure]
Too small gradients to visualize them just like in the game Breakout-v0.
Here you can see very clearly that the agent has learned to turn and shoot only left and right and never leaves the red area.
You can see that the ﬁsh in this area are very strongly highlighted and also the agent itself.
Outside this area there are almost no gradients to be seen.
B.6 A3C with LSTM: Seaquest-v