Explanation Augmented Feedback in Human-in-the-Loop Reinforcement Learning
Abstract
Human-in-the-loop Reinforcement Learning (HRL) aims to integrate human guidance with Reinforcement Learning (RL) algorithms to improve sample efﬁciency and performance.
A common type of human guidance in HRL is binary evaluative “good" or “bad" feedback for queried states and actions.
However, this type of learning scheme suffers from the problems of weak supervision and poor efﬁciency in leveraging human feedback.
To address this, we present EXPAND (EXPlanation
AugmeNted feeDback) which provides a visual explanation in the form of saliency maps from humans in addition to the binary feedback.
EXPAND employs a state perturbation approach based on salient information in the state to augment the binary feedback.
We choose ﬁve tasks, namely Pixel-Taxi and four Atari games, to evaluate this approach.
We demonstrate the effectiveness of our method using two metrics: environment sample efﬁciency and human feedback sample efﬁciency.
We show that our method signiﬁcantly outperforms previous methods.
We also analyze the results qualitatively by visualizing the agent’s attention.
Finally, we present an ablation study to conﬁrm our hypothesis that augmenting binary feedback with state salient information results in a boost in performance.
Introduction
Deep Reinforcement Learning (DRL) algorithms have achieved many successes in solving problems with high-dimensional state and action spaces [29, 3].
However, the current state-of-the-art DRL is yet to outperform human experts in many challenging problems.
One factor that limits DRL’s performance in these problems is its sample (in)efﬁciency.
It is often impractical to collect millions of training samples as required by standard DRL algorithms.
One way to curb this problem is to leverage additional human guidance by following the general paradigm of Human-in-the-Loop Reinforcement
Learning (HRL) [53].
This often allows the agent to achieve better performance and higher sample efﬁciency.
One popular form of human guidance in HRL is the binary evaluative signal on actions [18], in which humans provide a “good" or “bad" judgment for an action.
This framework allows non-expert humans to provide feedback, but its sample efﬁciency could be further improved by asking humans to provide stronger guidance.
For example, the binary feedback does not tell the agent why it made a mistake.
If humans can explain the “why" behind the evaluative feedback, then it is possible to leverage the explanation to further help the agent.
Taking training an autonomous driving agent as a motivating example, humans can point out to the agent that the “STOP" sign is an essential signal for the right action “apply-brake."
One way to convey this information is through natural language, but language comprehension is a difﬁcult problem itself.
For this type of visuomotor decision tasks, a more straightforward form of human explanation could be saliency information , in which humans highlight the important (salient) regions of the visual environment state.
The saliency information will indicate which visual features matter the most for the decision in the current state.
Saliency maps are shown to be a powerful means of assessing the agent’s internal representations in the (visual) explainable RL research [45, 50, 40, 11, 30, 35, 13].
In this work, we extend this idea to use a saliency map as a communication channel between RL agents and humans.
Hence human trainers can inform the RL agents about which regions they should focus on to accomplish the given task.
Note that requiring human trainers to provide explanations on their evaluations does not necessarily require them to be more adept than in the case of providing only binary feedback.
Like prior approaches utilizing binary evaluations [18], we assume the human trainers have a high-level understanding of the task.
In our driving agent example, the human trainers may not know things like the optimal angle of the steering wheel; however, we can expect them to be able to tell whether an observed action is good or bad, and what visual objects matter for that decision.
In this work, we present EXPAND:
EXPlanation AugmeNted feeDback (the overall ﬂow can be found at Appendix Fig. 4).
EXPAND augments the conventional binary feedback with human explanations regarding features in the visual state that are relevant to the decision.
We show that EXPAND agents require fewer interactions with the environment ( environment sample efﬁciency ) and fewer human signals ( human feedback sample efﬁciency ) by leveraging saliency information.
To efﬁciently learn with saliency information, we propose a novel method that applies multiple perturbations to irrelevant regions, thereby supplementing each saliency feedback with a set of constructed perturbed states.
The idea is to differentiate between relevant and irrelevant regions of the state when updating the agent’s value functions: perturbations to the irrelevant regions should not signiﬁcantly change the value function because they do not matter for the current decision.
Different kinds of perturbation in the irrelevant regions can essentially provide more feedback samples and hence reduce the feedback sample complexity.
To our knowledge, this is the ﬁrst work that incorporates human explanatory information as saliency maps in the setting of learning from human evaluative feedback.
Related Work
Leveraging human guidance for RL has been extensively studied in the context of imitation learning [36, 15], learning from demonstration [37, 14], inverse reinforcement learning [32, 1], reward shaping [31], learning from human preference [8, 16], and learning from saliency information provided by human [51].
Surveys on these topics are provided by Zhang et al.
[53] and Wirth et al.
Compared to these approaches, learning from human evaluative feedback has the advantage of placing minimum demand on both the human’s expertise and the trainer’s ability to provide guidance (e.g. the requirements of complex and expensive equipment setup).
Representative works include the
TAMER framework [18, 46], and the COACH framework [28, 4].
The TAMER+RL framework extends TAMER by learning from both human evaluative feedback and environment reward signal [19, 20].
DQN-TAMER further augments TAMER+RL by utilizing deep neural networks to learn in high dimensional state space [2].
Several approaches have been proposed to increase the information gathered from human feedback, which takes into account the complexities in human feedbackproviding behavior.
Loftin et al.
speed up learning by adapting to different feedback-providing strategies [25, 26]; the Advice framework [12, 7] treats human feedback as direct policy labels and uses a probabilistic model to learn from inconsistent feedback.
Other works also consider the action execution speed [33], the conﬁdence in predicting human feedback [48], and different levels of satisfaction indicated by human voice [43].
Although these approaches better utilize human feedback with improved modeling of human behaviors, weak supervision is still a fundamental problem in the evaluative feedback approach.
Human explanatory information has also been explored previously.
The main challenge of using human explanation is to translate human’s high-level linguistic representations to agent-understandable representation.
As an early attempt, Thomaz et al.
allow humans to give anticipatory guidance rewards and point out the object tied to the reward [44].
However, they assume the availability of an object-oriented representation of the state.
Krening et al.
Yeh et al.
resort to human explanatory advice in natural language [22, 49].
Still, they assume a recognition model that can understand concepts in human explanation and recognize objects.
In this work, we bridge the vocabulary gap between the humans and the agent by taking human explanations in the form of saliency maps.
Other works use human gaze data as human saliency information, collected with sophisticated eye trackers, to help agents with decision making in an imitation learning setting [52, 54, 17].
However, we refrain from comparing with these works since they involve humans in an ofﬂine manner.
In contrast, this work is closer to Arakawa et al., Xiao et al., where human trainers are required to be more actively involved during training [2, 48].
Finally, the way we exploit human explanation via state perturbation can be viewed as a novel way of data augmentation.
Data augmentation techniques are widely used in computer vision tasks [24, 39] and have recently been applied to deep RL tasks [21, 23].
Problem Setup
We intend to verify the hypothesis that the use of state saliency information and binary evaluative feedback can boost an RL agent’s performance.
We have an agent M that interacts with an environment E through a sequence of actions, states, and rewards.
Following standard practice in deep
RL, k ( k = 4 ) preprocessed consecutive image observations are stacked together to form a state s t =
[ x t − ( k − 1) , ..., x t − 1 , x t ] .
At each time-step t , the agent can select an action from a set of all possible actions
A = { 1 , ..., K } for the state s t ∈ S .
Then the agent receives a reward r t ∈ R from the environment E .
This sequence of interaction ends when the agent achieves its goal or when the time-step limit is reached.
This formulation follows the standard Markov Decision Process framework in RL research [42].
The agent’s goal is to learn a policy function π , a mapping from a state to action, that maximizes the expected return.
Additionally, we assume a human trainer who provides binary evaluative feedback H =
( h 1 , h 2 , ...h n ) that conveys their assessment of the queried state-action pairs given the agent trajectory.
We deﬁne the feedback as h t =
( x ht , b ht , x t , a t , s t ) , where b ht ∈ {− 1 , 1 } is a binary “bad" or “good" feedback provided for action a t and x ht =
{ Box 1 , ..., Box m } is a saliency map for the image x t in state s t .
Box i is a tuple ( x, y, w, h ) for the top left Euclidean coordinates x and y , the width w , and the height h of the bounding box annotated on the observation image x t .
Method
In EXPAND, the agent learns simultaneously from both environment reward and human feedback.
To learn from environment reward, we use an off-policy RL algorithm, the Deep Q-Networks (DQN) [29].
To learn from human feedback, we ﬁrst interpret binary feedback as labels on the optimality of the performed actions, which is similar to the interpretation in previous works [12, 7].
Since we are using DQN as our policy network, modeling binary feedback as labels on action optimality allows us to directly link human feedback with the advantage value [5] of an action.
The way we learn from human evaluative feedback will be discussed in detail in Section 4.3.
To learn with explanation augmented feedback, we amplify the saliency feedback by perturbing irrelevant regions in the state and expect the Q-value approximator to be indifferent to these perturbations when computing the action values.
The ideas manifest as various loss terms to update the DQN network.
The traininteraction loop of EXPAND can be found in Appendix B. Within an episode, the agent interacts with the environment and stores its transition experiences.
Every few episodes, it collects human feedback queried on a trajectory sampled from the most recent policy.
The DQN weights are updated twice, with the usual DQN loss and then with the proposed feedback loss, in a single training step.
This section covers the proposed loss terms and how we obtain perturbed states for these losses.
4.1 Learning with Environment Reward
Following DQN [29], a set of tricks is applied to stabilize training, such as experience replay, reward clipping, and soft-target network updates.
The learned policy π is deﬁned as the actions that maximize the Q-values.
Since π is a deterministic policy here, according to the Bellman equation, the state value function can be deﬁned as V π ( s ) =
Q π ( s, π ( s )) .
To ensure sufﬁcient exploration, we also use (cid:15) -greedy action selection mechanism, in which the probability (cid:15) of taking a random action is annealed down episodically.
The loss function in DQN that updates the Q-value function weights θ is:
[Equation]
(1) where ¯ θ is the target network parameters and D is the replay buffer that stores experience transition tuples ( s, a, r, s (cid:48) )
such that action a taken in state s brings immediate reward r and gets the agent to state s (cid:48) .
4.2 Learning with Binary Evaluative Feedback
Similar to DQN’s replay buffer, we maintain a feedback replay buffer D h , which stores the observed human feedback.
Feedback signals in D h are later sampled uniformly and are used to compute the feedback loss.
We use the advantage value to formulate our advantage loss .
Advantage value is the difference between the Q-value of the action upon which the feedback was given and the Q-value of the current optimal action calculated by the neural network.
Given the agent’s current policy π , state s , and action a for which the feedback is given, the advantage value is deﬁned as:
[Equation]
Hence, the advantage value quantiﬁes the possible (dis)advantage the agent would have if some other action were chosen instead of the current-best.
It can be viewed as the agent’s judgment on the optimality of an action.
Positive feedback means the human trainer expects the advantage value of the annotated action to be zero.
Therefore, we deﬁne a loss function, i.e., the advantage loss , which forces the network to have the same judgment on the optimality of action as the human trainer.
Intuitively, we penalize the policy-approximator when a marked “good" action is not chosen as the best action, or when a marked "bad" action is chosen as the best action.
For a feedback h =
( x h , b h , x, a, s ) , when the label is “good", i.e., b h = 1 , we expect the network to output a target value
ˆ A ( s, a ) = 0 , so the loss can be deﬁned as | ˆ A ( s, a ) − A π ( s, a ) | =
Q π ( s, π ( s )) −
Q π ( s, a ) .
When the label is “bad", i.e., b h = − 1 , we expect the network to output an advantage value
A π ( s, a ) < 0 .
Since here we do not have a speciﬁc target value for A π ( s, a ) , we resort to the idea of large margin classiﬁcation loss [34, 14], which forces Q π ( s, a ) to be at least a margin l m lower than the Q-value of the second best action, i.e., max a (cid:48) (cid:54) =
a Q π ( s, a (cid:48) ) .
One advantage of such interpretation of human feedback is that it directly modiﬁes the Q-values with the feedback information and does not require additional parameters to model human feedback.
Formally, for human feedback h
=
( x h , b h , x, a, s ) and the corresponding advantage value A s,a =
A π ( s, a ) , the advantage loss is:
[Equation]
Note that our method is different from the COACH framework [28], another method that interprets human feedback in terms of the advantage function.
In their formulation, the advantage value is exactly the advantage term in the policy gradient equation–human trainers are supposed to provide positive/negative policy-dependent feedback only when the agent performs an action better/worse than that in current policy.
Thus, such a formula is restricted to on-policy policy-gradient methods.
On the contrary, the advantage function in our formula aims to capture the relative utility of all the actions.
Here human feedback is direct policy advice, like that in the Advice framework [12], indicating whether an action is preferable regardless of the agent’s current policy.
This property makes our formula a better ﬁt for off-policy value-based methods like Q-learning.
4.3 Learning with Human Saliency Information
State saliency informs the agent about which parts of the state matter for achieving the goal.
These “parts" of the state could be speciﬁc regions or objects.
The intuition is that the agent must correctly identify the relevant visual features before it can make an optimal decision.
Hence if the agent knows where the relevant regions are, it can ﬁgure out the optimal actions faster.
In this work, each saliency map consists of a set of bounding boxes over the images, marking a region’s importance in achieving the given task.
We utilize this saliency information in a manner that directly affects the weights of the neural network.
It should be noted that the previous section only made use of the binary evaluations via the advantage loss .
In this section, however, we utilize the saliency feedback along with the binary feedback.
As a data augmentation technique, we ﬁrst describe how multiple perturbations are applied to an image.
Then we introduce the loss terms that EXPAND uses.
We propose two additional loss terms, namely the policy invariant loss and the value invariant loss that make use of human explanation along with binary evaluation.
Data augmentation through perturbation: We here propose a novel data augmentation technique, in order to exploit extra information from the human saliency map to further improve sample efﬁciency.
The key idea here is that, ideally, states with perturbation in irrelevant regions should have the same Q-values as the original states.
Hence the manipulations to irrelevant regions should not affect the agent’s policy.
A standard way of performing such perturbation is to apply Gaussian perturbations over irrelevant regions [11], essentially blurring them out and motivating the agent to focus more on the clear relevant regions.
Consider a feedback h
= ( x h , b
h , x, a, s ) .
We need to convert state s into some states s rt with perturbations on relevant regions and some other states ˜ s rt with perturbations on irrelevant regions.
Let M ( x, i, j ) denote a mask over relevant regions, where x ( i, j ) denotes the pixel at index ( i, j ) for image x .
We then have:
[Equation]
lies
[Equation]
Then we can perturb pixel ( i, j ) in image observation x according to mask M using a function φ deﬁned as above, where (cid:12) is the Hadamard Product and function G ( x, σ G ) is the Gaussian blur of the observation x .
Hence, we can get an image with perturbed relevant regions ( x r ) with mask M and perturbed irrelevant regions ( ˜ x r ) with mask ¬ M .
Policy invariant loss: The intuition behind the policy invariant loss is that under a set of perturbations over irrelevant regions in human explanation, the action marked “good" is always good and the action marked “bad" is always bad.
This means the agent’s judgment on the optimality of action should not change under perturbations over irrelevant regions in the human saliency map.
Thus, this loss is essentially the advantage loss calculated over states with perturbed irrelevant regions.
As deﬁned in the previous section, we use Gaussian perturbations of various ﬁlter sizes and variances (see Appendix D for detailed settings) to obtain a number of states with perturbed irrelevant regions.
The g th perturbation is denoted by ˜ s rg with the feedback
h
=
( x h , b h , x, a, s ) .
Formally, if we have g number of perturbations for a single state s in feedback h , the policy invariant loss is deﬁned as:
[Equation]
Value invariant loss: In previous deﬁnitions of the loss function, we utilized the difference in judgments on the optimality of actions between the agent and the humans.
We also note that, for humans, these states with perturbations on irrelevant regions are not “seen" differently from the original state.
Therefore, the Q-values of these perturbed states should be similar to those of the original state.
Hence, we deﬁne a mean squared error loss term over the two Q-values as our value invariant loss :
[Equation]
where A is the action set and g is the number of perturbations.
Combining Feedback Losses: We linearly combine the three losses to obtain the overall feedback loss:
[Equation]
where λ A , λ P and λ V are the weights of advantage loss , policy invariant loss and value invariant loss respectively.
The agent is trained with L DQN as well as L F .
As an extreme case, the value invariant loss can go down to zero when the policy network assigns an identical Q-value to actions in all the states, which will hurt the performance.
To prevent this we give a smaller weight to value invariant loss: λ V = 0 .
1 , λ P = 1 . 0 , λ
A = 1 . 0 .
We note that these losses improve human feedback sample efﬁciency and environment sample efﬁciency in different ways: value invariant loss imposes a stricter latent space similarity, whereas policy invariant loss helps differentiate between different actions.
Experimental Evaluation
To evaluate EXPAND, we conducted experiments on ﬁve tasks:
Pixel Taxi and four Atari games.
This section introduces the tasks and different metrics we use to compare EXPAND with other baselines.
Our baselines include Prioritized DQN [38] and an HRL algorithm DQN-TAMER [2] that simultaneously learns from environment reward and human binary feedback.
We also compare to three ablated versions of EXPAND with one loss term at a time: EXPAND-Advantage ( λ A =1.0, λ P =0, λ V =0), EXPAND-Value-Invariant ( λ A =1.0, λ P =0, λ V =0.1) and EXPAND-Policy-Invariant ( λ A =1.0, λ P =1.0, λ V =0).
The goal of experimenting with different variants of EXPAND is to determine the effect of each loss term on the learning performance.
Hence, this serves as an ablation study.
In all our experiments, our algorithm and the baselines employ the same DQN network architecture adopted by [29], which has three convolutional layers following by one fully-connected layer.
The same set of hyperparameters is used to train the models.
Details on the architecture and hyperparameters can be found in Appendix A.
We use multi-step returns and the prioritized experience replay mechanism [38] both in EXPAND and in the baselines.
Following the standard preprocessing procedure, each frame is converted from RGB format to grayscale and is resized to 84 × 84 .
The input to the networks is normalized to the range of [0, 1].
During training, we start with an (cid:15) -greedy policy ( (cid:15) = 1 . 0 ) and reduce (cid:15) by a factor of λ (cid:15) at the end of each episode until it reaches 0.01.
The reported results are averaged over at least 3 random seeds.
Inspired by the Taxi domain that is widely used in RL research [10], we design a task named Pixel
Taxi (see Appendix Fig. 4).
This is a grid-world setup where the agent, as a taxi, occupies one grid cell at a time, and passengers denoted by different colored dots occupy separate grid cells in the world.
Our taxi agent’s goal is to pick up the correct passenger and reach the destination cell.
For the agent to learn the target passenger instead of simply remembering the “locations", we randomize the passengers’ positions at the beginning of each episode.
A reward in this task is given only when the taxi drops off the correct passenger at the destination cell.
In addition, we also choose four
Atari games [6] with default settings: Pong , Asterix , Enduro , and Montezuma’s Revenge .
Original
Enduro and Montezuma’s Revenge can be inﬁnitely long and can make human training impractical.
Therefore, in Enduro , our goal is to teach the agent to overtake as many cars as possible withinenvironment steps (an episode); hence we denote this task as Enduro-1000 .
In Montezuma’s Revenge , we train the agent to solve the ﬁrst room within 1000 environment steps per episode; hence we denote this task as MR-1 .
We evaluate our work on two fronts.
The principal role of humans in the RL process is to improve the environment sample efﬁciency; hence this serves as our primary indicator of success.
Also, one of the major claims of this work is that augmenting binary feedback with human explanation in the form of saliency maps can improve human feedback sample efﬁciency; hence this is our second metric.
5.1 Obtaining Feedback
The steps for obtaining human feedback are described as follows (see Appendix B): in every N f ( N f = 4 ) episodes, we sample one trajectory and query the user for binary evaluative feedback as well as saliency information.
Although actively querying humans could be preferable, the goal of this work is to demonstrate how human explanation can augment binary feedback.
Following the setting of our baseline DQN-TAMER, we allow humans to “watch" the queried trajectories and provide feedback at will.
We use explanation information from human trainers for Asterix , Enduro , and
Montezuma’s Revenge .
For Pixel Taxi and Pong , we use a synthetic oracle, since oracles for these two tasks are relatively easy to construct.
In Asterix , Enduro , and MR-1 , we evaluated EXPAND and DQN-TAMER with human trainers.
The human interface we used to collect human feedback and explanation is similar to a video player (see
Appendix G), with which the trainers can start/pause the replay of the queried trajectory and provide explanations by directly drawing bounding boxes on the screen.
The trainers can also adjust the “video" frame rate based on their needs.
Similar to Deep-TAMER [46], we also apply each received feedback and explanation to frames that are displayed between 2 and 0.2 seconds before the feedback occurred – we assume that within this 1.8-second window, the salient regions/objects should be the same.
Thus, we can use an off-the-shelf object tracking method, like the CSRT tracker [27], to locate the highlighted regions/objects in previous frames.
Our experiments show this temporal consistency assumption works well in practice.
In addition to backward tracking, we also keep track of the highlighted regions/objects and perform object detection (e.g., automatically detect and annotate the cars ahead in Enduro ) in succeeding frames.
The tracking and detection tricks greatly reduce the number of times human trainers need to annotate the images, and in our experiments, we found that human trainers could provide their feedback and explanation with ease.
To perform a systematic analysis and get additional insights, we conduct a larger set of experiments that include 5 runs of all algorithms, the ablation study, and runs with different numbers of perturbations on Pixel Taxi and Pong with a synthetic oracle.
A synthetic oracle allows us to run a large number of experiments and give consistent feedback across different runs, providing fair and systematic comparisons between different approaches [12, 2].
In Atari Pong , we use a trained model from the Atari Zoo [41]; in Pixel Taxi , we use a self-trained DQN model.
To annotate “relevant" regions on the image observation, we use hard-coded models to highlight the taxi-cell, the destination-cell, and the target-passenger in Taxi , as well as the two paddles and the ball in Atari-Pong .
5.2 Results
Sample efﬁciency : Fig.
1 compares the environment sample efﬁciency as well as performance between EXPAND (in red), DQN-TAMER (in blue), and Prioritized DQN (in green) in all tasks.
We restrict the human experiments– Enduro-1000 , Asterix , and Montezuma’s Revenge MR-1 –to 100k environment samples, which corresponds to around 10k human feedback and approximately one-hour interaction time with a human trainer.
In Enduro-1000 and Asterix , EXPAND achieves a score that is over 30% higher than DQN-TAMER at the end of the training.
In MR-1 , which is a hard-exploration domain with sparse reward, we counted the number of times that the learning agent solved MR-1 .
Without sophisticated exploration strategies, EXPAND managed to solve MR-1 15 times on average, which is twice as many as DQN-TAMER.
As expected, Prioritized DQN could not solve the task even a single time within the 100k env.
steps for MR-1 and was worse in all other tasks.
This result demonstrates that EXPAND performs better than our baselines with limited environment samples.
In Pixel Taxi and Pong , we do not restrict the number of feedbacks from the oracle.
EXPAND learns a near-optimal policy in just 130k and 140k environment samples, whereas DQN-TAMER takes over 200k and 250k samples to reach a similar performance.
This is an over 35% improvement in environment sample efﬁciency.
Regarding human feedback efﬁciency, in Pixel Taxi and Pong ,
EXPAND requires about 30k and 35k binary and saliency feedback pairs, respectively, from the oracle to learn a near-optimal policy.
In contrast, DQN-TAMER requires more than 50k feedbacks in both tasks.
As seen from our study, we note that the marginal cost of asking for saliency information is low (owing to off-the-shelf object detection and tracking), making EXPAND a promising method on the front of human feedback sample efﬁciency.
Finally, the performance of Prioritized-DQN was poor in both tasks.
Ablation study : Fig.
2 shows that all four variants of EXPAND (EXPAND, and EXPAND with individual loss terms) eventually converged to near-optimal policies.
We note that EXPAND-PolicyInvariant (in purple) and EXPAND-Value-Invariant (in light green) perform signiﬁcantly better than
EXPAND-Advantage, highlighting that each saliency loss alone provides signiﬁcant improvements over the baseline.
Finally, combining these losses (EXPAND) boosts this performance even further, indicating that collecting saliency information in addition to binary evaluations is a better approach.
Varying number of perturbations : We experimented with the number of perturbations required in each state to get the best performance.
Figure 2 shows a comparison when the number of perturbations is varied among { 1 , 5 , 12 } for Pixel Taxi and Pong using a synthetic oracle.
The plots suggest that increasing perturbations only evoke slight performance gains, and therefore setting the number of perturbations to 5 for EXPAND is apt.
Visualizing agent attention : To verify that the agents have successfully learned to pay attention to the critical regions highlighted by human saliency maps, we visualize the attention maps of trained EXPAND agents with a method commonly used to provide visual interpretations of deep RL agents [11].
The algorithm takes an input image I and applies a Gaussian ﬁlter to a pixel location ( i, j ) to blur the image.
This manipulation adds spatial uncertainty to the surrounding region and produces a perturbed image Φ( I, i, j ) .
A saliency score for this pixel ( i, j ) can be deﬁned as how much the blurred image changes the policy π [11].
The results for Pong can be found in Fig.
3, where EXPAND learns to focus on the important objects (the ball and the opponent) as early as 20k environment steps.
This result provides another insight on why EXPAND performs better as object representation is a critical cognitive capacity lacking from current deep RL agents [9].
Conclusion & Future Work
In this work, we presented a novel method to integrate human explanation–as saliency maps on image observations–with their binary evaluations of agents’ actions in a Human-in-the-Loop RL paradigm.
We show that our proposed method, EXPAND, outperforms previous methods in environment sample efﬁciency, and shows promising results for human feedback efﬁciency.
We also see that supplementing human saliency feedback with perturbed irrelevant regions is helpful when multiple such perturbations are used.
Our results show that leveraging information about relevant parts of the image, for the task at hand, indeed helps the agent perform better.
We currently treat saliency feedback as human advice; however, the advice in the form of natural language interaction would be an improvement.
Moreover, we note that we have restricted “perturbations" to be Gaussian blurs to the state image.
Future work can experiment with different types of perturbations and even domain-dependent perturbations that involve object manipulation.
Appendix A. Hyperparameters
• Convolutional channels per layer: [32, 64, 64] • Convolutional kernel sizes per layer: [8, 4, 3] • Convolutional strides per layer: [4, 2, 1] • Convolutional padding per layer: [0, 0, 0] • Fully connected layer hidden units: [512, number of actions] • Update interval:• Discount factor: 0.
• Replay buffer size: 50, • Batch size:• Feedback buffer size: 50,000 in Atari games, 10,000 in Pixel Taxi• Feedback batch size: 64 in Atari games, 32 in Pixel Taxi • Learning Rate: 0.
• Optimizer: Adam • Prioritized replay exponent α = 0 .•
Prioritized replay importance sampling exponent β = 0 .•
Advantage loss margin l m = 0 .•
Rewards: clip to 1] • Multi-step returns: n =• (cid:15) episodic decay factor λ (cid:15) : 0.99 in Pixel Taxi , 0.9 in Atari games
B. Train - Interaction Loop
Algorithm 1 Train - Interaction Loop
Result:
Trained DQN agent M
Input: DQN agent M with randomly-initialized weights θ , replay buffer D , human feedback buffer D h , feedback frequency N f , total episodes N e , maximum number of environment steps per episode T , update interval b
Beginfor i = 1 to N e do for t = 1 to T do
Observe state s
Sample action from current DQN policy π with (cid:15) -greedy, observe reward r and next state s (cid:48) and store (s, a, r, s (cid:48) ) in D if t mod b == 0 then
Sample a mini-batch of transitions from D with prioritization
Compute the DQN loss L DQN and update θ over L DQN
Sample a mini-batch of human feedback from D h
Compute the feedback loss L F and update θ over L F end if end for if i mod N f == 0 then
Obtain the last trajectory τ from D
Query τ to obtain feedback H
i
Append
C. Overall Flow of EXPAND
Figure 4 shows the the overall ﬂow of EXPAND.
D. Settings of Gaussian Perturbation
• 1 Perturbation: ◦ ﬁlter size: 5, σ :• 5 Perturbations: ◦ ﬁlter size: 5, σ :◦ ﬁlter size: 5, σ :◦ ﬁlter size: 5, σ :◦ ﬁlter size: 11, σ :◦ ﬁlter size: 11, σ :• 12 Perturbations: ◦ ﬁlter size: 5, σ :◦ ﬁlter size: 5, σ :◦ ﬁlter size: 5, σ :◦ ﬁlter size: 7, σ :◦ ﬁlter size: 7, σ :◦ ﬁlter size: 7, σ :◦ ﬁlter size: 9, σ :◦ ﬁlter size: 9, σ :◦ ﬁlter size: 9, σ :◦ ﬁlter size: 11, σ :◦ ﬁlter size: 11, σ :◦ ﬁlter size: 11, σ :
E. Examples of Perturbation
Fig.
5 shows examples of different perturbations on irrelevant regions in image observations.
F. Additional Saliency Maps
Fig 6 shows how EXPAND gradually learns to attend to regions deemed relevant by the human during training.
G. Human Interface
Fig 7 shows the web-interface used to conduct the human experiment.
Before the experiment began, the trainers were briefed about the usage of this interface.
The trainers were asked to provide their consent to use any information they provide for any analysis and experiments.
The feedback interface,
Fig 7, is divided into two columns, the information pane (left) and the control pane.
(right).
On the information pane, users were able to adjust the frame rate at which they would want to view the “video" made using the state transitions.
Users could also view bounding-box related information, such as which ones were the drawn boxes and the boxes suggested by the implemented object tracker.
On the right pane, users could view the frame upon which they would give their saliency feedback.
Users could see the agent’s current action in the information pane.
For their ease, we provided controls like Pause , Play , skip to Next Frame, go to Previous Frame, Clear all bounding boxes on the frame, Save feedback and ﬁnally Finish the current feedback session.
To provide a binary feedback on agents’ actions, users were required to press keyboard keys “A" for a good-feedback, “S" for a bad-feedback, and “D" for no feedback.
They could provide saliency explanations via clicking on at the required position on the image frame and dragging to create a rectangular selection.