Attention Option-Critic
Abstract
Temporal abstraction in reinforcement learning is the ability of an agent to learn and use highlevel behaviors, called options.
The option-critic architecture provides a gradient-based end-to-end learning method to construct options.
We propose an attention-based extension to this framework, which enables the agent to learn to focus different options on different aspects of the observation space.
We show that this leads to behaviorally diverse options which are also capable of state abstraction, and prevents the degeneracy problems of option domination and frequent option switching that occur in option-critic, while achieving a similar sample complexity.
We also demonstrate the more interpretable and reusable nature of the learned options in comparison with option-critic through different transfer settings.
Experimental results in a relatively simple four-rooms environment and the more complex ALE (Arcade Learning Environment) showcase the efﬁcacy of our approach.
1. Introduction
Humans are effortlessly adept at many forms of abstraction.
We plan and perform actions at a high-level of decision making, and not at the level of individual muscle movements.
Such high-level actions typically last for an extended period of time.
This is known as temporal abstraction.
When observing our surroundings before making a decision, we rely and focus on only the important aspects of our sensory input, and ignore the unnecessary signals.
This is called state abstraction.
Within the options framework (Sutton et al., 1999; Precup, 2000) the end-to-end learning of hierarchical behaviors has recently become possible via the option-critic architecture (Bacon et al., 2017), which enables the learning of intra- option policies, the termination functions and the policy over options, to maximize the expected return.
However, if this is the sole objective for option discovery, the beneﬁt that options have over primitive action policies is questionable.
Indeed, the option-critic architecture eventually results in option degeneracy i.e. either one option dominates and is the only one that is used, or there is frequent termination of and switching between options.
Introduced to combat this problem, the deliberation cost model (Harb et al., 2018) modiﬁes the termination gradient to assign a penalty to option termination.
This leads to extended options, but is susceptive to a hard-to-interpret cost parameter.
Alternatively, the termination critic (Harutyunyan et al., 2019) employs a predictability objective for option termination to prevent option collapse and improve planning.
We adopt the view that options should be diverse in their behavior by explicitly learning to attend to different parts of the observation.
In doing so, we solve the degeneracy problem by ensuring that options are only used when their respective attentions are activated.
This lends credibility to the notion of options specializing to achieve speciﬁc behaviors.
For example, in the four-rooms environment (Sutton et al., 1999), it makes little sense to use the complete observation when deciding how to move out of a particular room.
Current option discovery methods in the function approximation setting do just this.
Our approach also, in effect, relaxes the strong assumption – made by many option discovery methods – that all options are available everywhere, and acts as a proxy towards learning the initiation sets for options (Sutton et al., 1999), which are otherwise inconvenient to directly learn using a gradient-based learning approach.
The view of bounded rationality (Simon, 1957) can be seen as one of the motivations for temporal abstraction.
The added capability of state abstraction takes this one step further, and serves as an additional rationale for our work.
2. Background
A discrete-time ﬁnite discounted MDP (Markov Decision
Process) M (Puterman, 1995; Sutton and Barto, 1998) is characterized by the tuple {S , A , R, P, γ } , where S is the set of states, A is the set of actions, R : S ×
A → R is the reward function, P : S ×
A × S → [0 , 1] is the transition probability function which speciﬁes the dynamics of the environment, and γ ∈ [0 , 1) is the scalar discount factor.
A Markovian stationary policy π : S ×
A → [0 , 1] is a probabilistic mapping from the set of states S to the set of actions A .
At each timestep t , the agent observes state s t ∈
S and takes an action a t ∈ A according to policy π , thereby receiving reward r t +1 =
R ( s t , a t ) and transitioning to state s t +1 ∈ S with probability P ( s t +1 | s t , a t ) .
For policy π , the discounted state value function is given by: V π ( s ) =
E π [ (cid:80) ∞ t =0 γ t r t +1 | s
=
s ] and the discounted action value function by: Q π ( s, a ) =
E π [ (cid:80) ∞ t =
γ t r t +1 | s 0 =
s, a 0 = a ] .
For a parameterized policy π θ with J ( θ ) =
V π ( s 0 ) as the objective function, the policy gradient theorem (Sutton et al., 2000) can be used to learn the optimal policy π ∗ θ that maximizes V π ( s 0 ) as:
[Equation]
where d π ( s | s 0 )
= (cid:80) ∞ t =
γ t P ( s t = s | s 0 , π ) is the discounted weighting of states with s 0 as the starting state.
In actor-critic methods (Konda and Tsitsiklis, 2000), the action values Q πφ ( s, a ) , parameterized by φ , are typically estimated by using temporal difference (TD) learning (Sutton, 1988).
For instance, the update rule for 1-step TD(0) follows as: φ = φ + αδ t ∇ φ Q πφ ( s t , a t ) where the TD(0) error δ t =
r t +1 + γQ πφ ( s t +1 , a t +1 ) − Q πφ ( s t , a t ) and α is the learning rate.
2.1. The Options Framework
A Markovian option ω ∈ Ω (Sutton et al., 1999) is a tuple that consists of an initiation set I ω ⊆ S , which denotes the permissible set of states where the option can be initiated, an intra-option policy π ω :
S ×
A → [0 , 1] , which speciﬁes a probabilistic mapping from states to actions, and a termination condition β ω : S → [0 , 1] , which signiﬁes the probability of option termination in a state.
Ω( s ) denotes the set of available options for state s and an option ω is available in state s if s ∈ I ω .
Ω is the union of all Ω( s ) , ∀ s ∈ S .
Similar to Bacon et al.
, we consider call-and-return option execution.
In this model, when the agent is in state s t , it chooses an option ω ∈ Ω( s t ) according to a policy over options π Ω .
The intra-option policy π ω is then followed until the current option terminates according to β ω after which a new option that is available at the new state is chosen by π Ω , and the process repeats.
Like many existing option discovery methods, we too make the assumption that all options are available everywhere, i.e., ∀ s ∈ S , ∀ ω ∈ Ω : s ∈ I ω .
However, we show that our approach relaxes this assumption, in effect, and provides an elegant way to learn distinct initiation sets for options.
The option-critic architecture (Bacon et al., 2017) provides an end-to-end gradient-based method to learn options.
For parameterized intra-option policies π ω,θ and option terminations β ω,ν , the option-value function is:
[Equation]
where Q U : S × Ω × A → R is the value of executing action a in the context of state-option ( s, ω ):
[Equation]
and U : Ω × S
→ R is the option-value on arrival (Sutton et al., 1999) and represents the value of executing option ω in state s (cid:48) :
[Equation]
where V Ω ( s ) = (cid:80) ω π Ω ( ω | s ) Q Ω ( s, ω ) is the option-level state value function.
The intra-option policies and option terminations can be learned by using the policy gradient theorem to maximize the expected discounted return (Bacon et al., 2017).
The gradient of this objective with respect to intra-option policy parameters θ when the initial condition is ( s 0 , ω 0 ) is:
[Equation]
where µ Ω ( s, ω | s 0 , ω 0 ) is the discounted weighting of stateoption pairs along trajectories that start with ( s 0 , ω 0 ) : µ [Equation] s, [Equation]
[Equation] ) =
[Equation]
Similarly, the gradient with respect to option termination parameters ν with initial condition ( s 1 , ω 0 ) is:
[Equation]
where A Ω ( s, ω )
=
Q Ω ( s, ω ) − V Ω ( s ) is the advantage of choosing option ω in state s .
2.2. Attention
The attention mechanism was ﬁrst proposed in language translation tasks (Bahdanau et al., 2015) but has since been applied in vision (Sorokin et al., 2015) and reinforcement learning (Mnih et al., 2014) as well.
It enables the localization of important information before making a prediction.
In our approach, soft attention (smoothly varying and differentiable) is applied as a learnable mask over the state observations.
3. Attention Option-Critic
We introduce the Attention Option-Critic (AOC) architecture to enable options to learn to be attentive to speciﬁc features in the observation space in order to diversify their behavior and prevent degeneracy.
An attention mechanism h ω,φ , parameterized by φ , is applied to the observation s for each option ω as: o ω = h ω,φ ( s ) (cid:12) s where (cid:12) denotes element-wise multiplication.
h ω,φ consists of values in [0 , 1] and is the same size as the original observation s .
The result o ω is used to determine the value of the option, the intraoption policy and the termination condition.
This is done for each option separately, and ensures only the required features from the observation determine the option’s behavior.
We refer to o as the list of all attention-modiﬁed observations for each option o = { o ω : ω ∈ Ω } .
The learning of the option terminations and intra-option policies is similar to the option-critic architecture.
The complete algorithm is shown in Algorithm 1.
The attention for each option is learned to maximize the expected cumulative return of the agent while simultaneously maximizing a distance measure between the attentions of the options, so that they are attentive to different features.
Additionally, some regularization is added to facilitate the emergence of desired option characteristics.
The attention parameters φ are updated with gradient ascent as φ =
φ
+ α φ ∇ φ (cid:2)
Q Ω ( o ω , ω ) + L (cid:3) , where L denotes the sum of the distance measure and the regularization, weighted by their respective importance.
More details are speciﬁed in the next section.
The attention mechanism brings an aspect of explainability to the agent, and allows one to easily understand each option’s focus and behavior.
Also, it provides a highly interpretable knob to tune options since the characteristics of the resulting options can be controlled by affecting how the attentions of the options are learned during training.
For example, constraining attentions to be distinct enables the diversity of options to be set explicitly as a learning objective.
Alternatively, penalizing differences in option attention values for states along a trajectory results in temporally extended options, which achieves an effect similar to the deliberation cost model (Harb et al., 2018), but in a more explainable way.
The resulting attention for each option also serves as an indication of the regions of state space where that option is active and can be initialized.
Thus, along with the intraoption policies and option terminations, AOC essentially learns the initiation sets of the options in that an option is typically only initiated in a particular state when the corresponding attention of that option in that state is high.
It is this result which prevents the options from degenerating.
Since every option cannot be executed or initiated everywhere, it prevents frequent option termination and switching, and also prevents option domination (Figure 6) by ensuring that a single option cannot always be followed.
3.1. Optimality of Learned Solution
Since each option receives different information, it is not immediately obvious whether the solution that is learned by AOC is ﬂat, hierarchically or recursively optimal (Dietterich, 2000).
However, each option learns to act optimally based on the information that it sees, and apart from some constraints enforced via option attentions, individual option optimality is driven through the policy over options to maximize the total expected return.
Since there is no pseudo reward or subtask assigned to each option, their attentions and areas of usage are learned to maximize this objective and we reason that in the absence of attention constraints, a ﬂat optimal policy will be learned in the limit.
In the presence of constraints, the optimality of the learned options will depend on the interactions between the multiple objectives.
Even in such cases, AOC is capable of achieving a ﬂat
[Figure]
optimal solution as shown empirically in the next section.
4. Experimental Results
In this section, we show the beneﬁt of attention and empirically demonstrate that it prevents option degeneracy, provides interpretability, and promotes reusability of options in transfer settings.
4.1. Learning in the four-rooms environment
We start off by showing the beneﬁt of attention in the fourrooms navigation task (Sutton et al., 1999) where the agent must reach a speciﬁed goal.
The observation space consists of one-hot-encoded vectors for every permissible state in the grid.
The available actions are up, down, left and right.
The chosen action is executed with probability 0.98 and a random action is executed with 0.02 probability.
The reward is +20 upon reaching the goal, and -1 otherwise.
The agent starts in a uniformly random state and the goal is set randomly for each run.
We use 4 options for learning, with a discount factor of 0.99.
The attention h ω,φ for each option ω is initialized randomly as a vector of the same length as the input observation s .
Thus, in this situation, the option attentions are independent of the state observation.
We employ a 2layer shared-parameter neural network to approximate the intra-option policy, the option termination functions, and the option values.
In our implementation of AOC (for all experiments), the network learns the option values Q Ω to which the (cid:15) -greedy strategy is applied to determine the pol-
[Figure]
icy over options π Ω .
Intra-option exploration is enforced with entropy regularization.
The architecture is shown in section C.
The option attentions, option values, intra-option policies and option terminations are learned in an end-to-end manner to maximize the total expected discounted return.
The cosine similarity between the option attentions is added to the loss to ensure that the learned attentions are diverse.
Furthermore, a regularization loss – the sum of absolute differences between attentions (for each option) of adjacent states in a trajectory – penalizes irregularities in the option attentions of states in close temporal proximity along trajectories.
This results in smooth attentions and leads to temporally extended options by minimizing switching, and achieves a similar effect to deliberation cost (Harb et al., 2018).
Thus, for the four-rooms domain, the L term in Algorithm 1 is enforced by adding w 1 L 1 + w 2 L 2 to the overall network loss function, where L 1 is the total sum of cosine similarities between the attentions of every pair of options, L 2 is the temporal regularization loss for option attentions, and w 1 and w 2 are the respective weights for these additional objectives.
We found that a value of 2.0 for both w 1 and wresulted in the most diverse options, judged quantitatively (see section A.5) and qualitatively (Figure 1).
Further details regarding hyperparameters and reproducibility are provided in section C.
The resulting option usage and their attentions are shown in Figure 1.
The learned options are distinct and specialized in their behavior, and they perform state abstraction by focussing on a subset of the observation to perform their speciﬁc tasks.
The option usage respects the corresponding area of attention, which indicates that the options are typically limited to this area and that their behavior can be reasonably interpreted from their attentions.
AOC also learns stable options and the behavior and usage of options does not vary signiﬁcantly during the course of training.
This is in contrast to option-critic (OC), which tends to learn degenerate options that are volatile and continuously change behavior.
A qualitative comparison that demonstrates AOC option stability is shown in section A.1.
Although AOC additionally needs to learn option attentions, it learns faster than OC, as shown in Figure 2a.
One possible reason could be that in AOC, options specialize to different regions and enable quicker learning because of less overlap between their usage.
Each learning curve is averaged over 15 independent runs, each with a random goal location.
A comparison between option domination in AOC and OC (see A.2) during training indicates that the latter prevents it.
4.2. Transfer in the four-rooms environment
We perform two experiments to assess the transfer capability of AOC in the four-rooms domain, both after 30, episodes of training.
The ﬁrst is goal transfer, where the location of the goal is changed to a new random location and the second is blocked hallway, where the goal is the same but a random hallway is blocked.
AOC transfer I and transfer II respectively represent the scenarios where the weights w 1 and w 2 are kept unchanged or are set to 0 to give priority to option learning over attention regularization, before learning in the new task.
From Figures 2b and 2c, it can be seen that in spite of the option volatility that aids OC transfer, AOC transfer II performs similarly in the goal transfer setting and both variants of AOC show superior initial performance in the blocked hallway setting with transfer II being faster overall.
The speed of AOC transfer
II is even more apparent when the agent needs to go all the way around the blocked hallway (see section A.3).
The slower transfer of AOC transfer I can be explained by the over-preference towards optimizing attention characteristics which AOC transfer II mitigates.
Each curve is averaged over 15 independent runs with different blocked hallways and different goals before and after transfer.
From another perspective, upon transfer, option-critic completely relearns the options.
Figure 3 shows a speciﬁc instance of transfer.
Comparing Figure 1a with Figures 3a and 3b shows that there is little similarity between the option behavior before and after transfer with OC.
We argue that for options to be beneﬁcial for generalization and lifelong learning, they should exhibit similar behavior upon transfer, and only change as required, so that previously learned behaviors can be leveraged, and so that options can be efﬁciently composed into even higher levels of behavior.
AOC exhibits this quality.
A comparison of Figures 1b and 1c with Figures 3c to 3f shows that option attentions remain ﬁxed indicating that each option remains in its assigned space, and that the option behavior remains relatively consistent upon transfer.
4.3. Arcade Learning Environment
We now demonstrate the performance of AOC in the Arcade
Learning Environment (Bellemare et al., 2013).
We useoptions with a discount factor of 0.99.
The input observation s is a stack of 4 frames.
The option attentions h ω,φ are state dependent and are learned with a convolutional neural network.
Each option’s attention has the same dimensions as a single frame, and is shared across all frames in the input stack.
We refer to this as the shared-attention model.
The option policies, values and terminations are learned with a shared-parameter deep neural network, similar to option-critic.
The architecture is shown in section C.
Apart from maximizing the total expected return, the attentions are constrained to exhibit some desired characteristics.
Attention diversity is enforced by maximizing the L1 norm between the object attentions of the options and attention sparsity is incentivized by penalizing non-zero attentions for
[Figure]
the background.
Lastly, attention regularity is promoted between object pixels by penalizing frequent changes in their attention values.
The objects and background are identiﬁed by ﬁnding the connected components in the observation (Figure 4b).
Thus, for the atari domain, the L term in Algorithm 1 is enforced by adding w 1 L 1 + w 2 L 2 + w 3 L 3 + w 4 Lto the network loss function, where L 1 , L 2 , L 3 are the losses for attention diversity, sparsity and regularity respectively.
The additional regularizer L 4 is added to prevent an option’s attention from collapsing to zeros.
w 1 , w 2 , w 3 and wrepresent their respective weights.
More details regarding hyperparameters are provided in section C.
For training in the Asterix environment, we found that the values 5000, 0.01, 100, and 1 for the weights w 1 , w 2 , wand w 4 respectively, resulted in diverse attentions and good performance.
Figure 4 shows the performance and learned option attentions.
Figure 4a shows that AOC achieves a similar sample complexity compared to OC, despite also having to learn the state-dependent attention mechanism.
We reason that learning the attentions enable options to specialize early on in the training process, and hence speed up training, despite having more parameters to learn.
Each
[Figure]
curve is averaged over 3 different random seeds.
Figures 4c and 4d show the resulting option attentions and indicate that option 0 and option 1 have respectively specialized to behaviors pertaining to the main sprite’s position in the upper or lower half of the frame.
Thus, AOC allows for learning diverse and interpretable options in complex environments too.
Additional atari results are shown in section B.
5. Related work
There has been extensive research on the beneﬁt of temporal abstraction for reinforcement learning (Parr and Russell, 1998; Dayan and Hinton, 1993; Dietterich, 2000; McGovern and Barto, 2001; Stolle and Precup, 2002; Mann and
Mannor, 2014).
Speciﬁc to the options framework (Sutton et al., 1999; Bacon et al., 2017), there have been many recent approaches to incentivize learned options to be diverse (Eysenbach et al., 2018), temporally extended (Harb et al., 2018), more abstract (Riemer et al., 2018), and easy to plan (Harutyunyan et al., 2019) and explore (Jinnai et al., 2019) with.
The interest option-critic method (Khetarpal et al., 2020) provides a gradient-based approach towards learning where to initialize options by modeling the initiation sets as differentiable interest functions.
However, the initialization of the interest functions is biased towards all options being available everywhere.
In contrast, our AOC approach is completely end-to-end and does not require any special initializations, and in effect, is able to learn distinct areas where options can be initialized and remain active.
Deep skill chaining (Bagaria and Konidaris, 2020) is another approach that relaxes the assumption of universal option use.
This method learns a chain of options by backtracking from the goal and ensuring that the learned initiation set of one option overlaps with the termination of the preceding option.
Although each option performs state abstraction, the resulting options are highly dependent on the given task and must be relearned upon transfer.
Furthermore, results were mostly conﬁned to navigation-based tasks.
The MAXQ (Dietterich, 2000) approach towards hierarchical reinforcement learning decomposes the value function of the target MDP into value functions of smaller MDPs.
Although this decomposition creates an opportunity to perform state abstraction, the overall approach is based on the heavy assumption that the subgoals, and the subtasks necessary to achieve them, are speciﬁed beforehand.
6. Conclusion
To the best of our knowledge, our method is the ﬁrst to combine temporal and state abstraction in a ﬂexible end-to-end gradient based approach and results in learned options that are diverse, stable, interpretable, reusable and transferable.
We demonstrate that the addition of an attention mechanism prevents option degeneracy, a major long standing problem in option discovery, and also relaxes the assumption of universal option availability.
It also provides a highly intuitive method to control the characteristics of the learned options.
From the lifelong learning perspective, an interesting future direction is to meta-learn the attentions and options across a range of tasks from the same environment.
This could lead to faster transfer, while keeping the existing beneﬁts of our approach.
From the view of model-based reinforcement learning, predictive approaches with option attentions could allow for efﬁcient long-horizon planning by predicting option activation through predicted attentions.
Lastly, the approach we have presented is versatile and can be applied to many existing option discovery methods.
We leave such avenues of possible combination as future work.
Appendix A. Other four-rooms experiments A.1. Comparison of option stability between AOC and OC
[Figure]
A.2. Comparison of dominant option usage in AOC and OC
A comparison of the usage of the dominant option in AOC and OC is shown in Figure 6.
At each training checkpoint, the dominant option usage is averaged over 50 test episodes for each of the 15 independent training runs.
The shaded region represents 1 standard deviation.
A.3. Blocked hallway transfer: hard transfers
There are cases where blocking a hallway may mean that the agent has to go all the way around this blockage to reach the goal.
For example, if the goal is in the top right room, the east hallway is blocked and the agent starts in the lower right room, then the agent must navigate all the way around the environment, through 3 hallways, to reach the goal.
This subset of blocked hallway runs are referred to as hard transfers.
A comparison between AOC and OC in handling such hard transfers is shown in Figure 7b and indicates the more apparent beneﬁt of AOC with hard transfers.
It was also observed that on some occasions with hard transfer, OC failed to learn altogether unlike AOC which always learned an optimal policy upon transfer.
The runs shown in Figure 7b are a subset (approximately half) of the runs shown in Figure 7a.
Note that Figure 7a is the same as Figure 2c.
[Figure]
A.4. Hardcoded option attentions
In the case of hardcoded attention where each option’s attention is manually limited to one speciﬁc and distinct room (i.e.for all states inside the room and 0 elsewhere), slower learning is observed.
This is likely because hardcoding attentions de facto removes option choice from the agent, and requires all options to be optimal to get good performance.
When we tried hardcoded attention with 8 options (2 per room), we got better performance, but still signiﬁcantly slower than AOC and OC.
Figure 8 shows the comparison of the learning curves.
Each curve is averaged over 15 runs and the shaded region indicates 0.25 standard deviation.
A.5. Quantitative measures for four-rooms options and attentions
All of the following quantitative measures are averaged over 15 independent runs with different goal locations.
A.5.1. Q UANTITATIVE MEASURE FOR ATTENTION DIVERSITY
After training, the argmax operation applied on the option dimension across the attention maps gives the option with most attention for each state in the environment.
Let the option which has the highest attention in most states be termed the most attentive option and let the ratio of its number of highest attention states to total states be called most attentive option coverage.
Similarly, let the option which has the highest attention in least states be termed the least attentive option and let the ratio of its number of highest attention states to total states be called least attentive option coverage.
The closer both the least and most attentive option coverages are to 25% (in the case of 4 options), the more diverse the attentions.
When the weights w 1 and w 2 are 2.0 (which we found to be the most optimal), least attentive option coverage = 8.07% and most attentive option coverage = 48.58%.
These values indicate that each option has a non-zero area where it is most attentive.
A.5.2. Q UANTITATIVE MEASURE FOR ATTENTION OVERLAP
After training, let the matrix of maximum attention values for each state (across options) be termed as max attention matrix .
Let the matrix of next maximum (2nd highest) attention values for each state (across options) be termed as second max attention matrix .
Let the difference between these two matrices be called diff .
Then, a measure of the percentage of state space area where only one option attends to can be calculated as sum (( diff > 0 . 3)&&( second max attention matrix < 0 . 05)) ∗ 100 /total states .
Here, && denotes the element-wise logical and operation.
This measure calculates the percentage of area where there is no competition among option attentions and there is clearly only one option’s attention for each state in this area.
The higher this measure is, the better.
When the weights wand w 2 are 2.0, this measure was 53.33%.
For the remaining 46.66% of the area, it was usually observed to be the case that 2 options’ attentions competed for this area (note that this also includes cases where the difference in option attentions is very high i.e. 0.5 or greater but where the second highest option attention was non negligible like 0.15).
A.5.3. Q UANTITATIVE MEASURES OF VARIANCE IN OPTION USAGE
The mean option usage for both AOC and OC is near 0.25 for each option (option domination balances out across runs in
OC).
The standard deviation of option usages for AOC and OC are respectively [0.19, 0.19, 0.22, 0.18] and [0.27, 0.33, 0.37, 0.35] i.e. OC has 3 to 4 times more variance.
A.5.4. Q UANTITATIVE MEASURE OF CONSISTENCY BETWEEN OPTION ATTENTIONS AND USAGE
The probability that an option is executed when its corresponding attention in a state is < 0 .
05 is only 0.089.
This indicates that option usage is largely consistent with the corresponding option attentions.
It should be noted that in the cases where multiple options have signiﬁcant non-zero attentions in a state, it can be expected that any of these options may be executed.
For example, Figure 9 shows the case where multiple options attend to states in the bottom right room.
In this case, there is some overlap between the usage of the options that have high attention in these states.
Usage in other rooms is still quite distinct.
[Figure]
B. Other atari experiments B.1. Atari shared-attention results
As described previously, in the shared-attention model, each option’s attention is shared across all the frames of the input stack.
The advantage of this approach is that the obtained attentions are much more distinct and the options are more specialized.
The disadvantage is that the learning performance and the option diversity is sensitive to the chosen hyperparameters.
Figure 10 shows the training curves and the option attentions when trained with hyperparameter values 5000, 0.01, 100, and 1 for the weights w 1 , w 2 , w 3 and w 4 respectively (these weights were obtained after tuning on the Asterix environment with the frame-dependent attention model).
From the ﬁgure, it can be observed that the AOC shared-attention model achieves similar performance compared to OC and also results in diverse options with distinct areas of focus.
B.2. Atari frame-dependent attention results
In the frame-dependent attention model, each option’s attention is learned individually for each frame on the input stack.
Frame stacking implicitly enforces temporal regularization between attentions of successive frames, so we do not specially account for this.
The advantage of this approach is the lower sensitivity towards the attention hyperparameters.
The disadvantage is the increased overlap between option attentions resulting in decreased option diversity.
Figure 11 shows the training curves and the option attentions when trained with hyperparameter values 5000, 0.01, 100, and 1 for the weights w 1 , w 2 , w 3 and w 4 respectively (these weights were obtained after tuning on the Asterix environment with the frame-dependent attention model).
From the ﬁgure, it can be observed that the AOC frame-dependent attention model achieves similar performance compared to OC and also results in diverse options with distinct areas of focus.
Comparing the learning curves of the shared-attention model and the frame-dependent attention model, it can be seen that the latter has slower initial performance, and this is expected since it must learn more parameters (since option attentions are learned individually for each input frame in the stack).
C. Reproducibility and training details
The models are implemented in PyTorch and experiments were run on an NVIDIA V100 SXM2 with 16GB RAM.
C.1. Four-rooms environment
For all experiments in the four-rooms domain, we use the following option learning model for both AOC and the OC baseline: a 2-layer neural (layerwise with 60 and 200 neurons followed by ReLU activation) with fully-connected branches for option values, intra-option policies (with softmax function) and the option terminations (with sigmoid function).
The parameters used for both AOC and baseline OC (after a hyperparameter search) are shown in Table 1.
We performed a grid search across multiple values for w 1 and w 2 , the weights for cosine similarity between the attentions
[Figure]
[Figure]
and the temporal regularization loss respectively.
The search space for both weights was the range [0, 5.0] in increments of 0.5.
The best values (judged according to qualitative attention diversity and quantitative measures explained above) were found to be 2.0 for both w 1 and w 2 .
The shaded regions in Figure 2a represent 0.5 standard deviation, and 0.25 standard deviation in Figures 2b and 2c.
[Figure]
[Figure] [Figure] [Figure]
[Figure]
C.2. Arcade Learning Environment
For experiments in the Arcade Learning Environment, the structure of the option learning model for both AOC and the OC baseline is shown in Table 2.
Each convolution layer is followed by ReLU activation.
The FC1 layer is followed by fully-connected branches for option values, intra-option policies (with softmax function) and the option terminations (with sigmoid function).
For AOC, the
[Table]
structure of the attention learning model is the same as in Table 2, but another layer FC2 is connected to FC1.
In terms of model architecture, the only difference between the shared-attention model and the frame-dependent attention model is the number of neurons in FC2.
For the former, it is equal to the number of pixels in a single frame of the input stack and for the latter it is equal to the total number of pixels in the input stack.
The parameters used for both models of AOC and baseline
OC (after a hyperparameter search) are shown in Table 3.
The input observation is a grayscale 84 × 84 × 4 tensor.
[Table]
We performed a grid search across multiple values for w 1 and w 2 (weights for attention diversity), w 3 (weight for attention sparsity), and w 4 (weight for attention regularity).
The search space for all weights was the range [ 10 − 1 , 10 5 ] in semilogarithmic increments.
The best weight values were found to be 5000, 1.0, 0.01 and 100 respectively, tuned on the shared-attention model for the Asterix environment.
Each atari learning curve is an average over 3 random seeds and the shaded region represents 1 standard deviation.