Temporal Abstractions-Augmented Temporally Contrastive Learning: An Alternative to the Laplacian in RL
Abstract
In reinforcement learning, the graph Laplacian has proved to be a valuable tool in the task-agnostic setting, with applications ranging from skill discovery to reward shaping.
Recently, learning the
Laplacian representation has been framed as the optimization of a temporally-contrastive objective to overcome its computational limitations in large (or continuous) state spaces.
However, this approach requires uniform access to all states in the state space, overlooking the exploration problem that emerges during the representation learning process.
In this work, we propose an alternative method that is able to recover, in a non-uniform-prior setting , the expressiveness and the desired properties of the Laplacian representation.
We do so by combining the representation learning with a skill-based covering policy, which provides a better training distribution to extend and reﬁne the representation.
We also show that a simple augmentation of the representation objective with the learned temporal abstractions improves dynamics-awareness and helps exploration.
We ﬁnd that our method succeeds as an alternative to the Laplacian in the non-uniform setting and scales to challenging continuous control environments.
Finally, even if our method is not optimized for skill discovery, the learned skills can successfully solve difﬁcult continuous navigation tasks with sparse rewards, where standard skill discovery approaches are no so effective.
INTRODUCTION
With the advent of deep reinforcement learning [Mnih et al., 2015] representation learning [c.f. Bengio et al., 2013] has become one of the main topics of interest in reinforcement learning (RL).
In fact, learning in environments with rich
[Figure]
observations and complex dynamics
[e.g., Bellemare et al., 2020] has motivated recent works on learning representations, e.g. controllable or contingent features [Bengio et al., 2017, Choi et al., 2019] on top of which one can potentially learn latent models in the perspective of planning [Hafner et al., 2019, Nasiriany et al., 2019, Schrittwieser et al., 2020] and control [Watter et al., 2015, Banijamali et al., 2018, Hafner et al., 2020].
In this work, we are interested in the task-agnostic setting in which an RL agent ﬁrst interacts with the environment to build a representation, φ , of the state space, S , without relying on any speciﬁc reward signal.
This representation can later be used to solve a task posed in the environment in the form of a reward function.
In this setting, the environment dynamics are the main informative interaction channel available to the agent.
This has motivated the use of graph Laplacian-based methods where the graph vertices correspond to the states and its edges to the transitions probabilities.
The Laplacian’s eigenvectors can been leveraged as a holistic state representation, termed the Laplacian representation, which captures the environment dynamics [Mahadevan, 2005, Mahadevan and Maggioni, 2007].
Wu et al.
[2019] have recently proposed an efﬁcient approximation of the Laplacian representation (L AP - REP ) by framing the graph drawing objective as a temporallycontrastive loss (see Section 2.2).
This formulation works around potentially prohibitive eigendecompositions and extends the Laplacian’s applicability to large (and continuous) state spaces.
However, it assumes access to a uniform sampling prior over all states in the state space.
In practice, this translates in the ability to reset the agent to any state in the environment, which artiﬁcially alleviates the exploration problem.
As we show in Section 5, this assumption is crucial for the quality of the learned representation.
In the absence of the uniform prior privilege, such sampling is not trivial to achieve since the agent has to ﬁrst explore and learn about the state space to be able to access arbitrary states.
In effect, one must handle the exploration along the representation learning in order to preserve the quality of the representation.
In this work, we propose TATC, an alternative representation learning framework to L AP - REP that extends a similar temporally-contrastive approach to a non-uniform-prior setting while preserving the desired properties and quality.
In practice, the representation is trained on data collected with a uniformly random policy.
However, without a uniform access to the state space, the collected data is concentrated around accessible starting states.
To achieve better data collection, we tie the representation learning problem to that of learning a covering strategy.
Brieﬂy, our method consists in using the available representation to train a skillbased covering policy that is in turn used to discover yet unseen parts of the state space, providing novel data to reﬁne and expand the representation.
Our approach, illustrated in Figure 1 shares a similar motivation with several previous works [Machado, 2019, Machado et al., 2017, 2018, Jinnai et al., 2020].
In addition to the aforementioned virtuous learning cycle between representation and skills, we propose to integrate the temporal abstractions captured by the skills in the contrastive representation learning objective.
This augmentation contributes to a better temporally-extended exploration and enforce the representation’s dynamics-awareness , i.e. how representative the representation-induced metric is of distances in the state space.
We empirically show our agent’s ability to progressively explore the state space and consistently extend the domain covered by representation in a non-uniform-prior setting.
We show that our representation leads to better value predictions than L AP - REP , and that it recovers the representation quality expected from a uniform prior.
We also evaluate our representation in shaping rewards for goal-achieving tasks, and we show it outperforms L AP - REP , conﬁrming both its superior ability in capturing dynamics and in scaling to larger environments.
Finally, the skills learned with our framework also prove to be successful at difﬁcult continuous navigation tasks with sparse rewards, where other standard skill discovery methods have limited efﬁcacy.
PRELIMINARIES 2.1 TASK-AGNOSTIC RL
We describe a task-agnostic reinforcement learning (RL) environment as a task-agnostic Markov decision process (MDP) M
= ( S , A , P, γ, d 0 ) where S is the state space, A the action space, P : S ×
A → ∆( S ) is the transition dynamics deﬁning the next state distribution given the current state and action taken, γ ∈ [0 , 1) the discount factor, and d 0 is the initial state distribution.
A policy π : S → ∆( A ) maps states s ∈ S to distributions over actions.
We denote by ∆ the probability simplex.
Knowledge acquired from task-agnostic interactions with the environment (e.g., a representation or a policy) can then be leveraged for speciﬁc tasks.
A task is instantiated with a reward function, R : S → R , which is combined with the task-agnostic MDP.
The task objective is to ﬁnd the optimal policy maximizing the expected discounted return,
E π,d 0 (cid:104) (cid:80) t γ t R ( s t , a t ) (cid:105) , starting from state s 0 ∼ d 0 and acting according to a t ∼ π ( ·| s t ) .
2.2 THE LAPLACIAN REPRESENTATION
The Laplacian representation (L AP - REP ), as proposed by
Wu et al.
[2019], can be learned with the following contrastive objective:
[Equation]
v ) [Equation] [Equation] , (1) where β is a hyperparameter, φ : S → R d is a ddimensional representation, π µ the uniformly random policy (random walk trajectories), D π µ a set of trajectories from π µ (random walks).
We use ( u, v ) ∼ D π µ to denote the sampling of a random transition from D π µ , and similarly u ∼ D π µ for a random state.
Wu et al.
[2019] showed the competitiveness of the Laplacian representation when provided with a uniform prior over S during the collection of
D π µ .
This objective is a temporally-contrastive loss: it is comprised of an attractive term that forces temporally close states to have similar representations, and of a repulsive term that keeps far apart temporally far states’ representations.
Here, the repulsive term was speciﬁcally derived from the orthonormality constraint of the Laplacian eigenvectors.
2.3 THE NON-UNIFORM PRIOR SETTING
In RL, representation learning is deeply coupled to the problem of exploration.
Indeed, the induced state distribution deﬁnes the representation’s training distribution.
For instance,
L AP - REP [Wu et al., 2019] has been learned in the speciﬁc uniform prior .
In this setting, D π µ , from Equation (1), is a collection of trajectories with uniformly random starting states, which provides a uniform training distribution to the representation learning objective.
In the case of a non-uniform prior, the induced visitation distribution can be quite concentrated around the starting states distribution when solely relying on random walks to collect data, hence the need for a better exploration strategy in order to achieve a better training distribution for φ .
To study the problem described above, we investigate the setting in which the environment has a ﬁxed predeﬁned state s 0 to which it resets with a probability p r every K steps; with K of the order of diameter of S .
With a uniformly random behavior policy, this setting is equivalent to a initial state distribution that is concentrated around s 0 and whose density decays exponentially away from it.
We will refer to this setting as the non-uniform-prior (non- µ ) setting, as opposed to the uniform-prior ( µ ) setting where the agent has uniformly access to the state space.
TEMPORAL ABSTRACTIONS AUGMENTED REPRESENTATION LEARNING
In this section, we present Temporal Abstractionsaugmented Temporally-Contrastive learning (TATC), a representation learning approach in which the representation works in tandem with a skill-based covering policy for a better representation learning in the non-uniform prior setting.
Before presenting the components of TATC and how they are trained, we ﬁrst provide an intuitive description of how it operates in the non-uniform prior setting.
A pseudo-code of the algorithm can be found at the end of this section.
3.1 TATC: A SKETCH OF THE ALGORITHM
In the non-uniform-prior setting, the agent is reset to a ﬁxed state s 0 after each episode with some probability p r .
At the beginning of each episode (not necessarily at s 0 , due to the probabilistic resetting), our agent can choose (with some probability p rw ) either to follow the uniform policy π µ or to act according to an exploratory skill-based policy.
Random walk data are, similarly to Wu et al.
[2019], collected to train the representation, while the skills are trained to extend the area of S that the agent is able to efﬁciently reach.
In order to leverage the rich compositionality of skills, the agent executes a sequence of L consecutive skills each time it decides to call the skill-based policy.
Initially, trajectories from π µ cover the vicinity of s 0 , making the representation reliable in that area, i.e. representative of its dynamics.
Inevitably, the skills trained on this representation beneﬁt from its emerging structure and progressively gain behavioral structure: they allow the agent to travel efﬁciently across this explored area.
In other words, the agent becomes capable of reaching the frontier of the explored regions faster, and it is able to collect, using π µ , novel data for the representation.
The latter is hence reﬁned, and its coverage extended.
With a reﬁned representation, the skills are able to reach even further areas.
This process emerges as a virtuous collaboration between the representation and the skills, where both beneﬁt from improving each other.
Eventually, by acquiring more knowledge from unexplored areas, the agent helps overcoming the loss in the representation’s expressiveness that we observed when solely relying on π µ in the non-uniform-prior setting.
In the remainder of this section we ﬁrst propose a generic alternative objective to Equation (1) that suits the non-uniform prior setting, then we describe the exploratory policy training.
Finally, we introduce an augmentation of the proposed objective based on the learned temporal abstractions, to improve exploration and enforce the dynamics-awareness of the representation.
3.2 TEMPORALLY-CONTRASTIVE REPRESENTATIONOBJECTIVE
As mentioned in Section 2.2, the repulsive term in L APREP ’s objective, in Equation (1), derives from the eigenvectors’ orthonormality constraint.
However, because the environment is expected to be progressively covered in the non-uniform prior setting, the orthonormality constraint can make online representation learning highly non-stationary.
For this reason, we adopt the following objective with a generic repulsive term that is more amenable to online learning:
[Equation]
In the following, we describe our representation-based skills training framework.
Beyond addressing the exploration need, these skills will later (Section 3.4) be used to augment the objective above to obtain TATC’s representation learning objective.
3.3 REPRESENTATION-BASED SKILLS TRAINING
In the non-uniform-prior setting, exploration is required to provide the representation with a better training distribution.
To this purpose, we adopt a hierarchical RL approach to leverage the exploration efﬁciency of skills, also known as options [Sutton et al., 1999].
Let φ : S → R d be our ddimensional representation.
The agent acts according to a bi-level policy ( π hi , π low ) .
The high-level policy π hi : S → ∆(Ω) deﬁnes, at each state s , a distribution over a set Ω of directions (unit vectors) in the representation space ( Ω = { δ | δ ∈ R d , (cid:107) δ (cid:107) 2 =1 } ).
Each direction corresponds to a ﬁxed length skill encoded by the low-level policy π low :
S × Ω → ∆( A ) .
These skills are trained to travel in the representation space along the directions instructed by π hi .
In short, given a sampled direction π hi ( ·| s ) ∼ δ ∈ Ω , the low-level policy executes the skill π low ( ·| s, δ ) for a ﬁxed number of steps c before π hi is called again.
Now, we describe the intrinsic rewards used to train the policies π low and π hi .
Low-level Policy.
π low is trained to follow directions picked by π hi in the representation space.
For a given δ ∈ Ω ⊂ R d , the corresponding skill π low ( ·| s, δ ) is trained to maximize the reward function:
[Equation]
where ( s, s (cid:48) ) is an observed state transition, and φ the representation being learned.
We use the cosine similarity as a way to encourage learning diverse directional skills.
Indeed, skills co-specialization is avoided by rewarding the agent for the steps induced along the instructed direction δ regardless of their magnitudes.
Connection to Behavioral Mutual Information.
It is worth noting that our reward design can be interpreted as a mutual-information-based intrinsic control.
We provide more details on this connection in Appendix E.
High-level Policy.
The high-level policy guides the covering strategy.
It does so by sampling the skills of the most promising directions in terms of the amount of exploration, measured by the travelled distance in the representation space.
For this purpose, we design a reward function deﬁned over a sequence of L consecutive skills.
Let { s hi k } Lk
= be the sequence of their initial states and their respective sampled directions, δ k ∼ π hi ( ·| s hi k ) .
Since φ is trained to capture the dynamics, the travelled distance in φ ’s space is a proxy of how far the agent has moved along the MDP dynamics.
In other words, the dynamics-awareness property provides a convenient way to quantify how far the choices made by π hi eventually brought the agent, which can be used to evaluate their exploratory potential.
This observation has motivated the proposed form of the high-level reward which we describe in the following.
Given a high-level trajectory, τ
[Equation]
) , with s hi f the ﬁnal state reached by the last skill, the high-level policy is trained to maximize the quantity:
[Equation]
where δ k ∼ π hi ( ·| s hi k ) is the direction sampled at s hi k .
From the policy optimization perspective, each of these quantities plays the role of the return cumulated along the sampled high-level trajectory and not just a single (high-level) step reward.
This term looks at reaching s hi f as the result of a sequential collaboration of L skills, rewarding them equally.
It values how far the whole sequence of skills has eventually travelled which is the result of selecting and executing each of the sampled directions δ k within the sequence. Thus, this term does depend directly, even if not explicitly , on those directions. Finally, we provide a discussion on the advantage of our high-level reward design (4) over a greedy one in Appendix B.
In the following section, we show how the skills trajectories can be used to augment the representation with the learned temporal abstractions
3.4 AUGMENTING REPRESENTATION LEARNING WITH TEMPORAL ABSTRACTIONS
A skill abstracts a temporally-extended behavior in a single (high-level) action.
As a temporal abstraction , it represents a factorized knowledge of the environment dynamics in the form of a policy.
Here, we propose to integrate these temporal abstractions to the representation objective to better capture the environment dynamics.
In order to preserve the temporal contrast of the base objective (2), we augment it with the following contracting term along skills trajectories:
[Equation]
where D s is a set of collected skills trajectories.
By minimizing this term, φ integrates temporally-extended dynamics: areas connected by skills are brought closer in the representation space.
This term will be referred to as the boredom term.
Its exploratory virtue is discussed in the following.
How does boredom help exploration ?
The interplay between the high-level policy reward function (4) and this boredom term (5) induces a progressive exploration mechanism.
In effect, π hi tends to sample skills that travel further, i.e. with larger R hi .
The more often a skill is sampled, the less rewarding it becomes to π hi due to the minimization of B ( φ ) .
This will favor sampling the remaining (undersampled) skills, hence encouraging the exploration of less visited parts of the state space.
This mechanism dynamically ﬁghts what can be considered as accumulated boredom along over-sampled skills trajectories which increases the agent curiosity and urge it to explore.
Finally, the proposed objective to train the representation φ consists in the base objective (2) augmented with the boredom term (5), and can be written as
L TATC ( φ ; D s , D π µ ) (cid:44) L cont ( φ ; D π µ ) + β (cid:48) B ( φ ; D s ) , (6) with β (cid:48) a hyperparameter controlling the strength of boredom term.  TEMPORAL ABSTRACTIONS AUGMENTEDTEMPORALLY-CONTRASTIVE LEARNING(TATC)INTHE NON-UNIFORMPRIORSETTING The proposed approach consists in simultaneously training the representation φ and the hierarchical agent ( π low , π hi ) . The agent progressively extends the explored area while maintaining the previously collected knowledge.
To do so, in the non-uniform-prior setting, the agent switches with some probability p rw between following a uniformly random policy π µ and executing the hierarchical policy (skills).
The latter helps reach further areas more efﬁciently, where data collected by π µ is used to train the representation φ . Along their training, the skills progressively extend to cover newly discovered areas. Algorithm 1 provides a pseudocode of the proposed approach, in the non-uniform-prior setting.  EXPERIMENTS
In this section, we investigate the behavior of TATC in two types of environments: gridworld environments with discrete state and action spaces, and continuous navigation environments for continuous state spaces (MuJoCo, Todorov et al.
For our method, we learn a 2D representation ( d = 2 ), and deﬁne Ω as a set of 8 unit vectors equally spaced on the unit sphere.
These directions are concatenated to the input state of π low .
Implementation details of all the experiments in this section can be found in Appendix C.
5.1 GRIDWORLD
We evaluate our approach in three gridworld domains: UM AZE , T-M AZE and 4- ROOMS .
These environments, visualized in Figure 2, raise different explorations challenges.
U-M AZE is a simple but relevant environment to test the Algorithm 1 TATC in the setting 1: Input: L, c, p rw , N 2: for iteration = 1 , 2 , . . . do 3: D π µ = ∅ , D s
= ∅ 4: for batch
= 1 , 2 , . .
, N do 5:
Reset to s 0 with probability p r .
6: p ∼ Unif ([0,1]) 7: if p < p rw then 8:
Run the uniformly random policy π µ to collect
L random walk trajectories { τ (cid:48) i } Li =1 of c steps each.
D π µ ← D π µ ∪ { τ (cid:48) k
} Lk = 10: else 11:
Run ( π hi , π low ) to collect L consecutive skills’ trajectories { ( τ k , δ k ) }
Lk =1 and their corresponding directions 12:
D s ← D s ∪ { ( τ k , δ k ) }
Lk = 13: end if 14: end for 15:
Optimize the policies ( π hi , π low ) using the intrinsic objectives 3 and16:
Optimize φ so as to minimize L TATC ( φ ; D s , D π µ ) (Equation 6).
17: end for dynamics-awareness of the representations; 2 T-M AZE raises the challenge of splitting the exploration focus at an intersection while maintaining the covering in both corridors;
4- ROOMS is similar to U-M AZE 3 , but requires learning more controlled skills for a useful exploration.
[Figure]
5.1.1 Progressive Representation Learning
Figure 3 shows the evolution of the representations throughout training.
The agent progressively explores the environment starting around s 0 , and builds the representation by continuously integrating newly discovered parts [Figure] e [Figure]
( f ) ( g ) ( h ) ( i ) ( j )
[Figure]
U-M AZE & 4- ROOMS .
The agent starts from the bottom left corner, or room, of the maze.
Figure 3 shows how the representation progressively expands away from s 0 along the corridor, or the rooms sequence.
Note that while the agent learns to reach and represent further areas, the full domain representation ﬂattens , indicating the representation’s success in capturing the maze dynamics.
T-M AZE .
The agent starts from the bottom left corner of the maze.
As in the U-Maze, it starts learning to travel along the corridor until it reaches the intersection.
There, the exploration focus is shared between both possible paths whose representations are progressively disentangled.
Eventually, the agent fully explores both corridors and ﬁnalizes its representation.
Note that, the discovery of one of the corridors did not hinder ﬁnishing the discovery of the other.
The boredom term proved to be important for such property (see
Appendix A).
Importance of Boredom.
Appendix A provides an ablation study showing the importance of the boredom term for the agent’s exploratory behavior and the representation’s dynamics-awareness.
5.1.2 Evaluating the Learned Representation
We now compare our representation against L AP - REP [Wu et al., 2019] in the non-uniform-prior setting.
First, to appreciate the sensitivity of L AP - REP to the uniformity of said prior, we trained L AP - REP in two settings: (i) the uniformprior setting where the agent can be set to any arbitrary state, as done by Wu et al.
[2019], and (ii) the non-uniform-prior setting deﬁned in Section 2.3.
In the following, we show that
L AP - REP is sensitive to this change in distribution while
TATC recovers the expressive potential that a uniform prior provides.
Prediction.
To evaluate the learned representations, we ﬁrst consider how well they linearly approximate a given task’s optimal value function.
To do so, we train an actor-critic agent [Mnih et al., 2016] with a linear critic on top of each representation.
In Figure 4, we note a signiﬁcant loss in the representational power of L AP - REP when the access to the state space is not uniformly distributed anymore.
This ﬁgure also shows that TATC outperforms L AP - REP in the non-uniform-prior setting, and succeeds in recovering L APREP ’s expressive power when it is learned with the unrealistic uniform prior.
Control.
We also compare the representations from the perspective of control, by training a deep actor-critic agent on top of each representation to solve a goal-reaching task in the same domains as above.
The agent is only rewarded upon reaching the goal state ( r = 1 ).
Figure 5 shows that
TATC consistently outperforms L AP - REP , which conﬁrms the competitive quality of our representation. [Figure] [Figure]
5.2 CONTINUOUS CONTROL
The second set of experiments focuses on continuous state and action spaces.
Here, we consider two mazes for the
MuJuCo Ant agent, as shown in Figure 6.
These are similar in shape to the ones from Wu et al.
[2019], but are larger and thus more challenging.
More details in Appendix C.
[Figure]
Axes scales were equalized. We can visually appreciate how the continuous state domains are mapped to a ﬂatter manifold reﬂecting the presence of the walls.
To visualize our learned representations in these environments, Figure 6 depicts a grid of positional states in each environment domain and their mapped representations.
Similarly to U-M AZE and 4- ROOMS , the learned representation translates the environment dynamics, and speciﬁcally the presence of walls, by mapping the original state space to a ﬂatter manifold.
5.2.1 Reward Shaping with Learned Representations
We demonstrate how TATC representation is able to improve an RL agent’s performance when the distances in the representation space are used for reward shaping, the same setting in which Wu et al.
[2019] evaluated L AP - REP .
We deﬁne a goal-achieving task by setting a goal state g at the end of the corridor (visualized in Appendix C).
The objective is to learn to navigate to a state s close enough to the goal area ( (cid:107) s − g (cid:107) 2 ≤ (cid:15) ).
We deﬁne the reward function based on the distance in representation space (TATC and
L AP - REP ).
More speciﬁcally, we train a soft actor-critic (SAC) agent [Haarnoja et al., 2018] to reach the goal with a dense reward deﬁned as r denset = −(cid:107) φ ( s t +1 )
− φ ( g ) (cid:107) 2 .
Similarly to Wu et al.
[2019], we also compare against the half-half mix of the dense reward and the sparse reward r
[Equation]
For this evaluation, L AP - REP was learned, unlike our representation, with a uniform prior over S as in Wu et al.
[2019], and d = 2 .
4 Figure 7 shows that our representation is effec- tive in reward shaping, with both mix and dense variants , and enjoys a comparable if not superior dynamics-awareness to L AP - REP .
This result stands while TATC is learned from a non-uniform prior which is less advantageous than the uniform prior used to train L AP - REP .
Note that L AP - REP with a non-uniform prior fails to guide till the goal.
Finally, these results further conﬁrm the conclusions drawn from the gridworld experiments and positions TATC as a competitive alternative to L AP - REP in this challenging setting.
[Figure]
5.2.2 Evaluating the Learned Skills
To evaluate the exploratory potential of TATC’s skills, we compare the learned skills in A NT M AZE -1 against 2 taskagnostic skill discovery methods, DIAYN [Eysenbach et al., 2019]
[Figure]
and DCO [Jinnai et al., 2020].
DIAYN is a mutual information-based approach to learn diverse set of skills, while DCO skills are based on a temporally-contrastive representation, similarly to TATC.
More speciﬁcally, DCO requires a pretrained L AP - REP which approximates the Laplacian’s second eigenvector; also called the Fiedler vector.
We train the required representation, as well as DCO, with the advantage of data collected from a uniform prior over
S .
For a fair comparison, we train 8 skills for both methods (DCO and DIAYN).
Once trained, the skills learned by each method are ﬁxed and used to train a discrete highlevel policy that can select across the available skills to solve a goal-reaching task with a sparse reward function
[Equation] [Figure]
The sparsity of the reward naturally poses a challenge as no additional signal can guide the agent towards the goal, unlike the evaluation setting of DCO and DIAYN by Jinnai et al.
Figure 8 shows that the skills learned by
TATC quickly assist to complete the task while the skills learned with DCO and DIAYN do not.
DIAYN’s limited performance in difﬁcult sparse-reward navigation tasks was also conﬁrmed by Kamienny et al.
These results suggest that in order to succeed, DCO and DIYAN skills may require a richer signal like the distance-based dense reward used by Jinnai et al.
[2020] to evaluate both of them – and where they show similar performances.
RELATED WORK
Our work is related to self-supervised learning [Bromley et al., 1993, Chopra et al., 2005], which brought recent advances in representation learning [Bachman et al.
, He et al., 2020, Chen et al., 2020, Grill et al., 2020, Caron et al., 2020].
These techniques have naturally been adapted to RL, especially contrastive methods.
While some of these beneﬁted from visually contrasting observations [Laskin et al., 2020, Yarats et al., 2021], others leveraged temporal contrasts to learn useful representations [Mazoure et al., 2020, Stooke et al., 2021, Li et al., 2021], which fall closer to our work.
We designed our covering policy as a hierarchical agent.
This has actually been the default setting to model temporally-extended actions [Sutton et al., 1999].
Our work shares the same motivation as Vezhnevets et al.
[2017] for training skills to follow latent directions.
Among the large body of work on skill discovery, the eigenoptions framework [Machado et al., 2017] and its extensions [Machado et al., 2018, 2021, Jinnai et al., 2020] are probably the closest to our skill training scheme.
Eigenoptions also ﬁt in the directional skills deﬁnition as they are trained to travel along the directions deﬁned by the eigenvectors of the Laplacian.
These vectors have a dimensionality of |S| which can be very large.
To contrast, our directional skills are deﬁned by an arbitrary and diverse set of directions in the learned representation space of adjustable dimensionality, which offers more tractability.
TATC skills and eigenoptions also share an interesting connection to mutual information based intrinsic control methods [Hansen et al., 2020], which we discuss in
Appendix E. Indeed, a similar discussion can be adapted to eigenoptions’ intrinsic reward function.
The exploration mechanism in TATC emerges from the interplay between the representation and the covering policy rewarding scheme.
As discussed in Section 3.4, the boredom term prevents the high-level policy from focusing exploration resources on previously over-sampled skills.
This penalization induces optimism towards the remaining skills, in the hope of deviating from previously explored regions.
Zhang et al.
[2021] share a similar motivation and derive an exploration bonus that aims at directly maximizing this deviation in terms of the policy occupancy.
Both methods are inspired from the optimism-in-the-face-of-uncertainty principle.
The incremental discovery paradigm has been previously adopted, either for exploration [Ecoffet et al., 2021], incremental skill discovery [Jinnai et al., 2020, Pong et al., 2020], or even state abstraction [Misra et al., 2020].
Finally, we use learned skills to penalize boredom
[Schmidhuber, 1991,
Oudeyer and Kaplan, 2009] in the representation space and encourage exploration.
The idea of using skills to foster curiosity has also been investigated by Bougie and Ichise [2020].
CONCLUSION
The Laplacian representation as proposed by Wu et al.
[2019] made the beneﬁts of spectral methods affordable in large state spaces.
Unfortunately, the quality of this representation is strongly tied to the uniformity of its training data distribution, as shown is Section 5.
This has motivated the method proposed in this work where we reconcile a similar temporally-contrastive representation with exploration demanding settings.
Our approach leverages the practical skill training framework that such representations allow.
The learned skills are used to better cover the state space and hence learn a better representation.
We validate our method in tabular as well as continuous environments.
Our representation learned in a non-uniform-prior setting shows a comparable representational power to the one acquired by a
Laplacian representation from a uniform prior, and our skills proved to be competitive in hard continuous control tasks.
With these results, we hope to bring such representations’ applicability one step closer to realistic contexts.
We have proposed to augment the representation objective with temporal abstractions captured in the acquired skills.
This beneﬁts exploration by inducing a boredom-ﬁghting mechanism, and enforces the representation’s dynamicsawareness.
Intuitively, this augmentation can be seen as bringing temporally close regions (connected through skills) closer in the representation space.
This observation may motivate further investigations on how temporal abstractions can improve the representational potential and beneﬁt taskagnostic representations in RL.
ACKNOWLEDGEMENT
AE would like to thank Emmanuel Bengio, Amy Zhang, and Ahmed Touati for the insightful discussions, and Jad
Kabbara and Riashat Islam for their feedback on an earlier version of this paper.
We also thank all the reviewers for their feedback and valuable suggestions.
The authors acknowledge the ﬁnancial support of CIFAR, NSERC and
Mila.