Learning Task Agnostic Skills with Data-driven Guidance
Abstract
To increase autonomy in reinforcement learning, agents need to learn useful behaviours without reliance on manually designed reward functions.
To that end, skill discovery methods have been used to learn the intrinsic options available to an agent using task-agnostic objectives.
However, without the guidance of task-speciﬁc rewards, emergent behaviours are generally useless due to the underconstrained problem of skill discovery in complex and high-dimensional spaces.
This paper proposes a framework for guiding the skill discovery towards the subset of expert-visited states using a learned state projection.
We apply our method in various reinforcement learning (RL) tasks and show that such a projection results in more useful behaviours.
1. Introduction
While autonomous learning of diverse and complex behaviors is challenging, signiﬁcant progress has been made using deep reinforcement learning (DRL).
The progress has been accelerated by the powerful representational learning of deep neural networks (LeCun et al., 2015), and the scalability and efﬁciency of RL algorithms (Mnih et al., 2015;
Schulman et al., 2017; Haarnoja et al., 2018; Lillicrap et al.).
However, DRL still involves an externally designed reward function that guides learning and exploration.
Manually engineering such a reward function is a complex task that requires signiﬁcant domain knowledge in such a way that hinders autonomy and adoption of RL.
Prior works have proposed using unsupervised skill discovery to alleviate these challenges by using empowerment as an intrinsic motivation to explore and acquire abilities (Salge et al., 2014; Gregor Accepted by the ICML 2021 workshop on Unsupervised Rein- forcement Learning , PMLR 139, 2021. Copyright 2021 by the author(s). et al., 2016;
Eysenbach et al., 2018; Sharma et al., 2019;
Campos et al., 2020).
Although skill discovery without a reward function can be helpful as a primitive for downstream tasks, most of the emergent behaviours in the learned skills are useless or of little interest.
This is a direct consequence of underconstrained skill discovery in complex and high dimensional state space.
One possible solution is to leverage prior knowledge to bias skill discovery towards a subset of the state space through a hand-crafted transformation of the state space (Eysenbach et al., 2018; Campos et al., 2020).
However, utilizing prior knowledge to hand-craft such a transformation contradicts the primary goal of unsupervised
RL, which is reducing manual design efforts and reliance on prior knowledge.
Instead, we explore how to learn a parameterized state projection that directs skill discovery towards the subset of expert-visited states.
To that end, we employ examples of expert data to train a state encoder through an auxiliary classiﬁer, which tries to distinguish expert-visited states from random states.
We then use the encoder to project the state space into a latent embedding that preserves information that makes expert-visited states recognizable.
This method extends readily to other mechanisms of learned stateprojections and different skill discovery algorithms.
Crucially, our method requires only samples of expert-visited states, which can easily be obtained from any reference policy, for example expert demonstrations.
The key contribution of this paper is a simple method for learning a parameterized state projection that guides skill discovery towards a substructure of the observation space.
We demonstrate the ﬂexibility of our state-projection method and how it can be used with the skill-discovery objective.
We also present empirical results that show the performance of our method in various locomotion tasks.
2. Related Work
Unsupervised reinforcement learning aims at learning diverse behaviours in a task-agnostic fashion without guidance from an extrinsic reward function (Jaderberg et al., 2016).
This can be accomplished through learning with an intrin- sic reward such as curiosity (Oudeyer & Kaplan, 2009) or empowerment (Salge et al., 2014).
The notion of curiosity has been utilized for exploration by using predictive models of the observation space and providing a higher intrinsic reward for visiting unexplored trajectories (Pathak et al., 2017).
Empowerment addresses maximizing an agent’s control over the environment by exploring states with maximal intrinsic options (skills).
Several approaches have been proposed in the literature to utilize empowerment for skill discovery in unsupervised RL.
Gregor et al.
(Gregor et al., 2016) developed an algorithm that learns intrinsic skill embedding and used generalization to discover new goals.
They used the mutual information between skills and ﬁnal states as the training objective and hence used a discriminator to distinguish between different skills.
Eysenbach et al.
(Eysenbach et al., 2018) used mutual information between skills and states as an objective while using a ﬁxed embedding distribution of skills.
Additionally, they used a maximum-entropy policy (Haarnoja et al., 2018) to produce stochastic skills.
However, most of the previous approaches assume a state distribution induced by the policy itself, resulting in a premature commitment to already discovered skills.
Campos et al.
(Campos et al., 2020) used a ﬁxed uniform distribution over states to break the dependency between the state distribution and the policy.
Certain prior work has addressed the challenge of complex and high dimensional state space by constraining the skill-discovery in a subset of the state space.
Sharma et al.
(Sharma et al., 2019) learned predictable skills by training a skill-conditioned dynamic model instead of a discriminator to model speciﬁc behaviour in a subset of the state space.
Eysenbach et al.
(Eysenbach et al., 2018) proposed incorporating prior knowledge by conditioning the discriminator on a subset of the state space using a hand-crafted and a task-speciﬁc transformation.
Our work addresses this challenge by guiding the skill discovery towards the subset of expert-visited states.
In contrast to inverse reinforcement learning, (Fu et al., 2018), we do not explicitly infer the extrinsic reward.
Crucially, we do not try to learn the expert policy directly in contrast to behaviour cloning or imitation learning (Ross et al., 2011).
Our proposed method resembles the algorithm proposed by Li et al.
(Li et al., 2020) in which they used a Bayesian classiﬁer that estimates the probability of successful outcome states, resulting in a more task-directed exploration.
However, their algorithm does not optimize the mutual information; hence it does not learn diverse skills via the discriminability objective.
3. Preliminaries
In this paper, we formalize the problem of skill discovery as a Markov decision process (MDP) without a reward function: M = ( S , A , P ) , where S is the state space, A is the action space, and
P : S × S ×
A → [0 , ∞ ) is the tran- sition probability density function.
The RL agent learns a skill-conditioned policy π ϑ ( a | s, z ) , where the skill z is sampled from some distribution p ( z ) .
A skill, or option (as ﬁrst introduced in (Sutton & Barto, 2018)), is a temporal abstraction of a course of actions that extends over many time steps.
We will also consider the informationtheoretic notion of mutual information between states and skills I ( S ; Z )
=
H ( Z ) − H
( Z | S ) , where H ( · ) is the Shan- non entropy.
3.1. Skill Discovery Objective
The overall goal of skill discovery is to ﬁnd a policy π ϑ capable of carrying out different tasks that are learned without extrinsic supervision for each type of behavior.
We consider policies of the form π ϑ ( a | s, z ) that specify different distributions over actions depending on which skill z they are conditioned on.
Although this general framework does not constrain how z should be represented, we deﬁne it as a discrete variable since it has been empirically shown to perform better than continuous alternatives (Eysenbach et al., 2018).
We follow the framework proposed by the "diversity is all you need" (DIAYN)
algorithm
(Eysenbach et al., 2018), in which skills are learned by deﬁning an intrinsic reward that promotes diversity.
Intuitively, each skill should make the agent visit a unique section of the state space.
This can be expressed as maximising the mutual information
I ( S ; Z ) of the state visitation distributions for different skills (Salge et al., 2014).
To ensure that the visited areas of the state space are spaced sufﬁciently far apart, we use a soft policy that maximises the entropy of the action distribution.
Formally, we maximize the following objective function:
[Equation]
The ﬁrst term H
[ A | S, Z ] means that the policy should act as randomly as possible and can be optimized by maximizing the policy’s entropy.
The second term −H [ Z | S ] dictates that each visited state should (ideally) identify the current skill.
The third term is the entropy of the skill distribution, which can be maximized by deliberately sampling skills from a uniform distribution during training.
Unfortunately, −H [ Z | S ] requires knowledge about p ( z | s ) , which is not readily available.
Consequently, we approximate the true distribution by training a classiﬁer q φ ( z | s ) , leading to a lower bound:
[Equation]
The lower bound follows from the non-negative property of the Kullback-Leibler divergence D KL ( p || q ) =
E p [log p ( z | s ) − log q ( z | s )] ≥ 0 , which can be rearranged to E p [log p ( z | s )] ≥ E p [log q ( z | s )]
(Agakov, 2004).
The classiﬁer q φ is ﬁtted throughout training with maximum likelihood estimation over the sampled states and active skills.
This leads to a scenario where the policy is rolled out for a (uniformly) sampled skill, and the classiﬁer is trained to detect the skill based on the states that were visited.
The policy is given a reward proportional to how well the classiﬁer could detect the skill in each state.
In the end, this should make the policy favor visiting disjoint sets of states for each skill, leading to a cooperative game between q φ and π ϑ .
3.2. Limitations of existing methods
A major challenge that arises when maximizing the objective in Equation 2, particularly in applications with highdimensional spaces, is that it becomes trivial for each skill to ﬁnd a sub-region of the state space where it is easy to be recognised by q φ .
In preliminary experiments, we observed that the existing methods discovered behaviours that covered small parts of the state space.
For the HalfCheetah environment (Brockman et al., 2016)
this resulted in many skills generating different types of static poses (see Figure 1) and not many skills exhibiting "interesting" behaviour such as locomotion.
Optimising for H [ A | S, Z ] should mitigate this issue to some extent.
Increasing the policy’s entropy incentivises the skills to progressively visit regions of the state space that are so far apart that not even highly stochastic actions will cause them to overlap accidentally.
However, it has been shown that mutual information based algorithms have difﬁculties spreading out to novel states due to low values of log q φ ( z | s ) for out-of-sample states (Campos et al., 2020).
4. Proposed Method
The main idea of our approach is to focus the skill discovery towards certain parts of the state space by using expert data as a prior.
The DIAYN algorithm can be biased towards a user-speciﬁed part of the state space by changing the discriminator to maximize E [log q φ ( z | f ( s ))] , where f represents some transformation of the state space (Eysenbach et al., 2018).
Instead of using a hand-crafted f to improve the skills discovered for a navigation task, we aim to learn a parameterized f χ by using expert data.
4.1. State Space Projections
We consider linear projections of continuous factored state representations on the form f χ : R | S | −→ R | E | with e = f χ ( s ) = χs , χ ∈ R | E |×| S | , and | E | < | S | .
In principle, the idea should apply to more complex mappings, such as a multi-layer perceptron.
However, we want to limit the scope of skill discovery to a hyperplane within the original state space.
For the same reason, we also omit any non-linearities in the encoder.
Squeezing the output through a Sigmoidal function would limit discriminability at the (potentially interesting) extremes of the encoding.
Similarly, a ReLU function would effectively eliminate all exploration along the negative direction of f χ i .
In summary, the objective of the DIAYN skill classiﬁer becomes:
[Equation]
We learn the parameters χ for the projection through an auxiliary discriminative objective.
Speciﬁcally, a binary classiﬁer h ψ : R | E | −→ { 0 , 1 } is trained to predict whether an (encoded) state was sampled from the marginal state visitation distribution of a random policy π rand or from the distribution of a reference (expert) policy π ∗ .
Let x ∼ D denote whether a state s was visited by the reference policy or not (in dataset D ), then the parameters of f χ are obtained through joint pretraining with h ψ by maximizing the log likelihood over D :
[Equation]
where the dataset D is collected prior to training the main
RL algorithm.
The ﬁrst half (random samples) are collected by rolling out π rand whereas the second half (reference samples) are collected by rolling out π ∗ .
After the objective in
Equation 4 is optimized, the discriminator h ψ is discarded and the projection encoding f χ ( s ) is extracted to be used for the objective in Equation 3.
Analogous to autoencoders (Hinton & Salakhutdinov, 2006), the idea is that the embeddings produced by f χ ( s ) should now contain a more compact representation of the state space without collapsing the dimensions that make "interesting" behaviour stand out.
While the use of a reference data changes our approach from a strictly unsupervised skill discovery algorithm, the discriminative objective in equation 4 resembles the objectives used in adversarial inverse reinforcement learning (e.g.
(Fu et al., 2018)).
However, it differs in that it makes no attempts at matching the behaviour of a reference policy as it is used only as a prior for simplifying the state space.
This approach could also be used with samples from several different reference policies with substantially different marginal state distributions.
As long as their variation can be explained sufﬁciently without full use of the entire state space, a projection should simplify skill discovery.
4.2. Implementation
For learning diverse skills, we use DIAYN as a basis framework.
DIAYN uses the Soft Actor-Critic (SAC) algorithm (Haarnoja et al., 2018) that is optimized using policy gradient style updates in contrast to the reparameterized version (DDPG style updates (Lillicrap et al.)).
They also use a
Squashed Gaussian Mixture Model to represent the policy a ∼ π ϑ = tanh GMM ( µ ϑ ( s ) , σ ϑ ( s )) .
The learning objective is to maximize the mutual information between the state and skill I ( S ; Z ) .
This objective is optimized by replacing the task rewards with a pseudo-reward
[Equation]
where q φ is trained to discriminate between skills and p(z) is the ﬁxed uniform prior over skills (Eysenbach et al., 2018).
A skill is sampled from z ∼ p ( z ) and used throughout a full episode.
In contrast to DIAYN, we use two Q-functions Q 1 θ ( s, a ) & Q 2 θ ( s, a ) where both Q-functions attempt to predict the same quantity.
This allows us to sample differentiable actions and climb the gradient of the minimum of the two
Q-functions (DDPG-style update (Lillicrap et al.)), giving us this objective:
[Equation]
Like in DIAYN, we also use a Squashed Gaussian Mixture
Model to promote diverse behaviour.
Figure 2 illustrates the training process of the proposed expert-guided skill discovery.
First, we train the encoder f χ jointly with the auxiliary classiﬁer h ψ using the external dataset D .
Secondly, we train the agent using an ofﬂine policy algorithm (SAC), in which the agent samples a skill z ∼ p ( z ) , and then interacts with the environment by taking action a t according the skill-conditioned policy π ϑ ( a t | s t , z ) .
The environment, then, transits to a new state according to the transition probability s t +1 ∼ p ( s t +1 | s t , a t ) .
We add this transition ( s z , z, a t , s t +1 ) to the replay buffer B .
Simultaneously, the policy is updated by sampling a mini-batch from the replay buffer M ∼ B , then encoding the next states e t +
=
f
χ ( s t +1 ) and passing them through the discriminator q φ ( z | e t +1 to get the intrinsic reward.
This reward is used by the Q-functions Q θ ( s t , a t , z ) to minimize the soft
Bellman residual and update the policy.
A pseudocode for the proposed approach can be found in the supplementary material
[Equation]
[Table] +4A2lMl00g6dzISZG6GEfoYbF4q49Wvc+TdO2iy0emDgcM69zLknTAQ36HlfTmltfWNzq7xd2dnd2z+oHh51jEo1ZW2qhNK9kBgmuGRt5ChYL9GMxKFg3XB6m/vdR6YNV/IBZwkLYjKWPOKUoJX6g5jghBKRNefDas2rewu4f4lfkBoUaA2rn4ORomnMJFJBjOn7XoJBRjRyKti8MkgNSwidkjHrWypJzEyQLSLP3TOrjNxIafskugv150ZGYmNmcWgn84hm1cvF/7x+itFNkHGZpMgkXX4UpcJF5eb3uyOuGUUxs4RQzW1Wl06IJhRtSxVbgr968l/Suaj7V3Xv/rLWaBZ1lOEETuEcfLiGBtxBC9pAQcETvMCrg86z
[Figure]
a8L0dLTrFzDL/
[Figure]
5. Experiments
In our experimental evaluation
[Figure]
we aim to demonstrate the impact of our approach of restricting skill discovery to a projection subspace.
We verify our method on both pointmazes and continuous control locomotion tasks.
All the code for running the experiments are publicly available on
GitHub 1 .
5.1. Point Maze
As an illustrative example, we begin by testing the algorithm on a simple 2D point-maze problem.
The term maze is used very generously here, as the environment consists of an open plane in R 2 enclosed by walls that restrict the agent to 17 < x < 67 and 17 < y < 67 .
At initialization, the agent is dropped down at x, y = 7+ (cid:15) 14 , (cid:15) ∼ U ( − 1 , 1) and incentivized to move towards the lower right by a reward proportional to a gaussian kernel centered at ( 914 , 314 ).
The agent is free to move by ± 170 in both the x and y direction.
[Figure]
We train a SAC agent against the extrinsic environment reward to convergence and set its ﬁnal policy as the reference policy π ∗ .
We then sample 10 trajectories of length(green dots in Figure 3) with π ∗ , as well as 10 trajectories of length 100 (blue dots in Figure 4) from a uniform random policy π rand .
The resulting dataset D consists ofsamples and is used to train h ψ ( x | e ) until it can distinguish states from π ∗ and π rand with around 98% accuracy.
For this experiment, we project down from 2D to 1D, making χ ∈ R 1 × 2 .
The resulting projection axis is visualized as a red line in Figure 3 and is the only thing exported to the next stage of the algorithm.
We then train two versions of the DIAYN algorithm; a baseline using the states as-is in the classiﬁer ( q φ ( z | s ) ) and our proposed method using the state projections ( q φ ( z | f χ ( s )) ).
All other hyperparameters are held equal in the two experiments, and the algorithms are trained for 400,000 environment interactions, each attempting to learn 10 distinct skills.
Figure 4 visualizes the results of the baseline and the state projection to the left and right, respectively.
The top row shows the states that were visited for ﬁve rollouts of each skill.
As expected, the baseline skills spread out in all [Figure] directions (albeit slightly more so towards the left) and converge on locations that are easy to distinguish with a 2D state representation.
In contrast, the skills generated in the left plot form lines along the state projection axis.
Their wide lateral spread follows from the (unbounded) entropy maximization objective.
Besides, any movement perpendicular to the projection axis will not affect the 1D vector passed to the classiﬁer.
5.2. Mujoco Environments
Next, we evaluate the algorithm on three continuous control problems from the OpenAI gym suite.
(Brockman et al., 2016).
The environments include HalfCheetah ( S = R 17 ,
A = R 6 ), Hopper ( S = R 11 , A = R 3 ) and Ant ( S =
R 27 , A = R 8 ).
We choose these environments because they involve substantially different locomotion methods.
Additionally, they have observation spaces with different dimensionality, which enable us to better investigate the projection impact.
We do one baseline run without any state projection for all three problems, one with a projection down to R 5 , and one with a projection down to R 3 .
We use our base SAC implementation to obtain reference policies π ∗ and sam- [Figure] n [Figure]
[Table]
v
DIAYN -1.
[Table] [Table] + ENC [Table]
DIAYN + ENC(5) [Table] -0.1 ±
[Table]
ple 10 trajectories of length 1000 with fairly high returns (Ant: 5063 .
9 ± 469 . 8 , Cheetah: 10656 .
4 ± 673 . 5 , Hopper: 3348 . 0 ± 316 . 0 ).
The DIAYN algorithm is otherwise identical to (Eysenbach et al., 2018) in terms of hyperparameters;
Q , π , q φ , and h ψ use MLP architectures with 2 hidden layers of width 300, the entropy bonus weight α is set to 0.1, and the number of skills is set to 50.
We limit each skilldiscovery run to 2.5 million environment interactions but repeat each experiment 5 times with different random seeds (including training of SAC agents for π ∗ ).
For quantitative evaluation, we look at the displacement along the target locomotion axis for the extrinsic objective.
In our approach, we would expect to observe skills that cover this axis well, i.e., skills that run forward and backward at different speeds.
To test this, we roll out each skill deterministically 2 , record its movement over 1000 time steps (or until it reaches a terminal state) and observe the inter-skill spread.
A similar assessment is possible by only looking at the environment’s rewards.
However, the environment reward also includes terms for energy expenditure, staying alive (for Ant/Hopper), and collisions (Ant), which would obscure the results.
Figure 6 shows the displacement distribution of the 50 skills across all runs.
The same information is summarized numerically in Table 1.
For a qualitative evaluation, we have also composed a video with every skill across all runs 3 .
6. Discussion
For HalfCheetah and Hopper, the runs with state encoding (+ ENC(3|5)) exhibit a substantially larger spread than the baseline.
The best forward-moving cheetah skill movesunits forward ( = 3311 environment return), and the best backwards-moving cheetah skill moves 186 units backwards ( = − 4025 environment return).
For the hopper environment, the best forward-moving skill manages to jump units forward, which corresponds to an environment reward of 3268, which is on the same level as the reference data used to ﬁt its encoder.
The results in the Ant environment are less impressive.
There is hardly any difference in how the displacements are distributed for the three approaches, and the total movement is almost negligible.
For reference, a good Ant agent trained against the extrinsic reward should obtain displacements in the 100s when evaluated over the same trajectory horizon.
Looking at the generated Ant behaviour, we found that the skills produced with encoders typically moved even less than those generated by the baseline.
This is not because it is impossible to generate a linear projection that promotes locomotion at various speeds, as the state representation of all three problems contains a feature for linear velocity along the target direction.
Moreover, the skill classiﬁer does reach a high accuracy (some breaking 90%), so the algorithm manages to ﬁnd distinguishable skills.
We, therefore, suspect that the procedure used to ﬁt the encoder is insufﬁcient for this environment.
While it does pick up on linear velocity, it also picks up on several other features from the state space, which might have made it easier for the algorithm to make the skills distinguishable.
To better understand the results of the Ant experiment, we investigate the projection matrix learned at the start of the algorithm.
Figure 7 gives a representative example of a projection learned for an ENC(3) run.
In the diagram, each bar indicates the impact each feature of the state space has on the ﬁnal embedding.
The orange bar highlights the feature corresponding to linear torso velocity in the x-direction, i.e. the direction in which the extrinsic objective rewards an agent for running in.
All the bars to the left correspond to joint conﬁgurations, link orientations, and all the bars to the right correspond to other velocities.
The feature for velocity in the target direction is well represented.
However, so are the features for the 8 joint velocities (8 rightmost bars in each group).
Since it is a lot easier to move a single joint than to coordinate all of them for locomotion, the algorithm might more easily converge to this strategy than ﬁgure out a way to walk.
Moreover, because the projection mixes features for movement of single joints with features for locomotion of the entire body, it becomes more difﬁcult for the classiﬁer to distinguish the two.
For instance, an ant that ﬁgures out how to walk may (in the projected space) look similar to one that only twitches some of its joints. [Figure]
7. Conclusion
In this work, we propose a data-driven approach for guiding skill discovery towards learning useful behaviors in complex and high-dimensional spaces.
Using examples of expert data, we ﬁt a state-space projection that preserves information that makes expert behavior recognizable.
The projection helps discover better behaviors by ensuring that skills similar to the expert are distinguishable from randomly initialized skills.
We show the applicability of our approach in a variety of RL tasks, ranging from a simple 2D point maze problem to continuous control locomotion.
For future work, we aim to improve the embedding scheme of the state projection to be suitable for a wider range of environments.
Acknowledgment. We would like to thank Kerstin Bach and Rudolf Mester for their useful feedback.
A. Pseudocode
input : Replay buffer B , dataset of expert states D .
Initialize policy π ϑ , Q-functions { Q 1 θ ,Q 2 θ } , discriminator q φ , encoder f χ .
Function Pre-train-Encoder( f χ , D ) :
Initialize classiﬁer h ψ . Sample actions from a random policy: a t ∼ π rand ( a t | s t ) . Step environment using the random actions: s t +1 ∼ p ( s t +1 | s t ,a t ) . Add visited states to the dataset: D ← D ∪ { ( s t +1 , 0) } .
Update encoder ( f χ ) through joint training with the classiﬁer h ψ to maximize the likelihood of D .
Discard the classiﬁer h ψ .
return f χ for epoch ← 1 to num _ of _ epochs do for t ← 1 to environment _ steps _ per _ epoch do
Sample a skill:
z ∼ p ( z ) .
Sample action from the skill-conditioned policy: a t ∼ π ϑ ( a t | s t ,z ) .
Step environment: s t +1 ∼ p
( s t +1 | s t ,a t ) .
Add a transition to the replay buffer: B ← B ∪ { ( s t ,z,a t ,s t +1 ) } .
endfor i ← 1 to train _ steps _ per _ epoch do Sample batch of transitions from the replay buffer: M ∼ B .
Encode states: e t +1 = f
χ ( s t +1 ) .
Compute intrinsic reward: r = log q φ ( z | e t +1 ) − log p
( z ) .
Update Q-functions { Q 1 θ ,Q 2 θ } to minimize the soft Bellman residual.
Update policy π ϑ using the minimum of the two Q-functions.
Update discriminator q φ with MLE.
end endoutput: Learned skill-conditioned policy π ϑ .
B. Additional Experimental Details
This appendix extends 5 with additional plots and commentary.
Figure 8, 9, 10 show maximum, average and minimum return for the three environments.
C. Implementation Details
Conceptually, our skill-discovery algorithm is the same as
DIAYN (Eysenbach et al., 2018). There are, however, a few implementation differences that we empirically found to work just as well.
Below follows a brief rundown of the key implementation details of the algorithm used in the documented experiments.
Two Q-functions Q 1 θ ( s, a ) & Q 2 θ ( s, a ) are used, both with target clones Q 1 θ (cid:48) & Q 2 θ (cid:48) that are continuously updated with polyak averaging.
Both Q-functions attempt to predict the same quantity:
[Equation]
The policy distribution is a mixture of Gaussians with four components.
The policy network predicts the mixture logits, as well as the means and log standard deviations of the Gaussians.
The output is squashed through a hyperbolic tangent function, similar to (Haarnoja et al., 2018).
The policy is updated by climbing the gradient of the minimum of the two Q functions (DDPG-style (Lillicrap et al.)).
[Equation]
This requires that the actions sampled from the policy are differentiable.
Each gaussian component of the mixture is reparametrized the standard way, and the mixture is reparametrized with Gumbel-Softmax (Jang et al., 2016).
Q 1 , 2 θ is trained by descending on the squared temporal difference (TD) errors generated by the minimum of the target networks Q
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Figure] [Figure] m u m
R e
[Figure]
n
[Figure] [Figure]
m R
[Figure]
t u
[Figure]