Rethinking Learning Dynamics in RL using Adversarial Networks
Abstract
Recent years have seen tremendous progress in methods of reinforcement learning.
However, most of these approaches have been trained in a straightforward fashion and are generally not robust to adversity, especially in the meta-RL setting.
To the best of our knowledge, our work is the first to propose an adversarial training regime for Multi-Task Reinforcement Learning, which requires no manual intervention or domain knowledge of the environments.
Our experiments on multiple environments in the Multi-Task Reinforcement learning domain demonstrate that the adversarial process leads to a better exploration of numerous solutions and a deeper understanding of the environment.
We also adapt existing measures of causal attribution to draw insights from the skills learned, facilitating easier re-purposing of skills for adaptation to unseen environments and tasks.
Introduction
Recent years have seen tremendous progress in methods for reinforcement learning with the rise of “Deep Reinforcement Learning”
(DRL; Mnih et al., 2015).
In robotics, DRL holds the promise of automatically learning flexible behaviors end-to-end while dealing with multidimensional data, as mentioned in Arulkumaran et al.
The level has risen to such heights that our algorithms are capable of learning policies that can defeat human professionals in many different games such as Chess, Go (e.g. by AlphaZero (Silver et al., 2017)), and many more complicated games such as Dota-2 (OpenAI, as presented in Berner et al.
robot control (Gu et al., 2016; Lillicrap et al., 2015; Mordatch et al., 2015), and meta-learning (Zoph & Le, 2016).
Despite this recent progress, the predominant paradigm remains to train straightforward algorithms that are not robust to adversarial attacks, as shown in Zhou & Doyle (1998);
Garcıa & Fernández (2015).
In this work, we take a game-theoretic view to do adversarial training since doing so helps learn diverse skills with little supervision.
As such, we draw intuition from the statement that “nothing makes you learn better than coevolving adversaries”.
Previous approaches try to create adversarial examples to make the models better suited for outliers (Pinto et al., 2017b;
Chen et al., 2019), though these require manual intervention.
Our work is novel in that we wish for the model to learn adversarially without manual intervention.
Our proposed algorithm also learns the set of adversarial skills, leading to better exploration and more accessible domain adaptation to unseen tasks.
This is achieved not by changing the tasks or rewards but by allowing the embedding network to learn multiple skill distributions without domain knowledge of the environment or the task at hand.
1.1 Problem Statement
In this work, we consider the Multi-Task Reinforcement Learning or the Meta-RL setting.
The goal here is to teach a given agent to perform a set of tasks ( { τ 1 , τ 2 , ..τ n } ), where in most cases, all tasks garner equal rewards for the goal state.
The problem is interesting since manual intervention is not scalable here, and exploration is more crucial than the standard reinforcement setting.
For instance, it is a sub-optimal solution when the agent learns to solve only one of the tasks, say τ 1 , completely ignoring the other tasks { τ 2 , ..., τ n } .
If all the tasks give proportional rewards, the agent would have solved the environment without the incentive to explore the other tasks.
To alleviate this, we propose the adversarial training regime that brings robustness and an added exploration factor to our policy.
1.2 Contributions
In this section, we present the main contributions of the paper:
A novel adversarial training regime:
The main contribution of our work is an adversarial training regime for on-policy optimization using concepts from game theory literature, enabling us to learn distinct and robust latents for each task.
To the best of our knowledge, our work is the first to propose an adversarial training regime for Multi-Task Reinforcement Learning, which requires no manual intervention or domain knowledge of the environments.
(see Section 3)
Theoretical guarantees: We also provide theoretical guarantees that aid optimization in the adversarial regime (see Appendix C).
Following the results from the above proof will help readers avoid the generic limitations of an adversarial training regime, as highlighted in Appendix E. Furthermore, we also show that the skills learned are mutually exclusive to the task.
This property is helpful to re-purpose the skills for different and unseen tasks, as discussed in the upcoming sections.
A Causal Insight into RL: We propose an efficient framework to approximate the average causal effect of each embedding on the reward obtained for a given task in the on-policy domain.
This facilitates a deeper understanding of the latent variables and helps re-purpose them for other unseen tasks (see Section 4).
With our proposed causal attribution framework, we can also show that the model can learn multiple solutions with the minimum number of distinct skills necessary for the given set of tasks.
(see Appendix G)
Experimental Results: We demonstrate that our method outperforms our baseline in several simulated environments such as PointMass, 2D navigation, and robotic manipulation tasks from MetaWorld.
(see Section 5)
Results on Domain Adaptation: We also show that the model trained using the adversarial training regime leads to better generalization and more accessible domain adaptation to unseen and complex tasks, even in sparse rewards (see Section 5.3).
This result is per the added exploration factor the agent benefits from when trained in the adversarial regime.
This exploration factor has been studied under the context of embedding efficiency of latents.
(see Appendix 2)
Preliminaries
Before we delve into the details of our adversarial training regime-based algorithm, we first outline our terminology, standard reinforcement learning setting, and two-player adversarial games from which our paper is inspired.
2.1 Standard reinforcement learning in MDPs
We study reinforcement learning in Markov Decision Processes (MDPs).
We denote s ∈ R S as the continuous state of the agent; a ∈ R A as the action vector and p ( s t +1 | s t , a t ) as the probability of transitioning from state s t to s t +1 when executing action a t .
Actions are drawn from a policy distribution π θ ( a | s ) , with parameters θ .
This work uses a Gaussian distribution whose mean and diagonal covariance are parameterized via a neural network.
At every step, the agent receives a scalar reward r ( s t , a t ) and the goal is to maximize the sum of discounted rewards: E τ π [ (cid:80) ∞ t =
γ t r ( s t , a t )] .
We use TE-PPO (Hausman et al., 2018) as our baseline since it also uses latent skills for learning policy and serves as the perfect baseline to showcase the effect of an adversarial training regime in the Meta-RL setting.
2.2 Two-player adversarial games
The adversarial setting we propose can be expressed as a two-player γ -discounted non-zero-sum
Markov Game.
Instead of learning the policy directly in an adversarial manner, we choose to learn the skills used by the policy in an adversarial manner.
Under this setting, the MDP is similar to [Figure] the previous notion, with an added variable from the adversary – a skill that reduces the average discounted return on a given task.
Player 1 (Encoder + policy) will try to act as a protagonist and solve the given task.
In contrast, Player 2 (Encoder alone) will serve as an adversary and learn skills that best fool the protagonist for the given task.
While Player 2 acts, the policy network is frozen, and only the embedding network is learned.
We call our algorithm Adversarial TE-PPO (or ATE-PPO) and describe our model in more detail in Section 3.
A brief overview of our algorithm is presented in Algorithm 1.
As discussed in the subsequent section, our end-objective function works in the minimax domain and differentiates our work from TE-PPO.
Figure 1 depicts the high-level working of TE-PPO and ATE-PPO, respectively.
We encourage readers to read the respective paper for further details about the TE-PPO algorithm.
Adversarial Training Regime in RL
Our approach, similar to the setup of TE-PPO, and comprised of three different networks which are trained in an adversarial training regime: an Encoder network ( E ), a Policy network ( π ), and an Inference network ( I ).
The Encoder tries to learn a skill embedding – something unique for each task at hand, and the Policy network uses this embedding to select actions maximizing its reward.
The adversarial modeling framework is most straightforward when the models are multilayer perceptrons.
Instead of training two different sets of agents/policies in an adversarial fashion, we do so at the skill/encoder level.
We will first learn the encoder parameters that provide the “Worst
Faithful Embedding” - skills that reduce the discounted rewards for the given policy and minimize mutual information between the skills and one-hot encoding of tasks.
By doing so, any spurious correlations between skills would disappear since we are learning for the worst case.
We then learn the agent initialized at the given “Worst Faithful Embedding” to learn policies immune to adversarial perturbations.
Our mathematical formulation of the above intuition would follow along, as shown.
To learn the skill embedding z , we represent a mapping between the one-hot encoded task embedding τ to a continuous skill space z as z =
E ( τ ; θ e ) ; where E is a differentiable function represented by a multilayer perceptron with parameters θ e and z is the skill required for achieving task τ .
To take advantage of these skill embeddings, we define a second multilayer perceptron π ( z ; ϕ π ) that helps the agent learn the actions to take and maximize its reward.
Similar to Hausman et al.
, we also allow the encoder to learn in this step to facilitate learning optimal skills for the given task.
To achieve this training regime, our model plays a two-player minimax game where the agent will try to learn a good policy and embedding with a protagonist loss function L pro :
[Equation]
However, the adversary perturbs the embedding slightly to fail at the task, which you cannot detect by computing mutual information between task and embedding.
The adversary will work with a fixed policy and an adversarial loss function L adv :
[Equation]
Hence, you are forced to learn a good policy and embedding which work despite these perturbations.
Here p 0 ( s 0 ) is the initial state distribution, α is a weighing term – a trade-off between deceiving the agent network and minimizing the entropy terms denoted by H ( . ) .
Furthermore, the Q-function
Q π ( s, a, τ ) is defined as:
[Equation]
where action a i is drawn from the distribution π ( .
| s, τ ) , and the new state is
s i +1 ∼ p ( s i +1 | a i , s i ) .
Note that α 0 is a constant and is treated as a hyperparameter.
The first half of the equation defined by (cid:80) ∞ i =
γ i r τ ( s i , a i ) is the discounted expected returns and is common to our generic reinforcement learning algorithms.
The latter is an entropy regularization term defined by (cid:80) ∞ i =
γ i H [ π ( a i | s i , τ )] which is conventionally applied to many policy gradient schemes, with the critical difference that it takes into account not only the entropy term of the current but also future actions.
This approach has been used before by Hausman et al.
(2018), but with a different objective function and trained in a straightforward reinforcement learning fashion, without any adversarial training dynamics.
To apply this entropy regularization to our setting of latent variables (skill embeddings), some extra mathematical rigor is required.
Borrowing from the toolkit of variational inference (Agakov, 2004), we can construct a lower bound on the entropy term from Equation 3 as (see Appendix B) such that:
[Equation]
where,
[Equation]
Here, s i +1 ∼ p ( s i +1 | a i , s i ) , and (cid:98) r
( s i , a i , z, τ ) = (cid:2) r t ( s i , a [Equation] +
[Equation] log[ I ( z | a i , s [Equation] ] +
[Equation] [Equation] ] (cid:3) .
Note that this is where the inference network I comes into play.
Furthermore, H is the history of states saved in the buffer used for the inference.
From the above equation, we simplify our objective function as follows and set α 1 as α , where we denote Mutual Information by I , and Jensen Shannon
Divergence by J SD :
[Equation]
Although, we use the objective presented in Equation 4 to show theoretical guarantees (see Appendix C), we simplify the equation further, making the implementation tractable (see Appendix D).
In our implementation, we use the modified objective presented in Appendix D in an interleaved fashion.
This is similar to adversarial networks in the on-policy setting.
Furthermore, we would also like to extend this training regime to obtain a data-efficient, off-policy algorithm that could be applied to a natural robotic system in the future.
(see Appendix A)
Causal Perspective into RL
Using latent embeddings as skills can aid the study of the causality of the skill upon the task at hand.
In this work, we propose a simple framework to compute the Average Causal Attribution of the skills on the task: the average difference between potential outcomes under different treatments.
The
Average Causal Effect (ACE) of a variable x on another random variable y is formally defined as:
[Equation]
The ACE requires the computation of two quantities: the interventional expectation ( E [ y | do ( x i = α )] ) and the baseline ( baseline x i ).
Here, do corresponds to the do-operation (Pearl, 2009) of fixing a variable x
i to a specific value α .
Input: Data sampling mechanism T , Data D , number of episodes N , Encoder Networks E ,
Policy Network π , Number of Adversarial steps A , and Number of Protagonist steps P .
{+n ← 
+}
; τ i ← OneHot ( i )
▷ Create one-hot vectors of size k τ ← { ττ 2 ...
τ k } ▷ Create Task Embedding while n < N do n ← n + 1 ; {T
T 2 ...
T k } ← T ( D ) ▷ Sample k tasks from data distribution.
a ← 0 ; p ← 0 ; while a < A do a ← a + 1 ; min E L adv
▷ Learn adversarial skills keeping policy fixed endwhile p < P do p ← p + 1 ; min E ,π L pro ▷ Learn protagonist skills learning both policy and encoder end end
Interventional Expectation The interventional expectation is a function of x i as all other variables are marginalized.
By definition:
[Equation]
Naively, evaluating the interventional expectation would involve sampling all other input features from the empirical distribution, keeping feature X i = α , and then averaging the output values.
However, this computation assumes that the input features do not cause each other and would be time-consuming to run for the entire training data.
Furthermore, we also assume that the features are d-separated: given an intervention on a particular variable, the probability distribution of all other neurons does not change, i.e. p ( x j | do ( x i = α ))
=
p ( x j ) , if i ̸ = j .
Since the latent learned by our models are time-agnostic, our assumption is justifiable.
To make the computation feasible, we make use of first-order Taylor expansion of the causal mechanism f
( x | do ( x i = α )) around the mean vector µ = [ µ 1 , µ 2 , · · · , µ k ] T is given by:
[Equation]
Taking expectations on both sides (marginalizing over all other input neurons), we arrive at the
Interventional expectation:
[Equation]
Where f ( . ) is the average reward for a given task, x is the input vector or the latent embedding, and µ is the mean of input vectors when x i = α .
Note that the first-order term vanishes since E ( x | x i = α ) = µ .
This is extremely important since computing the gradient through a step in a reinforcement learning environment is impossible since the transition function is generally nondifferentiable.
To make it differentiable, additional approximations and workarounds would be needed to achieve the same.
Hence, we do not extend the same Taylor approximation to the second-order or higher for the same reason.
Such approximations of deep non-linear neural networks via Taylor’s expansion have been explored in the context of causality (Chattopadhyay et al., 2019).
However, their overall goal was different from the reinforcement learning domain.
Baseline An ideal baseline would be any point along the decision boundary where predictions are neutral.
However, as showed by the work performed by Kindermans et al.
(2019), when the reference baseline is fixed to a specific value, attribution methods are not affine invariant.
Instead, we define baseline as:
[Equation]
Rationally, the baseline is defined as E y
[ y | do ( x i = (cid:98) x i )] , which includes all values of the input feature.
If the expected value y is constant for all possible intervention values α , the baseline would also be the same constant and result in the causal attribution ACE ydo ( x i = α ) = 0 .
Furthermore, the L1-norm of the causal attribution – E x i [ | E y
[ y | do ( x i = α )] | ] could serve as an metric of “ Average Importance ” of a given attribute.
This importance metric is then normalized for easier interpretation.
Note that the proposed framework is model-agnostic and can be adapted to similar models which work with latents (TE-PPO and ATE-PPO in this paper).
This paper’s causal attributions have been computed over the agent trained on ATE-PPO.
Although the approximation is very coarse, we can still obtain various insights from the proposed framework.
(see Appendix G)
Experimental Results and Discussions
We evaluate our approach in two domains in simulation: simple environments such as PointMass task, 2-D navigation task, and a set of challenging robot manipulation tasks from Meta-World.
We consider the Gaussian embedding space for all experiments for both algorithms (ATE-PPO and TE-PPO).
5.1 Learning versatile skills
We highlight this property of learning versatile skills from our experiments on relatively more straightforward environments such as the PointMass environment and 2-D Navigation environment.
PointMass Environment Similar to Haarnoja et al.
(2017), we present a didactic example of multi-goal PointMass tasks demonstrating the variability of solutions that our method can discover.
In this experiment, we consider a case where four goals are located around the initial location, and each is equally important to the agent.
This leads to a situation where multiple optimal policies exist for a single task.
In addition, this task is challenging due to the sparsity of the rewards – as soon as one solution is discovered, it becomes exceedingly difficult to keep exploring other goals.
Due to these challenges, most existing DRL approaches would be content with finding a single solution.
Furthermore, even if a standard policy gradient approach discovered multiple goals, it would have no incentive to represent various solutions.
However, with the added goal to maximize H ( z ) , our
ATE-PPO algorithm should lead to better exploration and learning of different skills.
Our proposed approach outperforms the baseline, as depicted in Table 1.
Figure 2 expresses more insights into the learning trajectory and the final agent behavior.
[Figure]
[Figure]
[Figure]
To better understand the skill learned by our model, we use the causal framework proposed in
Section 4 to compute the average causal attribution of each feature of the skill vector.
The average importance of each component is shown in Figure 3.
This result is interesting because the feature z 2 consistently does not contribute to the agent’s behavior.
Although the latent space is ample, and the model has sufficient capacity, our policy network can learn only the minimum number of skills required to solve the task.
[Figure] [Figure]
2-D Navigation Environment Furthermore, we also experiment on the 2-D navigation task (Finn et al., 2017) with a slight modification.
At each time step, the 2D agent takes action (its velocity, clipped in [ − 0 .
and receives a penalty equal to its L2 distance to the goal position (i.e., the reward is “ − distance”).
However, we restrict the agent’s movement to a small corridor at the beginning, so the vertical state space is clipped between [ − 0 . 2 , 0 . 2] .
In this paper, we sample three tasks to solve in the current experimental setup.
The agent starts from (0 , 0) , and the goals of the tasks are to reach respective points in the Cartesian plane: (1 , 0) , (0 . 6 , 1) , and (0 .
6 , − 1) denoted by various colors in Figure 4.
We find that the ATE-PPO model can successfully learn the task and achieve better performance than the TE-PPO algorithm, as depicted in Table 1.
We notice that our ATE-PPO model successfully learns skills that are diverse in the future and not the present.
Figure 4 shows the resulting trajectories by the agent trained on TE-PPO and ATE-PPO, respectively.
Furthermore, in more detail, we discuss the ACE plots in Appendix G.2.
Our proposed approach outperforms the baseline, as depicted in Table 1.
5.2 Learning optimal representations
Next, we evaluate whether we can learn better representations with the help of the adversarial training regime.
We create a pool of tasks from Meta-World with an overlapping skill set for most tasks in this problem.
In our experiment, we set this overlapping skill as the act of “pushing”.
With this constraint, we create an environment MT5 by selecting five tasks for training: Push, Open window,
Close window, Open drawer, and Close drawer.
The multi-task evaluation tests the ability to learn multiple tasks simultaneously without accounting for generalization to new tasks.
This task helps evaluate the efficiency of the skills learned by the model.
ATE-PPO significantly improves TE-PPO’s performance without significant hyperparameter tuning and using common network architecture across both experiments.
TE-PPO achieves an average success rate of 0 .
2 ± 0 .
01 , while our ATE-PPO achieves an average success rate of 0 .
41 ± 0 .
We summarize more detailed results of the causal analysis of latent in Appendix G.3.
The above illustration shows that the adversarial training regime does offer a boost in performance in the MT5 environment.
Next, we evaluate whether our model can adapt to a new unseen task with minimum adaptation steps.
We use our pre-trained model from the previous section for this task, which was trained on the MT5 environment.
We selected a diverse set of tasks for this experiment with varying degrees of domain shift:
• Push Wall: The goal of this task is similar to the initial Push task from MT5, but we now also have to bypass a wall to reach the goal.
• Coffee Button:
The goal of this task is to press a button on the coffee machine.
We randomize the position of the coffee machine to avoid overfitting.
Push Back: The goal of this task is to push the same object backward, similar to the Push task from MT5.
We also randomize puck positions to avoid overfitting.
• Faucet Open: The goal of the task is to turn on the faucet.
Specifically, we would like to rotate the faucet counter-clockwise and randomize faucet positions to avoid overfitting.
We run the agent for all the above domain adaptation tasks for ten epochs.
Although the domainadapted code is not sufficiently trained to solve the task perfectly, our pre-trained ATE-PPO model outperforms the TE-PPO across most target tasks (if not, competitive), as shown in Table 1.
Furthermore, we run both models in the standard reinforcement learning setting without any adversarial training regime for better comparison.
Related Works
[Table]
This section describes prior studies broadly related to the adversarial training regime of skills.
Below, we divide prior research under two prominent subheadings: (i) Adversarial training regime and (ii) Multi-Task reinforcement learning.
6.1 Adversarial Training Regime
In reinforcement learning, learned policies should be robust to uncertainty and parameter variation to ensure predictable behavior.
Furthermore, learning policies should employ safe and effective exploration with improved sample efficiency to reduce the risk of costly failure.
These issues have long been recognized and studied in reinforcement learning (Zhou & Doyle, 1998; Garcıa & Fernández, 2015).
However, these issues are exacerbated in deep RL using neural networks, which, while more expressible and flexible, often require more data to train and produce potentially unstable policies.
In Pinto et al.
(2017a), two supervised agents were trained, with one acting as an adversary for self-supervised learning, which showed an improvement in robot grasping.
Other adversarial multiplayer approaches have been proposed, including Heinrich & Silver (2016).
However, these approaches require the training of two agents.
Inspired by these, Heinrich & Silver (2016) proposed an approach to learning two policies for a single agent, one as a protagonist and the other as an adversary.
However, this requires hard-coding parts of the agent where an adversarial force is applied to fool the model.
Although these approaches lead to a more robust model, our goal in this paper is slightly different.
Instead, our goal is to propose an adversarial training regime without significant hard coding.
We perform the adversarial training regime at the skill level, allowing better exploration.
Since the same policy network is used in both cases, the policy understands the effect of wrong actions, increasing understanding of the environment due to the increased exploration of the state space.
6.2 Multi-Task reinforcement learning
Learning multiple tasks at once in reinforcement learning is quite common with the intuition that more tasks would suggest higher diversity and hence better generalization in downstream tasks.
This property is best showcased in the work (Eysenbach et al., 2018), where they learn diverse skills without any reward function.
Furthermore, sequential learning and the need to retain previously known skills have always been a focus (Rusu et al., 2016; Kirkpatrick et al., 2017).
In the space of multi-task reinforcement learning with neural networks, Teh et al.
(2017) proposed a framework that allows sharing knowledge across tasks via a task agnostic prior.
Similarly, Cabi et al.
(2017) uses off-policy learning to learn about many different tasks while following a primary task.
Other approaches (Devin et al., 2017; Denil et al., 2017) propose architectures that can be reconfigured easily to solve various tasks.
Subsequently, Finn et al.
(2017) uses meta-learning to acquire skills that can be fine-tuned effectively.
The use of latent variables and entropy constraints to induce diverse skills has been considered before (Daniel et al., 2012;
End et al., 2017) albeit in a different framework and without using neural network function approximators.
Further, Konidaris & Barto (2007) used the options framework to learn transferable options using the so-called agent space.
Inspired by these ideas, Hausman et al.
(2018) introduces a skill embedding learning method that uses deep reinforcement learning techniques and can concisely represent and reuse skills.
Their works draws on a connection between entropyregularized reinforcement learning and variational inference literature (Todorov, 2008; Toussaint, 2009; Neumann et al., 2011; Levine & Koltun, 2013;
Rawlik et al., 2013; Fox et al., 2015).
The notion of latent variables in policies has been previously explored by works such as controllers (Heess et al., 2016), and options (Bacon et al., 2017).
The auxiliary variable perspective introduces an informative-theoretic regularizer that helps the inference model produce more versatile behaviors.
Learning these versatile skills has been previously explored by Haarnoja et al.
(2017), which learns an energy-based, maximum entropy policy via a Q-learning algorithm, and Schulman et al.
(2017) which uses an entropy regularized reinforcement learning policy.
Hausman et al.
(2018) uses a similar entropy-regularized reinforcement learning and latent variables but differs in the algorithmic framework.
Conclusion
We present an adversarial training regime that learns manipulation skills continuously parameterized in a skill embedding space.
Our algorithm takes advantage of these skills and creates an adversary at the skill level instead of the policy level.
The skills are learned by taking advantage of the increased explorability of known variables and exploiting a connection between reinforcement learning, variational inference, adversarial networks, and minimax algorithm from game theory.
We derived an adversarial training regime for reinforcement learning and demonstrated its robustness in multiple environments.
Primarily, we showed that our model could learn skills that diverge in the future rather than the present.
We also introduce a methodology for computing ACE, which would help re-purposing skills and using them efficiently in o.o.d environments.
Our experiments with the help of ACE also indicate that even in the presence of enough capacity, our model is capable of discovering multiple solutions and learning the minimum number of distinct skills and latents necessary to solve the given tasks.
Furthermore, the learned skills are more adaptable and make for better cross-domain adaptation than the traditional reinforcement learning approach.
We believe the proposed framework is more ubiquitous than just reinforcement learning and could be adapted to other domains such as supervised learning, representation learning, and many others.
Reproducibility Statement
In this paper, we work with three different datasets, which are all open-sourced.
Furthermore, we experiment with two different models - TE-PPO and ATE-PPO.
The TE-PPO model was run after reproducing from their open-source code 1 .
Additional details about setting up these models, and the hyperparameters are available in Appendix H.
Our source code is made available for additional reference 2 .
A Extending to Off-Policy Setting
In this section, estimate the discounted sums from Equation 4 from previously gathered data by learning a Q-value function, yielding an off-policy algorithm.
Similar to Hausman et al.
(2018), we assume the availability of a replay buffer B (containing full trajectory execution traces including states, actions, task id, and reward) that is inherently filled during training.
In conjunction with these trajectory traces, we also store the probabilities of each selected action and denote them with the behavior policy probability b ( a | z, s, τ ) and the behavior probabilities of the embedding b ( z | τ ) .
Given this replay data, we formulate the off-policy perspective of our algorithm.
We start with the notion of a lower-bound Q-function that depends on both states s and a and is conditioned on both the embedding z and the task id τ .
To learn a parametric representation of Q φπ , we make use of the Retrace algorithm (Munos et al., 2016), which quickly allows us to propagate entropy augmented rewards across multiple time steps while minimizing the bias of the algorithm relying on the parametric Q-function.
Formally, we fit Q φπ by minimizing the squared loss:
[Equation]
where
[Equation]
wherein,
[Equation]
We compute the terms contained in (cid:98) r by using r τ and z from the replay buffer and re-compute the (cross-)entropy terms.
Here, φ ′ denotes the parameters of a target Q-network (Mnih et al., 2015) that we occasionally copy from the current estimate φ and c k are the per-step importance weights.
Further, we bootstrap the infinite sum after N -steps with E π (cid:104) Q πφ ′
( s N , .
; z N , τ ) (cid:105) instead of introducing a λ parameter as in the original paper.
Equipped with this Q-function, we can update the policy and embedding network parameters without requiring additional environment interactions by optimizing the same objective as Eq. 4.
We highlight that the above derivation and steps also hold when the task id is constant, i.e., for the regular reinforcement learning setting rather than the meta-reinforcement learning setting.
The following sections in Appendix C present a more detailed theoretical analysis of our adversarial network training regime, essentially showing that the training criterion converges appropriately given the necessary conditions are guaranteed.
In practice, we cannot sequentially train the two networks.
We must implement the game using an iterative back and forward approach.
Optimizing E to completion would lead to overfitting, and the Encoder would not learn the adequate skill embedding to be used by the policy network.
Instead, we alternate between k steps of optimizing the protagonist and one stage of optimizing the adversary.
This results in the protagonist being maintained near its optimal solution, so long as adversarial skills change slowly enough.
It might seem counterintuitive to minimize the discounted rewards by the embedding function.
However, as long as the hyperparameter α is set correctly, the Jensen-Shannon Divergence objective prioritizes the discounted rewards criterion.
With this assumption, we can ensure that minimizing the expected returns would merely act as an implicit noise (Hjelm et al., 2018).
B Variational Bound Inference.
We borrow ideas from variational inference literature to introduce an information-theoretical regularization that encourages versatile skills.
In particular, we present a lower entropy of marginal entropy
H [ p ( x )] , which will prove helpful when applied to our objective function from Sec. 3.
Note that this section is not novel and is only provided to help readers better understand our proposed methodology.
Proposition B.1.
The lower bound on the marginal entropy H [ p ( x )] corresponds to:
[Equation]
[Equation]
where q ( z | x ) is the variational posterior
Proof.
[Equation]
From the above Equation
[Equation]
we can construct a lower bound for our entropy term H [ π ( a i | s i , t )] as follows:
[Equation]
Where CE is cross entropy.
To simplify the second part of the result from Equation
[Equation]
we simplify it as follows:
[Equation] [Equation]
Since, the skill embedding is conditionally independent of the state of the agent given task t , we can simplify p ( z | s, t ) as p ( z | t ) .
Similarly, since the action a is conditionally independent of the task t given the latent z , we can simplify π ( a | s, t, z ) as π ( a | s, z ) .
This is possible since z carries all the necessary information from t , which is required to solve the task.
With the above simplifications,
Equation 14 further simplifies to:
[Equation]
Note that q ( z | a, s, t ) is the variational inference distribution we are free to choose.
Since q ( z | a, s, t ) is intractable [Equation] we resort to a sample-based evaluation of the Cross-Entropy term.
This bound holds for any q .
Similar to Hausman et al.
[Equation] we avoid conditioning q on task t to ensure that a given trajectory alone will allow us to identify its skill embedding.
Substituting the results from
Equation 14, Equation [Equation] we can rewrite Equation 13 as:
[Equation]
C Theoretical Results
We will first prove that the minimax game has a global optimum for a given policy network and a given embedding network in the case of an on-policy setting, where the optimal policy is already known to us.
This proof is presented in Appendix C.1.
These proofs guarantee the working of our algorithm and set the necessary conditions for the model to train appropriately.
C.1 On-Policy Optimality
Proposition C.1.
For a given E , the optimal policy π is
[Equation]
and the bound on Q φπ ( s, a, z, τ ) is going to be such that:
[Equation]
Proof.
The training criterion for the given policy network π , given any encoder E , is to maximize the quantity V ( E , π ) ( −L pro ) from Equation 1.
Our equation now reduces to:
[Equation] [Equation]
Here, we will assume the rewards to be bounded by the range [0 , R max ] .
Note that any reward function, sparse or otherwise, can be transformed to the above range and subject to the same proof.
Furthermore, we assume perfect optimality of the inference network q , and the resulting Cross
Entropy loss can be omitted since it would be 0.
With the use of theory from Kullback–Leibler divergence and Shannon Entropy, we know that H ( x ) ≤ log | x | .
In our case, the entropy H [ π ( a | s, z )] can be bounded to log | a t | , where a τ is the action space of the given task.
For simplification, we approximate | a t | to | a max | , where a max denotes the action space with maximum cardinality.
We can further simplify the equation above as follows:
[Equation]
The Equation 4 can now be reformulated as:
[Equation]
Proposition C.2.
The global minimum of the training criterion C ( E ) is achieved if and only if
[Equation]
At this point, C ( E ) achieves a minimum value bounded by C ( E ) < 0 and is not a trivial solution where the embedding function is an identity function.
Furthermore, the property that z and t are mutually exclusive is held.
Proof.
Since the Jensen–Shannon divergence between two distributions is always non-negative and zero if they are equal.
Since we want to minimize Equation 19, we escape the trivial solution where the embedding network is an identity function.
Note that, since the input to the embedding network is a one-hot encoded embedding of the task i
d, it satisfies the condition of the sharp peak and is uniform in terms of tasks and skills.
For the time being, let us assume α is high enough and the J SD objective is being tuned rather than the term that helps deceive the agent network.
In this scenario, we have two unique cases: (i) when z ∼ τ , and when (ii) z ̸∼ t .
Note that the Jensen-Shannon divergence would be equal to 0 in the first case and positive in the second.
Since our goal is to minimize the objective, the model would converge towards the goal where z ̸∼ τ , and escape the trivial solution of the identity function.
However, to ensure that the J SD objective is being optimized with higher precedence, we set α such that:
[Equation]
[Equation]
Since we have already shown that the optimal solution is when z ̸∼ τ , or when z and τ are mutually exclusive, and JSD would be at its maximum value of 1.
This mutual exclusivity property is essential since we would like the same skill to be shared across tasks, as long as the tasks are based on similar structures.
For instance, turning a doorknob or screwing a cap on a bottle involves identical skills.
Using these fundamental skills to improve task structure in meta-learning would facilitate more accessible posterior adaptation and causal inference.
With the assurance that our intuition is following the mathematical rigor, we can select α such that:
[Equation]
Provided this condition is met, the training criterion C ( E ) achieves a minimum value bounded such that C ( E ) < 0 .
We select hyperparameters ( α 3 , α ) such that the following condition is satisfied.
D Tractable optimization of objective
Here, we show that the equation presented in Equation 4 might not be the ideal form to compute the loss.
This is because the computation of the Jensen-Shanon Divergence forces the shape of both skills z and task τ to be the same.
Furthermore, since z is derived from τ , it is not entirely straightforward how to compute H ( τ | z ) .
To subvert these issues, we simplify the above equation as follows.
Recall,
[Equation]
Note that our original objective equation does have the L.H.S. term.
Thus, we simplify the objective to:
We simplify H ( τ | z ) as H ( τ ) − H ( z ) + H ( z | τ ) .
Note that the H ( τ ) term cancels out and we are left with 2 α ( H ( z ) − H ( z | τ )) as our second term in Equation 4.
This simplification is what we use while computing, but we use the Jensen-Shannon Divergence simplification from Equation 4 to complete our proof of convergence of the model.
[Equation]
Since α is a hyperparameter of our choice, we can assign 2 α as α ′ and simplify the equation as follows:
[Equation]
The above equation is what we use to make the computation tractable.
E Limitations of Adversarial Training Regime
As discussed briefly in this section, working in an adversarial training regime does pose its own set of limitations.
If the hyperparameters are not carefully set, it might become unstable and inconsistent when the adversary completely nullifies the learning from the protagonist.
This could occur if the conditions highlighted in Appendix C are not met.
This could also happen if the number of iterations the adversary is trained is much greater than the number of iterations of the protagonist forcing the final weights to be closer to an agent trying to minimize the rewards instead of maximizing.
Furthermore, setting α in Equation 2 very low will give more importance to learning skills that fool the agent but are not necessarily diverse and distinct.
Although our adversarial framework does outperform the traditional training regime in multiple environments, it isn’t easy to support the need for such a framework due to its inherent counter-intuitiveness.
It requires more theoretical studies to understand this behavior.
F Embedding Efficiency
We want our learned skills to be efficient and diverse in the latent embedding space.
To evaluate the efficiency of our embeddings, we compute the diversity of these latents as the volume parallelopiped.
Suppose our embedding vector is z ∼ R n .
If the volume the skills encompass is higher, more volume of the R n space it covers is higher, and we can better adapt to hierarchical skills with the same embedding space.
With the given intuition, we compute the efficiency of the embedding as the square of the volume parallelopiped by the embedding.
Appendix F.1 discusses our approach to computing this efficiency in more detail.
Our experiments show that the latent skills learned from the ATE-PPO algorithm are more efficient and diverse than the latent skills learned from the TE-PPO algorithm.
We summarize our findings in Table 2.
Furthermore, when computing the volume of the space it covers, we can scale each latent feature by a constant, if needed, without any loss of information.
[Table]
F.1 Computing Volume of latent embedding
Proposition F.1.
Let Π be an m -dimensional parallelotope defined by edge vectors B = { v 1 , v 2 , ..., v m } , where v i ∈ R n for n ≥ m .
That is, we are looking at an m -dimensional parallelotope embedded inside n -dimensional space.
Suppose A is the m × n matrix with row vectors B given by:
[Equation]
 Then the m -dimensional volume of the paralleletope is given by:
[Equation]
Proof.
Note that AA T is an m × m square matrix.
Suppose that m = 1 , then:
[Equation]
so the proposition holds for m = 1 .
From this base equation, we prove the above theorem by induction.
Now, we induct on m .
Let us assume the proposition holds for m ′ such m ′ ≥ 1 .
If we can also prove that the proposition holds for m ′
+1 , we would have proved the above theorem.
Letting A m ′ denote the matrix containing rows v 1 to v m ′ , we can write A =
A m ′ +1 as:
[Equation]
We may decompose v m ′
+1 orthogonally as:
[Equation]
where v ⊥ lies in the orthogonally complement of the base (i.e., the height of our parallelepipe), and v ⊥ · v i = 0 , ∀ 1 ≤ i ≤ m ′ .
Furthermore, v ∥ must be in the span of vectors { v 1 , v 2 , ..., v m ′ } , such that:
[Equation]
We apply a sequence of elementary row operations to A , adding a multiple − c i of row i to row m ′
+ 1 , ∀ 1 ≤ i ≤ m ′ .
We can then write the resulting matrix B as:
[Equation]
Each E i is an elementary matrix adding a multiple of one row to another.
Notice that the above operation corresponds to shearing the parallelotope so that the last edge is perpendicular to the base.
We see that these operations do not change the determinant as:
[Equation]
Through block multiplication, we can obtain BB T as follows:
[Equation]
Furthermore, notice that
[Equation]
Therefore, we have
[Equation]
Taking the determinant, we can simplify det( BB T ) as
[Equation]
By definition, ∥ v ⊥ ∥ is the height of the parallelotope, and by the induction hypothesis, det( A m ′ , A Tm ′ ) is the square of the base.
Therefore, we have proved the above theorem by induction.
G Analysis of Skills learned
In this section, we discuss a few additional results, along with a study of the effect of each latent feature on the agent’s behavior.
G.1 PointMass Environment
[Figure]
[Figure]
C a u s a l A tt r i b u t i o n s ( A C E ) z
[Figure]
C a u s a l A tt r i b u t i o n s ( A C E ) z
1 z 2 z 3 z(c) Goal: Top 4 3 2 1 0 1 2 3 4 Intervention Value
(alpha)
C a u s a l A tt r i b u t i o n s ( A C E ) z
1 z 2 z 3 z(d)
Goal:
Bottom notice
To better understand the effect of each feature on the behavior of the model we use an input perturbation method.
Here we fix all features to a given task: say Top but vary a given feature within the skill to study the change in the agent’s behavior.
Figure 11 depicts the results from our input perturbation experiment.
[-4 3 2 1 0 1 2 3 4 Intervention Value (alpha)
-][Figure]
C a u s a l A tt r i b u t i o n s ( A C E ) z
1 z 2 z 3 z(a)
Goal: Right 4 3 2 1 0 1 2 3 4 Intervention Value (alpha)
C a u s a l A tt r i b u t i o n s ( A C E ) z
1 z 2 z 3 z(b) Goal: Top 4 3 2 1 0 1 2 3 4 Intervention Value (alpha)
C a u s a l A tt r i b u t i o n s ( A C E ) z
1 z 2 z 3 z
[Figure]
[Figure]
G.2 2-D Navigation Environment
The average importance of each feature is shown in Figure 8.
From the ACE plots, we can successfully assert that feature z 2 is an auxiliary variable and does not play any significant role in the model’s behavior since the ACE value is close to 0.
Furthermore, we notice that the z 3 and z 4 causal analysis are pretty similar, and z 1 seems to be an essential feature in the pool of latent features.
To better understand the effect of each feature on the behavior of the model, we use an input perturbation method.
Here we fix all features to a given task: say Top, but vary a given feature within the skill to study the change in the agent’s behavior.
Figure 12 depicts the results from our input perturbation experiment.
G.3 Meta-World(MT5) Environment
4 3 2 1 0 1 2 3 4 Intervention [Figure] (alpha) 0.
C a u s a l A tt r i b u t i o n s ( A C E ) z
1 z 2 z 3 z
[Figure]
C a u s a l A tt r i b u t i o n s (
[Figure] C [Figure]
The average importance of each feature is shown in Figure 10.
[Figure]
[Figure]
I m p o r t a n ce
Average importance zzzz
From the ACE plots, we can successfully assert that feature z 2 is an auxiliary variable and does not play any significant role in the model’s behavior since the ACE value is close to 0.
Furthermore, we notice that the z 1 variable is significant for tasks that involve closing, whereas z 2 and z 3 variable are significant when tasks involve the action of opening.
Performing an input-perturbation method, similar to previous sections might not be feasible for the
MetaWorld environment, as it involves much higher complexity of observation and action space for a simple visualization. Top Right Bottom
{+Left 4 3 2 1 0 1 2 3 4 
+}
Top
Right
Bottom
Left 4 3 2 1 0 1 2 3 4
Top
Right
Bottom
{+Left 4 3 2 1 0 1 2 3 4 
+}
Top
Right
Bottom
Left
Goal: Left
Goal: Right
Goal: Top
Goal: Bottom
Top
Right
Bottom
Left 4 3 2 1 0 1 2 3 4
Top
Right
Bottom
Left 4 3 2 1 0 1 2 3 4
Top
Right
Bottom
Left {+4 3 2 1 0 1 2 3 4 
+}
Top
Right
Bottom
Left [-4 3 2 1 0 1 2 3 4 
-]Goal: Left Goal: Right
Goal: Top
Goal: Bottom
Top
Right
[-Bottom Left 4 3 2 1 0 1 2 3 4 
-][Table]
Top
Right
Bottom
Left 4 3 2 1 0 1 2 3 4
Top
Right
Bottom
Left 4 3 2 1 0 1 2 3 4
Top
Right
[-Left 4 3 2 1 0 1 2 3 4 
-]
Bottom
[-Left 4 3 2 1 0 1 2 3 4 
-]
Left
Goal: Left Goal: Right
Goal: Top
Goal: Bottom
Top
Right
Bottom
Left 4 3 2 1 0 1 2 3 4
Top
Right
Bottom
Left 4 3 2 1 0 1 2 3 4
Top
Right
Bottom
Left 4 3 2 1 0 1 2 3 4
Top
Right
Bottom
Left
Goal: Left Goal: Right
Goal: Top
Goal: Bottom
Top Right Bottom 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1. Top Right Bottom 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.
Top Right Bottom
Goal: Top
Goal: Bottom
(a) The effect of z 1 on the behavior of the environment. As we can notice from the above plots, z 1 does not seem to have a significant effect on the behavior of the agent. We hypothesize that the z 1 feature could have low importance and unnecessary, or could be a basic skill used across all tasks. This would explain why perturbation of the z 1 feature does not lead to any difference in behavior, i.e. not a discriminating feature across tasks.
Top Right Bottom 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1. Top Right Bottom 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.
Top Right Bottom
[Table]
(b) The effect of z 2 on the behavior of the environment.
As we can notice from the above plots, z 2 seems to have little to no effect on the behavior of the agent.
This feature seems to play a very similar role to z 1 as discussed earlier.
Top Right Bottom 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.
Top
[Table]
Bottom 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1. Top [Table]
Bottom
[Figure]
Top Right Bottom 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.
Top Right Bottom 0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.
[Table]
[Figure]
H Model and Hyperparameters
This section discusses the model and hyperparameters used in our experiments.
Table 3 depicts the hyperparameters we used for training our TE-PPO and ATE-PPO algorithm on the PointMass environment.
Similarly, Table 4 depicts the hyperparameters we used for training our TE-PPO and ATE-PPO algorithms in the 2-D navigation environment.
Finally, Table 5 depicts the hyperparameters we used for training our TE-PPO and ATE-PPO algorithm in the Meta-World (MT5) environment.
[Table]
Algorithm
[Table]
rameters
[Table]
sizes (20 , 20) (20 , 20) enc
[Table]
Inference hidden sizes (20 , 20) (20 , 20) inf
[Table]
Policy hidden sizes (32 , 16) (32 , 16) pol_hidden_sizes Activation function of hidden layers tanh
[Table]
Likelihood ratio
[Table]
range
[Table]
Latent dimension
[Table]
Inference window length  inference_window
Embedding maximum standard deviation 0 .0 .embedding_max_std
Policy entropy coefficient 1 e −1 e −policy_ent_coeff
Encoder entropy coefficient 1 e −1 e −enc_ent_coeff Inference
[Table]
coefficient 5 e −5 e −inf_ent_coeff
Optimizer-Specific Hyperparameters
Protagonist mini-batch size  pr_batch_size
Adversary mini-batch size − ad_batch_size
Inference mini-batch
[Table]
[Table] [Table]
[Table] [Table]
sizes (
[Table] [Table]
20) inf
[Table] [Table]
[Table]
16) pol
[Table]
[Table]
[Table] window [Table]
dard
[Table]
embedding
[Table]
std
[Table]
1 e −1 e −policy_ent_coeff Encoder
[Table]
coefficient 1 e −1 e −enc_ent_coeff
Inference entropy coefficient 5 e −5 e −inf_ent_coeff
Optimizer-Specific Hyperparameters
Protagonist mini-batch size  pr_batch_size
Adversary mini-batch size − ad_batch_size
Inference mini-batch size  inf_batch_size
Protagonist learning rate 1 e −5 e −pr_lr
Adversary learning rate − 1 e −ad_lr
Inference learning rate 1 e −5 e −inf_lr
[Table]
rameters
Encoder hidden sizes (20 , 20) (20 , 20) enc_
[Table]
Inference hidden sizes (20 , 20) (20 , 20) inf_hidden_sizes
Policy hidden sizes (32 , 16) (32 , 16) pol_hidden_sizes
Activation function of hidden layers tanh tanh hidden_nonlinearity
Likelihood ratio [Table] lr_clip_range
Latent dimension  latent_length
Inference window length  inference_window
Embedding maximum standard deviation 0 .0 .embedding_max_std
Policy entropy coefficient 2 e −2 e −policy_ent_coeff
Encoder entropy coefficient 2 e −2 e −enc_ent_coeff
Inference entropy coefficient 5 e −5 e −inf_ent_coeff
Optimizer-Specific Hyperparameters
Protagonist mini-batch size  pr_batch_size
Adversary mini-batch size − ad_batch_size
Inference mini-batch size  inf_batch_size
Protagonist learning rate 1 e −5 e −pr_lr
Adversary learning rate − 1 e −ad_lr
Inference learning rate 1 e −5 e −inf_lr