L EARNING A S UBSPACE OF P OLICIES FOR O NLINE A DAPTATION IN R EINFORCEMENT L EARNING
A BSTRACT
Deep Reinforcement Learning (RL) is mainly studied in a setting where the training and the testing environments are similar.
But in many practical applications, these environments may differ.
For instance, in control systems, the robot(s) on which a policy is learned might differ from the robot(s) on which a policy will run.
It can be caused by different internal factors (e.g., calibration issues, system attrition, defective modules) or also by external changes (e.g., weather conditions).
There is a need to develop RL methods that generalize well to variations of the training conditions.
In this article, we consider the simplest yet hard to tackle generalization setting where the test environment is unknown at train time, forcing the agent to adapt to the system’s new dynamics.
This online adaptation process can be computationally expensive (e.g., fine-tuning) and cannot rely on meta-RL techniques since there is just a single train environment.
To do so, we propose an approach where we learn a subspace of policies within the parameter space.
This subspace contains an infinite number of policies that are trained to solve the training environment while having different parameter values.
As a consequence, two policies in that subspace process information differently and exhibit different behaviors when facing variations of the train environment.
Our experiments 1 carried out over a large variety of benchmarks compare our approach with baselines, including diversity-based methods.
In comparison, our approach is simple to tune, does not need any extra component (e.g., discriminator) and learns policies able to gather a high reward on unseen environments.
I NTRODUCTION
In recent years, Deep Reinforcement Learning (RL) has succeeded at solving complex tasks, from defeating humans in board games (Silver et al., 2017) to complex control problems (Peng et al., 2017; Schulman et al., 2017).
It relies on different learning algorithms (e.g., A2C - (Mnih et al., 2016), PPO - (Schulman et al., 2017)).
These methods aim at discovering a policy that maximizes the expected (discounted) cumulative reward received by an agent given a particular environment.
If existing techniques work quite well in the classical setting, considering that the environment at train time and the environment at test time are similar is unrealistic in many practical applications.
As an example, when learning to drive a car, a student learns to drive using a particular car, and under specific weather conditions.
But at test time, we expect the driver to be able to generalize to any new car, new roads, and new weather conditions.
It is critical to consider the generalization issue where one of the challenges is to learn a policy that generalizes and adapts itself to unseen environments .
Different techniques have been proposed in the literature (Section 6) to automatically adapt the learned policy to the test environment.
In the very large majority of works, the model has access to multiple training environments (meta-RL setting).
Therefore, the training algorithm can identify which variations (or invariants) may occur at test time and how to adapt quickly to similar variations.
But this setting may still be unrealistic for concrete applications: for instance, it supposes that the student will learn to drive on multiple cars before getting their driving license.
In this paper, we address a simpler yet harder to tackle generalization setting in which the learning algorithm is trained over one single environment and has to perform well on test environments; preventing us from using meta-RL approaches.
A natural way to attack this setting is to start by learning a single policy using any RL algorithm, and to fine-tune this training policy at test time, over the test environment (See red/blue points in Figure 1a), but this process may be costly in terms of environment interactions.
Very recently, the idea of learning a set of diverse yet effective policies (Kumar et al., 2020b; Osa et al., 2021) has emerged as a way to deal with this adaptation setting.
The intuition is that, if instead of learning one single policy, one learns a set of ’diverse’ policies, then there is a chance that at least one of these policies will perform well over a new dynamics.
The adaptation in that case just consists in selecting the best policy in that set by evaluating each policy over few episodes (K-shot adaptation).
But the way this set of policies is built and the notion of diversity proposed in these methods have a few drawbacks: these models increase diversity by using an additional intrinsic reward which encourages the different policies to generate different distributions of states.
This objective potentially favors the learning of policies that are sub-optimal at train time.
Moreover, these approaches make use of an additional component in the policy architecture (e.g., a discriminator) that may be difficult to tune, particularly considering that, at train time, we do not have access to any test environment and thus cannot rely on validation techniques to tune the extra architecture.
Inspired by recent research on mode connectivity (Benton et al., 2021; Kuditipudi et al., 2019) and by (Wortsman et al., 2021) which aims to learn a subspace of models in the supervised learning setting, we propose to learn a subspace of policies in the parameter space as a solution to the online adaptation in the RL setting (see Figure 1a).
Each particular point in this subspace corresponds to specific parameter values, and thus to a particular policy.
This subspace is learned by adapting a classical RL algorithm (PPO and A2C in our case, see Section 3.3) such that an infinite continuum of policies is learned, each policy having different parameters.
The policies thus capture and process information differently, and react differently to variations of the training environment (see
Figure 1b).
We validate our approach (Section 5) over a large set of reinforcement learning environments and compare it with other existing approaches.
These experiments show that our method is competitive, achieves good results and does not require the use of any additional component of hyper-parameters tuning contrarily to baselines.
S ETTING Reinforcement Learning:
Let us define a state space S and an action space A .
In the RL setting, one has access to a training Markov Decision Process (MDP) denoted M defined by a transition distribution P ( s ′ | s, a ) : S ×
A × S → R + , an initial state distribution P ( i ) ( s ) : S → R + and a reward function r ( s, a ) : S ×
A → R .
A policy is defined as π θ ( a | s ) :
S ×
A → R + , where θ denotes the parameters of the policy.
A trajectory sampled by a policy π θ given a MDP M is denoted τ ∼ π θ ( M ) .
The objective of an RL algorithm is to find a policy that maximizes the expected cumulative (discounted) reward:
[Equation]
where R ( τ ) is the discounted cumulative reward over trajectory τ .
Online adaptation:
We consider the setting where the policy trained over M will be used over another MDP (denoted ¯ M ) that shares the same state and action space as M , but with a different dynamics and/or initial state distribution and/or reward function 2 .
Importantly, ¯ M is unknown at train time, and cannot be used for model selection, making the tuning of hyper-parameters difficult.
Given a trained model, we consider the K-shot adaptation setting where the test phase is decomposed into two stages: a first phase in which the model adapts itself to the new test environment over K episodes, and a second phase in which the adapted model is used to collect the reward.
We thus expect the first phase to be as short as possible (few episodes), corresponding to a fast adaptation to the new environment.
Let us consider that a model π θ generates a sequence of trajectories ¯ τ 1 , ¯ τ 2 , ...., ¯ τ + ∞ over ¯ M , the performance of such a model, is defined as:
[Equation]
which corresponds to the average performance of the policy π θ over ¯ M after K episodes used for adapting the policy.
Note that we are interested in methods that adapt quickly to new a test environment and we will consider small values of K in our experiments.
In the following, for sake of simplicity, K will refer to the number of policies evaluated during adaptation since each policy may be evaluated over more than a single episode when facing stochastic environments.
L EARNING SUBSPACES OF POLICIES
Motivation and Idea:
To illustrate our idea, let us consider a toy example where the train environment contains states with correlated and redundant features, in such a way that multiple subsets of state features can be used to compute good actions to execute.
Traditional RL algorithms will discover one policy π θ ∗ that is optimal w.r.t the environment.
This policy will typically use the state features in a particular way to decide the optimal action at each step.
If some features become noisy (at test time) while, unluckily, π θ ∗ particularly relies on these noisy features, the performance of the policy will drastically drop.
Now, let us consider that, instead of learning just one optimal policy, we also learn a second optimal policy π θ ∗′ , but enforcing θ ∗ ′ to be different than θ ∗ .
This second policy may tend to make use of various features to compute actions.
We thus obtain two policies instead of one, and we have more chances that at least one of these policies is efficient at test time.
Identifying which of these two policies is the best for the test environment (i.e., adaptation) can simply be done by evaluating each policy over few episodes, keeping the best one.
Our model is built on top of this intuition, extending this example to an infinite set of policies and to variable environment dynamics.
Inspired by Wortsman et al.
(2021) proposing to learn a subspace of models for supervised learning, we study the approach of learning a subspace of policies in the parameter space , and the use of such a model for online adaptation in reinforcement learning.
Studying the structure of the parameter space has seen a recent surge of interest through the mode connectivity concept (Benton et al., 2021; Kuditipudi et al., 2019; Wortsman et al., 2021) and obtain good results in generalization, but it has never been involved in the RL setting.
As motivated in the previous paragraph, we expect that, given a variation of the training environment, having access to a subspace of policies that process information differently instead of a single policy will facilitate the adaptation.
As a result, our method is very simple, does not need any extra hyper-parameter tuning and achieves good performance.
3. S UBSPACES OF P OLICIES
Given Θ the space of all possible parameters, a subspace of policies is a subset ¯Θ ⊂ Θ that defines a set of corresponding policies ¯Π
= { π θ } θ ∈ ¯Θ .
Since our objective is to learn such a subspace, we have to rely on a parametric definition of such a subspace and consider ¯Θ as a simplex in Θ .
Let us define N anchor parameter values ¯ θ 1 , ....
¯ θ N ∈ Θ .
We define the Z -space as the set of possible weighted sum of the anchor parameters: Z = (cid:8) z
= ( z 1 , ...
z N ) ∈ [0 , 1] N | (cid:80)
z i = 1 (cid:9) .
The subspace we aim to learn is defined by:
[Equation]
In other words, we aim to learn a convex hull of N vertices in Θ .
Note that policies in this subspace can be obtained by sampling z ∼ p ( z ) uniformly over Z .
The advantages of this approach are: a) the number of parameters of the model can be controlled by choosing the number N of anchor parameters, b) since policies are sharing parameters (instead of learning a set of independent policies), we can expect that the learning will be sample efficient.
Such a subspace is illustrated in Figure 1a through the "pentagon" (i.e., N = 5 ) in which angles correspond to the anchor parameters and the surface corresponds to all the policies in the built subspace.
K-shot adaptation: Given a subspace of policies ¯Θ , different methods can be achieved to find the best policy over the test environment.
For instance, it could be done by optimizing the distribution p ( z ) at test time.
In this article, we use the same yet effective K-shot adaptation technique than
Kumar et al.
(2020b) and Osa et al.
(2021): we sample K episodes using different policies defined by different values of z that are uniformly spread over Z .
In our example, it means that we evaluate policies uniformly distributed within the pentagon to identify a good test policy (blue star).
Note that, when the environment is deterministic, only one episode per value of z needs to be executed to find the best policy, which leads to a very fast adaptation.
3. L EARNING A LGORITHM
Learning a subspace of policies can be done by considering the RL learning problem as maximizing:
[Equation]
Considering that ¯Θ is a convex hull as defined in Equation 3, and using the uniform distribution p ( z ) over Z , the loss function of Equation 4 can be rewritten as:
[Equation]
Maximizing such an objective function over ¯ θ 1 , ....
¯ θ N outputs a (uniform) distribution of policies trained to maximize the reward, all these policies sharing common parameters.
Avoiding subspace collapse:
One possible effect when optimizing L (¯ θ 1 , .... ¯ θ N ) is to reach a solution where all θ k values are similar.
In that case, all the policies would have the same parameters value, and will thus all achieve the same performance at test-time.
Since we want to encourage the policies to process information differently, and following Wortsman et al.
(2021), we encourage the anchor policies to have different parameters.
This is implemented through the use of a regularization term denoted C (¯ θ 1 , .... ¯ θ N ) that measures how much anchor policies are similar in the parameter space.
This auxiliary loss is defined as a pairwise loss between pairs of anchor parameters:
[Equation]
The final optimization loss is then:
[Equation]
where β is an hyper-parameter (see Section 5 for a discussion abot the tuning of this term) that weights the auxiliary term.
Initialize: ¯ θ 1 , ¯ θ 2 , ϕ
(Critic), n batch size 1 for k = 0 , 1 , 2 ...
do
Sample z 1 , ..., z n ∼ U [0 , 1]
Define θ z i ← z
i ¯ θ
+
(1 − z i )¯ θ
Sample trajectories { τ i } n 1 using { π θ zi }
Update ¯ θ 1 and ¯ θ 2 to maximize: 1 n (cid:80) ni =1 (cid:98)
L PPO ( θ z i ) − β cosine 2 (cid:0) ¯ θ 1 , ¯ θ 2 (cid:1)
Up.
ϕ to minimize: 1 n (cid:80) ni =1 (cid:98)
L MSE ( ϕ, z i ) 7 end
3. L INE OF P OLICIES (L O P)
In the case of N = 2 , the subspace of policies corresponds to a simple segment in the parameter space defined by ¯ θ 1 and ¯ θ 2 as extremities.
¯ θ 1 and ¯ θ 2 are combined through a scalar value z ∈ [0; 1] :
[Equation]
Computationally, learning a line of policies 3 is similar to learning a single policy for which the number of parameters is doubled, making this particular case a good trade-off between expressivity and training speed.
It corresponds to the following objective function:
[Equation]
We provide in Algorithm 7 the adapted version of the clipped PPO algorithm (Schulman et al., 2017) for learning a subspace of policies.
In comparison to the classical approach, the batch of trajectories is first acquired by multiple policies sampled following p ( z ) (line 2-3).
Then the PPO objective is optimized taking into account the policies used when sampling trajectories (line 4).
At last, the critic is updated (line 5), taking as an input the z value so that it can make robust estimations of the expected reward for all the policies in the subspace.
Adapting off-policy algorithms would be similar.
Additional details are provided in appendix.
Note that, for environments with discrete actions, we have made the same adaptation based on the A2C algorithm since A2C has less hyperparameters than PPO and is easier to tune, with similar results.
E XPERIMENTS
We perform experiments in 6 different environments.
Implementations based on the [anonymous] library together with train and test environments will be released upon acceptance.
For each environment, we consider one train environment on which we trained the different methods, and multiple variations of the training environment for evaluation resulting in 50 test environments in total.
The details of all the environment configurations and detailed performance are given in Appendix B. Note that the complete experiments correspond to hundred of trained policies, and dozens of thousands of policy evaluations.
For simple control environments (i.e., CartPole, Pendulum and AcroBot), we introduce few variations of the physics constant at test-time, for instance by varying the mass of the cart, the length of the pole.
For complex control environments (i.e., HalfCheetah and Ant using the
BRAX library (Freeman et al., 2021), we both use variations of the physics (e.g., gravity), variations of the agent shape (e.g., changing the size of the leg, or of the foot) and sensor alterations.
At last, in
MiniGrid and ProcGen we perform experiments where the agent is trained in one particular levels, but is evaluated in other levels (single levels on MiniGrid, and set of 10 levels in ProcGen).
Note that
ProcGen is a pixel-based environment where the architecture of the policy is much more complex than in control environments.
Toy experiments on a simple Maze 2d are given in Appendix B.8 to show the nature of the policies learned by the different methods.
We compare our approach LoP 4 with different state-of-the-art methods: a) The Single approach is just a single policy learned on the train environment, and evaluated on the test ones.
b)
The DI-
AYN+R (reward) method is an extension of DIAYN (Eysenbach et al., 2018) where a set of discrete policies is learned using a weighted sum between the DIAYN reward and the task reward:
[Equation]
Critically, this model requires to choose a discriminator architecture to compute log p ( z | s ) and modifies the train reward by defining an intrinsic reward that may drastically change the behavior of the policies at train time.
c)
At last, we also compare with the model proposed in (Osa et al., 2021) denoted Lc (Latent-conditioned) that works only for continuous actions.
This model is also based on a continuous z variable sampled uniformly at train time, but only uses an auxiliary loss without changing the reward.
This auxiliary loss is defined through the joint learning of a density estimation model log P ( z | s, a ) where back-propagation is made over the action a .
As in DIAYN+R, this model needs to carefully define a good neural network architecture for density estimation.
Since
Lc cannot be used with environment that have discrete actions, we have adapted DIAYN+R (called
DIAYN+R Cont. )
using a continuous z variable (instead of a discrete one) and a density estimation model log P ( z | s ) as in Osa et al.
Note that we do not compare to (Kumar et al., 2020a) for the exact same reason as the one identified in (Osa et al., 2021): SMERL assumes that the reward is known over the complete trajectories which results in unnatural adaptation of on-policy RL algorithms like PPO.
Moreover, preliminary experiments with SMERL does not demonstrate any advantage against DIAYN+R correctly tuned.
We also provide (see Table 4 in Appendix B.1) results where K independent policies are learned, the best one being selected over each test environment.
This approach obtains lower performance than the proposed baseline and needs K more training samples making it unrealistic in most of the environments.
As network architectures, we use multi-layer perceptrons (MLP) with ReLU units for both the policy and the critic (detailed neural network architectures are described in Appendix).
For DIAYN+R log P ( z | s, ... ) is also modeled by a MLP with a soft-max output.
For Lc and DIAYN+R Cont., log P ( z | s, ... ) is modeled by a MLP that computes the mean of a Gaussian distribution with a fixed variance.
For these baselines, z is concatenated with the environment observation as an input for the policy and the critic models.
To choose the hyper-parameters of the different methods, let us remind that test environments cannot be used at train time for doing hyper-parameters search and/or model selection which makes this setting particularly difficult.
Therefore, we rely on the following procedure: a grid-search over hyper-parameters is made, learning a single policy over the train environment.
The best value of the hyper-parameters is then selected as the one that provides the best policy at train time.
These hyper-parameters are then used for all the different baselines.
Concerning the β value, for LoP, we report test results for β = 1 . 0 while, for Lc and DIAYN+R, we use the best value of β on test environments.
This corresponds to an optimistic evaluation of the baseline performances; aiming at showing that our method is much more efficient since it does not need such a beta-tuning ( β = 1 .giving good performance in the majority of cases).
Said otherwise, we compare our model in the less favorable case where baselines have been unrealistically tuned.
For the adaptation step, each policy is evaluated over 10 episodes for stochastic environments orsingle episode for deterministic environments.
We repeat this procedure over 10 different training seeds, and report the reward over the different test environments together with standard deviation.
All detailed results are available in Appendix.
A NALYSIS
We report the test performance of the models on different environments in Table 1.
In all the environments, the adaptive models perform better than learning a single policy over the train environment which is not surprising.
In most of the cases, LoP is able to achieve a better performance than other methods.
For instance, on HalfCheetah where we evaluate the different methods over 16 variations of the train environments, LoP achieves an average reward of 10589 while Lc and DIAYN+R obtain respectively 9547 and 9680 (standard deviations are reported in Appendix B).
Some examples of the discovered that behaviors in Ant and HalfCheetah 5 for the different methods, and for different values of z are illustrated in Figures 1b, 5 and 6.
This outlines that learning models that are optimal on the train task reward, but with different parameter values, allows us to discover policies react differently to variations of the training environment.
It seems to be a better approach than encouraging policies to have a different behaviors (i.e., generating different state distributions) at train time.
Same
[Table] [Table]
[Figure]
conclusions can be drawn in most of the environments, including MiniGrid where LoP is able to explore large mazes while being trained only on small ones.
On the ProcGen environment, where the observation is an image processed through a complex ConvNet architecture (See Appendix B.7), enforcing functional diversity (DIAYN+R) does not allow to learn good policies while the LoP model is able to better generalize to unseen levels.
Note that the performance at train time is the same for all the different approaches reported (see the Table 3 for instance) but quickly decreases in DIAYN for larger values of β while it stays stable for LoP where the best results are obtained for β = 1 . 0 .
Interestingly, in CartPole, DIAYN+R performs quite well.
Indeed, when analyzing the learned policies, it seems to be a specific case where it is possible to obtain optimal policies that are diverse w.r.t the states they are sampling (by moving the cart more or less on the right/left while maintaining the pole vertical).
We have also performed experiments where test environments have the same dynamics as the training environment, but with defective sensors (i.e., some features at test time have a null value – see
Appendix B.2 Table 7 on the Ant environment).
The fact that LoP behaves also well confirms the effectiveness of our approach to different types of variations, including noisy features on which baselines methods were not applied in previous publications.
Sensitivity to hyper-parameters: One important characteristic of LoP is that it can be used with β = 1 . 0 and does not need to define any classifier architecture as opposed to DIAYN+R and Lc.
Indeed, as shown in Figure 3 (left), the training performance of DIAYN drastically depends on a good tuning of β .
Lc, which is less sensible, needs to use a correct classifier architecture as in
DIAYN.
LoP is simple to tune since the cosine term is usually easy to satisfy and our approach, at convergence, always reaches a 0 value on this term when β > 0 . 0 .
As illustrated in Appendix B. and B2, it is also interesting to note than, on the BRAX environments, the number of environment interactions needed to train LoP is similar than the one needed to train a single policy and LoP comes with a very small overhead in comparison to classical methods.
Online adaptation: One interesting property is the number of policies (and thus of episodes) to test over a new environment to get a good performance.
For LoP and Lc, given a trained model, one can evaluate as many policies (i.e., different values of z ) as desired.
For DIAYN+R, testing more policies also means training more policies which is expensive and less flexible.
Table 3 (right) provides the reward of the different methods when testing K policies on different HalfCheetah settings: as expected, the performance of DIAYN+R tends to decrease when K is large since the model has difficulties to learn too many diverse policies.
For LoP and Lc, spending more episodes to evaluate more policies naturally leads to a better performance: these two models provide a better way to deal with the exploration-exploitation trade-off at test time.
Again, please consider that Lc also needs to define an additional neural network architecture to model log P ( z | s, a ) while LoP does not, making our approach simpler.
Beyond a Line of Policies: While LoP is based on the learning of N = 2 anchor parameters, it is possible to combine more than two anchor parameters.
We study two approaches combining N =anchor parameters (that can be extended to N = 3 ): a) the first approach is a convex combination of policies (CoP) where z is sampled following a Dirichlet distribution.
(b) The second approach is a
Bézier combination (BoP) as explained in Appendix A.2.
The results are presented in Table 3 (right) over multiple HalfCheetah environments.
It can be seen that these two strategies are not so efficient.
LoP is thus a good trade-off between the number of parameters to train and the performance (Note that BoP and CoP need more samples to converge), at least given the particular neural network architectures we have used in this paper.
We also performed an in-depth analysis of the evolution of the reward when K is increasing for LoP and CoP in Halfcheetah test environment (Figure 9 in
Annex).
While we expected CoP to outperform LoP when K is high, the best reward becomes stable when K=20 for both methods, and in most test environments, CoP is not able to reach the same best reward as LoP.
Analysis of the learned policies: To better understand the nature of the policies discovered by the different approaches, we have made a qualitative study in which we analyze i) the robustness of the methods to corrupted observations, ii) the functional diversity induced by the different models, and iii) the specificity of the different learned policies to particular test environments.
First, LoP is more robust to input feature corruption (see Table B.2 for the results, and Table 5 for the setting in
Appendix) and we conjecture that it is because the diversity in the parameter space allows this model to learn policies that does not take into account the same input features equally.
We also measure the functional diversity induced by the different models by training a posteriori a classifier that aims at recovering which policy (i.e which value of z ) has generated particular trajectories (Exact protocol in Figure 8 in Appendix, with the training curves).
On LoP with K = 5 , such a classifier obtains a 82% accuracy at validation time showing that the 5 policies are quite diverse, but less than the DIAYN+R policies where the classifier reaches a 100% accuracy which is logical knowing the auxiliary loss introduced by DIAYN which enforces this type of diversity.
It is interesting to note that with the trajectories generated in the test environments with LoP policies, the accuracy of the classifier is reaching 87 %: when LoP is facing new environments, it tends to generate more diverse policies.
We think that it is due to the fact that, since the policies have different parameter values, they react differently to states that have not been encountered at train time.
At last, Figure 4 (right) (and Figure 7 in appendix for K=10,20) illustrates which upon K = 5 policies is used for different test environments.
It shows that both LoP and DIAYN+R use different policies over different test environments, showing that these methods are able to solve new environments by learning various policies and not a single but robust one.
Examples of policies on a simple maze2d are given in
Figure 4 (left) and Appendix which illustrate the diversity of the discovered policies.
R ELATED W ORK
Our contribution shares connections with different families of approaches.
First of all, it focuses on the problem of online adaptation in Reinforcement Learning which has been studied under different terminologies: Multi-task Reinforcement Learning (Wilson et al., 2007;
Teh et al., 2017), Transfer
Learning (Taylor and Stone, 2009; Lazaric, 2012) and Meta-Reinforcement Learning (Finn et al., 2017; Hausman et al., 2018; Humplik et al., 2019).
Many different methods have been proposed, but the best majority considers that the agent is trained over multiple environments such that it can identify variations (or invariant) at train time.
For instance, Duan et al.
(2016) assume that the agent can sample multiple episodes over the same environments and methods like (Kamienny et al., 2020;
Liu et al., 2021) consider that the agent has access to a task identifier at train time.
More recently, diversity-based approaches have been adapted to focus on the setting where only one training environment is available.
They share with our model the idea of learning multiple policies instead of a single one.
For instance, DIAYN (Eysenbach et al., 2018) learns a discrete set of policies that can be reused and fine-tuned over new environments.
It has been adapted to online adaptation in (Kumar et al., 2020a) where the authors propose to combine the intrinsic diversity reward together with the training task reward.
This trade-off is obtained through a threshold-based method (instead of a simple weighted sum) with good results.
But this method suffers from a major drawback identified in (Osa et al., 2021): it necessitates to sample complete episodes at each epoch which is painful and not adapted to all the RL learning algorithms.
Osa et al.
(2021) also proposed an alternative based on learning a continuous set of policies instead of a discrete one without using any intrinsic reward.
The method we propose is highly connected to recent researches on mode connectivity with neural networks.
Mode connectivity is a set of approaches and analyses that focus on the shape of the parameter space.
It has been used as a tool to study generalization in the supervised learning setting (Garipov et al., 2018), but also as a way to propose new algorithms in different settings (Mirzadeh et al., 2021).
Obviously, the work that is the most connected to our approach is the model proposed in (Wortsman et al., 2021) that provides a way to learn a subspace of models in the supervised learning setting.
Our contribution adapts this approach to RL for learning policies in a completely different setting which is online adaptation.
At last, our work is sharing similarities with robust RL which aims at discovering policies robust to variations of the training environment (Oikarinen et al., 2020; Zhang et al., 2021).
The main difference is that robust RL techniques learn policies efficient ‘in the worst case’ and are not focused on the online adaptation to test environments (the objective is usually to learn a single policy efficient on different variations while we are learning multiple policies, just selecting one at test time).
C ONCLUSION AND P ERSPECTIVES
We investigate the idea of learning a subspace of policies in the reinforcement learning setting, and describe how this approach can be used for online adaptation.
While simple, our method allows to obtain policies that are robust to variations of the training environments.
Contrarily to other techniques, LoP does not need any particular tuning or definition of additional architectures to handle diversity, which is a critical aspect in the online adaptation setting where hyper-parameters tuning is impossible or at least very difficult.
Future work includes the extension of this family of approaches in the continual reinforcement learning setting, the deeper understanding of the the built subspace and the investigation of different auxiliary losses to better control the shape of such a subspace.
R EPRODUCIBILITY S TATEMENT
We have made several efforts to ensure that the results provided in the paper are fully reproducible.
In Appendix, we provide a full list of all hyperparameters and extra information needed to reproduce our experiments.
The source code is available on SaLinA repository such that everyone can reproduce the experiments 6 .
A I MPLEMENTATION DETAILS
In this section, we provide details about the implementations of the baselines and our models.
Appendix B provides details and additional results for each environment we used.
A. L O P-PPO
We detail the losses used in algorithm 7.
First, we recall the clipped-PPO surrogate objective described in Schulman et al.
For a given trajectory τ = { ( s t , a t , r t ) } T 0 collected by the policy π θ old , and denoting ρ t ( θ ) =
π θ ( a t | s t ) π θold , the goal is to maximize:
[Equation]
Where function A π θold is computed thanks to a value function V ϕ by using Generalized Advantage
Estimation (Schulman et al.
This function is simultaneously updated by regression on mean-squared error over the rewards-to-go (cid:98) R t .
In our case, this function not only takes s t as an input, but also the value z :
[Equation]
In HalfCheetah and Ant experiments, we sampled actions from a reparametrized Gaussian distribution using a squashing function (Ward et al.
, but we set the standard deviation fixed (so it is an hyper-parameter encouraging exploration, called action std in Tables 3 and 6): the policy network only learns the mean of this distribution.
A. B O P AND C O P
The only change between LoP and these models resides in the way we combine the N anchor policies.
For CoP, it is just the generalization of LoP for N > 2 (see 3.1).
BoP, makes use of a
Bezier parametric curve that uses Bernstein polynomials (the anchor parameters being the control points ).
For N = 3 , it is defined by:
[Equation]
Concerning the policies z evaluated at test time, BoP uses the same strategy as LoP by testing values that are uniformly distributed in [0; 1] .
For CoP, we opted for sampling K policies using a Dirichlet distribution over [0 , 1] 3 .
A. DIAYN+R AND L C
In order to find the best trade-off between maximizing environment rewards and intrinsic rewards in
DIAYN+R algorithm, we add the hyper-parameter β :
[Equation]
[Equation]
As an alternative to DIAYN+R Osa et al.
(2021) proposes an algorithm where the discriminator takes not only observations as an input but also the policy output, updating both discriminator q ϕ and policy π θ when back propagating the gradient.
In this case, it is not necessary to add an intrinsic reward.
While Osa et al.
(2021) illustrate their methods with TD3 and SAC, we adapted it to PPO.
The surrogate loss is given by:
[Equation]
B E XPERIMENTS DETAILS AND ADDITIONAL RESULTS B. H ALF C HEETAH
Task originally coming from OpenAI Gym (Brockman et al., 2016).
Instead of using MuJoCo engine, we decided to use Brax (Freeman et al., 2021) as it enables the possibility to acquire episodes on GPU.
We use the vanilla environment for training.
The policy and the critic are encoded by two different multi-layer perceptrons with ReLU activations.
The base learning algorithm is PPO.
Test environments: we operated modifications similar as the ones proposed in (Henderson et al., 2017).
Morphological variations: we changed the radius and mass of specific body parts (torso, thig, shin, foot).
Variations in physics: we changed the gravity and friction coefficients.
Table 2 precisely indicates the nature of the changes for each environment.
[Table]
friction, we also tried an increase/decrease by 50% (respectively tagged "Huge" and "Tiny").
[Table]
[Table]
B.
A NT
Task originally coming from OpenAI Gym (Brockman et al.
Instead of using MuJoCo engine, we decided to use Brax (Freeman et al.
as it enables the possibility to acquire episodes on GPU.
We use the vanilla environment for training.
The policy and the critic are encoded by two different multi-layer perceptrons with ReLU activations.
The base learning algorithm is PPO.
Test environments: As for HalfCheetah, we operated variations in physics (gravity and friction coefficients).
We also designed environments with a percentage of masked features to simulate defective sensors (They are sampled randomly and remain the same for each run).
Table 5 precisely indicates the nature of the changes for each environment
[Table] [Table]
[Table]
Value lr policy: 0 .lr critic: 0 .n parallel environments: n acquisition steps per epoch: batch size: num minibatches: update epochs: discount factor: 0 .clip ration: 0 .action std: 0 .gae coefficient: 0 .reward scaling: 1 . gradient clipping: 10 .
n layers (policy): n neurons per layer (policy): n layers (critic): n neurons per layer (critic): LoP β : { 0 . 1 , 1 , 10 } DIAYN β : { 0 . 1 , 1 , 10 } lr discriminator: 0 .n layers (discriminator): n neurons per layer (discriminator): Lc β : { 0 . 1 , 1 , 10 } dimensions of z: lr discriminator: 0 .n layers (discriminator): n neurons per layer (discriminator):
[Table]
B. C ART P OLE
We use the openAI gym implementation of CartPole as a training environment.
The 6 test environments are provided by Packer et al.
(2018) where three different factors may vary: the mass of the cart, the length of the pole and the force applied to the cart.
The length of each episode is 200.
The policy and the critic are encoded by two different multi-layer perceptrons with ReLU activations.
The base learning algorithm is A2C.
[Table]
[Table] [Table]
[Table]
A CRO
[Table]
We use the openAI gym implementation of Acrobot as a training environment.
The 4 test environments are provided by Packer et al.
(2018) where two different factor may vary: the intertia factor and the lengh of the system.
We have used A2C as a learning algorithm.
[Table] [Table]
[Table]
B. P ENDULUM
We use the openAI gym implementation of Pendulum as a training environment.
The 3 test environments are provided by Packer et al.
(2018) where two different factor may vary: the mass and the length of the pendulum.
We have considered 5 discrete actions between − 1 and +1 .
We have used
A2C as a learning algorithm
[Table]
[Table] [Table]
B. M INI G RID
We have use Gym Minigrid to perform experiments on mazes Chevalier-Boisvert et al.
We have used the MultiRoom-N2-S4 for training considering one single maze.
At test time, we have tested on three different MultiRoom-N2-S4 environments composed of two rooms, but also on three
MultiRoom-N4-S5 composed of four rooms.
This allows us to evaluate the generalization power of the different methods to larger mazes.
We have used A2C as a learning algorithm.
[Table]
[Table]
[Table] [Table]
0.619 ± 0. 0.348 ± 0.
[Table]
±
[Table]
[Table]
0.433 ± 0.
0.865 ± 0. 0.627 ± 0.
[Table]
[Table] ± 0.
[Table]
0.194 ± 0.
0.617 ± 0.
0.348 ± 0.
0.656 ± 0.
[Table] 0.0 ± 0.
0.294 ± 0.
0.0 ± 0.
0.24 ± 0.
[Table] 0.0 ± 0.
0.004 ± 0.
0.0 ± 0.
0.137 ± 0.
[Table]
0.0 ±
[Table]
B. P ROC G EN (F RUIT B OT )
We performed experiments on pixel-based environment with ProcGen.
We used the FruitBot game for training considering 10 levels sampled uniformly at each episode.
At test time, we selecteddifferent environments, each of them composed of 10 uniformly sampled levels, sampled uniformly, and that were not seen at train time.
We used the CNN architecture described in the ProcGen paper (IMPALA architecture (Espeholt et al.
For LoP, the two first blocks are fixed and only the parameters of the last block depend on the value of z .
For DIAYN, the z value is provided as a one-hot vector stacked to the observation.
As for Brax environment, we used PPO algorithm.
Hyper-parameter
[Table] [Table]
B. M AZE 2 D WITH WALLS
To visualize the policies learned by the different methods
[Table]
we just implemented a simple discrete maze (4 actions = up,down, left, right) where the objective is to go from the top-middle tile to the bottom-middle tile by moving through a corridor (size is 21 × 11 ).
Reward is -1 at each step until goal is reached and the maximum number of steps is − 100 .
The optimal policy in the training environment achieves − 16 .
At test time, we generate walls in the corridor such that the agent has to avoid these walls to reach the goal.
The observation space is a 5 × 5 square around the agent.
Policies are learned by using PPO.
Illustrations of the trajectories over the train and the 4 test environments are illustrated in Figures 11,12,13, [Table] n [Table]
clipping
[Table] [Table]