PA N DR: F AST A DAPTATION TO N EW E NVIRONMENTS FROM O FFLINE E XPERIENCES VIA D ECOUPLING P OLICY AND E NVIRONMENT R EPRESENTATIONS
A BSTRACT
Deep Reinforcement Learning (DRL) has been a promising solution to many complex decision-making problems.
Nevertheless, the notorious weakness in generalization among environments prevent widespread application of DRL agents in real-world scenarios.
Although advances have been made recently, most prior works assume sufficient online interaction on training environments, which can be costly in practical cases.
To this end, we focus on an offline-training-onlineadaptation setting, in which the agent first learns from offline experiences collected in environments with different dynamics and then performs online policy adaptation in environments with new dynamics.
In this paper, we propose Policy
Adaptation with Decoupled Representations (PAnDR) for fast policy adaptation.
In offline training phase, the environment representation and policy representation are learned through contrastive learning and policy recovery, respectively.
The representations are further refined by mutual information optimization to make them more decoupled and complete.
With learned representations, a PolicyDynamics Value Function (PDVF) (Raileanu et al., 2020) network is trained to approximate the values for different combinations of policies and environments.
In online adaptation phase, with the environment context inferred from few experiences collected in new environments, the policy is optimized by gradient ascent with respect to the PDVF.
Our experiments show that PAnDR outperforms existing algorithms in several representative policy adaptation problems.
I NTRODUCTION
Reinforcement learning (RL) has achieved great successes in many fields, e.g., Game Playing (Silver et al., 2016), Robotics (Andrychowicz et al., 2020), Supply Chain (Ni et al., 2021), Medicine
Discovery (Schreck et al., 2019) and so on.
Despite of the successes, conventional RL is well known to be lack of generalization ability (Packer et al., 2018).
To be specific, a policy well optimized in a specific environment can be catastrophically poor even in similar environments.
The poor generalization among environments and policys severely prevents more practical applications of RL.
In the literature on RL generalization (Kirk et al., 2021), many approaches like data augmentation (Yarats et al., 2021) and domain randomization (Peng et al., 2018) are proposed to increase the similarity between training environments and the testing environments (i.e., the ones to generalize or adapt to).
These methods often require knowing the variations of environment and having the access to environment generation.
In contrast, some other works aim to achieve fast adaptation in testing environments without the need for such access and knowledge.
Gradient-based Meta RL, e.g., MAML (Finn et al., 2017), aims to learn a meta policy which can adapt to a testing environment within few policy gradient steps.
Also in this branch, context-based Meta RL, e.g., PEARL (Rakelly et al., 2019), leverages a context-conditioned policy which allows to achieve adaptation by the generalization among contexts.
Useful context representation is learned in training environments to capture the variations; then the context of a testing environment can be inferred from few probed interaction experiences (Fu et al., 2021).
Most prior works consider an online training setting where the agent is allowed to interact with training environments arbitrarily within the interaction budget.
However, in real-world problems, online interaction is usually expensive while offline experiences are often available and relatively sufficient.
For example, in E-commerce, adapting the advertising strategies obtained from massive offline ads-users interaction data to a new user can effectively save the budgets of advertisers.
On this point, a recent work (Raileanu et al., 2020) studies fast adaptation under an offline-trainingonline-adaptation setting, where only offline experiences collected on training environments are available before online policy adaptation in new environments.
This is a more practical and general setting which is meanwhile more challenging, since the policy learning and variation capture need to be conducted in an offline manner.
To solve this problem, Raileanu et al.
(Raileanu et al., 2020) propose an extended value function, called Policy-Dynamic Value Network (PDVF).
By definition, a PDVF additionally takes an environment representation and an policy representation as inputs, and evaluates the corresponding values.
In principle, the explicit representation allows the generalization of value estimation among environments and policies, which can be utilized to realize policy adaptation.
To be specific, given the offline experiences generated from different combinations of training policies and training environments, the representations of environment and policy are learned from transitions and state-action sequences by dynamics prediction and policy recovery respectively.
Thereafter, a quadratic form of PDVF is approximated, according to which the policy is optimized during online adaptation with the environment representation inferred from few interactions in the testing environment.
The first drawback is that the representations of environment and policy can have redundant and entangled information since they are from the same interaction experiences of the corresponding combination.
It negatively affects the generalization conveyed by the representations which is vital to fast adaptation.
Another limitation is that the quadratic approximation of PDVF inevitably makes a sacrifice in expressivity for convenient closed-form optimization, in turn this cripples the optimality of policy obtained during adaptation.
In this paper, we focus on the offlinetraining-online-adaptation setting and follow the paradigm of (Raileanu et al., 2020).
We propose P olicy A daptatio n with
D ecoupled
R epresentations (PAnDR) for more effective fast adaptation of RL.
The conceptual illustration of PAnDR is shown in Fig.
Compared to prediction adopted by (Raileanu et al., 2020), we adopt trajectory-level context contrast for environment representation learning.
By this means, the variation is captured in an integral manner and the inter-environment difference is emphasized, thus offering more useful information and better robustness.
One can link our comparison here to that between reconstruction and contrastive learning, where the superiority of the latter has been widely demonstrated (He et al., 2020; Laskin et al., 2020;
Fu et al., 2021).
Moreover, we propose a information-theoretic representation learning method to refine environment and policy representations for information decoupleness and completeness.
This is achieved by mutual information minimization and maximization among environment and policy representations and their joint.
In addition, distinct to the quadratic form in (Raileanu et al., 2020), PAnDR leverages a typical multi-layer nonlinear neural network for the approximation of the PDVF.
Then during online adaptation, the policy is optimized by gradient ascent.
In this sense, PAnDR gets rid of such constraints to improve value approximation and policy adaptation.
In experiments, we evaluate PAnDR in a series of fast adaptation tasks and analyze the contribution of each part of proposed methods.
We summarize our main contributions as follows: 1) We leverage trajectory-level contrastive learning and propose a novel mutual information based approach for better environment and policy representations.
We get rid of the constraint of prior quadratic approximation of PDVF and demonstrate the effectiveness of nonlinear approximation in improving online policy adaptation.
We show that
PAnDR consistently outperforms representative existing algorithms in our experiments, sometimes with a large margin.
P RELIMINARIES 2. R EINFORCEMENT L EARNING
Consider a Markov Decision Process (MDP) defined as M = ⟨ S, A, P, R, γ ⟩ , with the state space
S , the action space A , the transition probability P : S ×
A × S → [0 , 1] , the reward function R :
S ×
A → R and the discount factor γ ∈ [0 , 1) .
An agent interacts with the MDP M by performing its policy π : S ×
A → [0 , 1] , generating the trajectories as a t ∼ π ( ·| s t ) , s t +1 ∼ P ( · | s t , a t ) and r t =
R ( s t , a t ) .
The value function of policy π is defined as V π ( s ) =
E π [ (cid:80) ∞ t =
γ t r t | s
=
s ] .
The objective of an RL agent is to optimize its policy to maximize the expected discounted cumulative rewards J ( π ) =
E s 0 ∼ ρ 0 ( · ) [ V π ( s 0 )] , where ρ 0 is the initial state distribution.
2. M UTUAL I NFORMATION O PTIMIZATION
Mutual Information (MI) is a widely used metric that estimates the relationship between pairs of variables in machine learning.
Given two random variables X and Y , MI is a measure of dependency among them, defined as I ( X ; Y ) =
E p ( x,y ) (cid:104) p ( x,y ) p ( x ) p ( y ) (cid:105) , with the joint probability distribution p ( x, y ) and the marginal distributions p ( x ) , p ( y ) .
MI estimation and optimization are fundamental techniques in representation learning, e.g., in Information Bottleneck (Alemi et al., 2017).
In most problems, it is intractable to compute MI directly since the exact distributions of interested variables are not known.
Thus, approximate estimation of MI is usually resorted to, which is nontrivial especially in high-dimensional space.
For the general purpose of MI optimization, lowerbound and upper-bound approximations of MI are often used as surrogate objectives (Poole et al., 2019).
Among the options, variational approximation is a common way to compute MI lower bound:
[Equation]
The lower bound holds since the KL divergence is non-negative.
For the upper bound, a recent work called Contrastive Log-ratio Upper Bound (CLUB) (Cheng et al., 2020) provides the following form:
[Equation]
where q ( x | y ) is approximated, e.g., by maximum likelihood estimation (MLE), of actual conditional distribution p ( x | y ) when p is not known.
We provide a detailed discussion on related work in Appendix E.
M ETHODOLOGY
In this section, we propose our framework for fast adaptation, named Policy Adaptation with Decoupled Representation (PAnDR).
We follow the paradigm of (Raileanu et al., 2020) and consider the offline-training-online-adaptation setting with a set of training environments Ω train =
{M i } Mi =1 and a set of testing environments Ω test =
{M j } Nj =1 .
Given a set of training policies Π train = { π k } Ok =1 , an offline experience buffer is established as D =
(cid:83) M,Oi =1 ,k
=1 D i,k , where D i,k contains the s, a, rtrajectories generated by π k in M i .
M, N, O denote the sizes of corresponding set.
In this paper, we call a set of ( s, a, r ) generated in an environment as the context of it, and a set of ( s, a ) collected by a policy as its behavior , denoted by c and b respectively.
With offline experience buffer D , PAnDR learns the representations of environment and policy through context contrast and policy recovery, respectively (Sec. 3.1). To eliminate the redundant [Figure] 𝓓 = {(𝒔,𝒂,𝒔′)} 𝑷(𝒙|𝜫,𝜴)
𝑫 𝒊,𝒌𝒊,𝒌=𝟏,𝟏 𝑴,𝑶 𝑫 𝒊,∙𝒊=𝟏𝑴 𝑫 ∙,𝒌𝒌=𝟏𝑶 𝒛 𝒃 𝜱 𝒃 𝒛 𝒆 𝜱 𝒆 𝒛 𝝅 𝜱 𝝅
Maximize 𝑰(𝒛 𝒃 ;𝒛 𝒆 ,𝒛 𝝅 ) ~ Minimize 𝑰(𝒛 𝒆 ;𝒛 𝝅 ) { 𝝅 𝒊 ,𝒆 𝒋 } information entangled in the two representations while retaining necessary information, we further introduce a novel approach by mutual information minimization and maximization to refine the representations (Sec. 3.2).
Thereafter, we train a nonlinear neural network to approximate PDVF conditioning on both environment and policy representations and introduce the online adaptation with gradient-based policy optimization (Sec. 3.3).
3. S ELF - SUPERVISED R EPRESENTATION L EARNING
The first part of PAnDR is to learn effective representations for environment and policy from offline experiences, replying on which the environment generalization and policy adaptation are carried out.
We learn these representations in a self-supervised manner via context contrast and policy recovery, to capture the pivotal information of environment variation and policy behavior.
Learning Environment Representation via Context Contrast By following the general idea of self-supervised Contrastive Learning (He et al., 2020), we make use of the contrast of contexts collected in different training environments to extract context embedding as environment representation.
For a training environment M i , we use the contexts sampled from D i, · = (cid:83)
Ok =1 D
i,k as anchor sample
c
i and positive sample c + i ; in turn, the context sampled from the offline experiences generated in other environments, i.e., D i ′ , · for any i ′ ̸
= i , is taken as negative sample c − i .
With the contrastive context samples described above, we train a context encoder ϕ
e that maps context to embedding z e by ensuring the representation of similar sample pairs (anchors and their positives) stay close to each other while dissimilar ones (anchors and their negatives) are far apart.
Formally, the context encoder is optimized with InfoNCE loss similar to (He et al., 2020;
Laskin et al., 2020):
[Equation]
where W denotes the parameter matrix updated along with ϕ e .
Note that z e is differentiable with respect to ϕ e ; while z + e , z − e are the embeddings for positive and negative context samples calculated by a momentum averaged version of ϕ e .
Compared to learning environment representation by transition-level dynamics prediction as adopted in (Raileanu et al., 2020), PAnDR captures environment variations in an integral manner and emphasizes inter-environment differences by context-based contrastive learning.
Thus, the environment representation can be more effective and robust, as similarly demonstrated in (Fu et al., 2021).
Learning Policy Representation via Policy Recovery For a training policy π k , we learn the embedding of the behavior data sampled from D · ,k = (cid:83) Mi =1 D i,k as policy representation.
As shown in the left of Fig.
2, a behavior b is fed in the behavior encoder ϕ π , producing the embedding z π .
With the purpose of making z π contain the essential policy information, we utilize a z π -conditioned policy decoder ϕ dec π which takes as input a state s and the embedding z π , then deterministically outputs a prediction of the actual action made by π .
We call this policy recovery (or Behavioral
Cloning (Grover et al., 2018)).
With the offline experiences, ϕ π , ϕ dec π are updated by minimizing the l 2 prediction loss:
[Equation]
3. MI- BASED R EPRESENTATION R EFINEMENT
A consequent concern on the representations obtained above is the redundant and entangled information in them.
This is because the representations for both environment and policy are learned from the offline experiences.
The entangled information exists in both representations will negatively affect the generalization among environments and policies, thus hinders value approximation and policy adaptation in turn.
To address this problem, we propose a novel information-theoretic approach to refine the learned representations.
The key idea is 1) to eliminate the redundant information contained in environment and policy representations for the decoupleness, while 2) to ensure the essential information in their joint representation are completely retained in the two representations.
This is achieved by two ways of mutual information (MI) optimization, as illustrated in the middle part of Fig.
MI Minimization for Decoupleness Firstly, for the decoupleness, we are interested in reducing the entangled information in the representations of environment and policy.
To be specific, we minimize the MI between the embeddings z e and z π that are obtained from context and behavior data sampled from the offline experiences, i.e., min I ( z e ; z π ) .
We then convert the MI minimization into minimizing the CLUB upper bound of MI, as introduced in Sec. 2.2.
With the approximation conditional distribution q ψ 1 parameterized by ψ 1 , our loss function for representation decoupleness (RD) is formulated as:
[Equation]
Concretely, as to any D i,k , we use the embedding pairs of c i , b k ∼ D
i,k for the samples from joint distribution p ( z e , z π ) ; and use those of c i ∼ D i, · , b k ∼ D · ,k for the marginal distributions p ( z e ) , p ( z π ) .
Note that ψ 1 is not updated by L RD , while the approximate conditional distribution q ψ 1 is trained through MLE by minimizing the following loss:
[Equation]
where the joint distribution is sampled in the same way as in Eq.
The updates defined in Eq.and Eq.
4 are performed iteratively: q ψ 1 approximates the conditional distribution with respect to the embeddings given by ϕ e , ϕ π ; and in turn q ψ 1 is used in the estimation of CLUB.
We refer the readers to the original paper of CLUB (Cheng et al., 2020) for better understanding.
MI Maximization for Completeness
Secondly, we expect the information contained in the joint representation of environment and policy can be completely retained in both environment and policy representations.
This is also necessary especially when the decoupleness described above is required, in case that the vital information of environment variation or policy behavior is overly reduced during MI minimization.
This is achieved by maximizing the MI between the concatenation of the embeddings z e , z π and a target environment-policy joint representation z b which contains complete information, i.e., max I ( z e , z π ; z b ) .
We resort to extracting the target representation z b also from the offline experiences.
Similar but distinct to the environment representation, z b is learned by context-behavior joint contrast, i.e., only the context-behavior data sampled from the same D i,k are regarded as mutually positive samples.
We omit the similar formulations here and an illustration is shown in the right part of Fig.
Thereafter, we maximize the variational lower bound of
I ( z e , z π ; z b ) , i.e., minimize the loss function for representation completeness (RC) below:
[Equation]
where q ψ 2 is the variational distribution parameterized by ψ 2 , and the joint distribution are sampled by c i , b k ∼ D
i,k .
Note that the negative entropy term −H ( z b ) =
E p ( z b ) [log p ( z b )] irrelevant to the optimization of z e , z π is neglected in Eq.
Therefore, combining the self-supervised learning losses and the MI-based refinement losses, the total loss function for the representations of environment and policy is:
[Equation]
(6) where α, β are the balancing weights.
The pseudocode of overall representation training is in Alg.in Appendix.
3. V ALUE A PPROXIMATION AND P OLICY A DAPTATION
So far, we have introduced the training process of environment and policy representations on the offline experiences.
With the trained representations, we are ready to introduce the approximation of PDVF (Raileanu et al., 2020) and the policy optimization during online adaptation.
Value Function Network Approximation As described in Section 1, to evaluate the values of different environment-policy configurations, PDVF additionally conditions on the representations of environment and policy.
Distinct from the quadratic approximation adopted in (Raileanu et al., 2020), PAnDR leverages a typical multi-layer nonlinear neural network, denoted by V θ
( s, z e , z π ) with parameter θ .
This setting frees the function expressivity constrained by quadratic form.
We suggest that this is important because the value approximation for multiple environments and policies is much more difficult than the conventional situation (e.g., V π ).
Achieving better value approximation is crucial to the following policy adaptation.
For any offline experience D i,k , we can obtain the corresponding representations z e , z π , conditioning on which V θ is trained by Monte Carlo method (Sutton & Barto, 1998):
[Equation]
(7) where G ( s ) = (cid:80) Tt =
γ t r t letting s 0 = s .
The pseudocode is provided in Alg. 2 in Appendix.
Policy Adaptation via Representation Gradient Ascent After training V θ network well, in principle we can evaluate the values for any environment-policy configuration to a certain degree.
For online adaptation, we first sample a training policy from Π train and use it to interact with the testing environment for a few steps.
Then the embedding of testing environment z on e can be inferred with the collected context.
Next, starting from an arbitrary policy representation z 0 π (randomly initialized by default), by gradient ascent (GA), we can optimize the policy representation along the direction of increasing values of V θ with learning rate η :
[Equation]
After a certain number of GA, the obtained policy representation can be used to interact with the testing environment with the help of the z π -conditioned policy decoder ϕ dec π (described in Sec. 3.1), which is trained offline to predict policy’s action.
We adopt the best-performing policy representation as z ∗ π encountered during the GA process for our adapted policy in our experiments.
It is worth noting that, given s and z o ne , the GA optimization itself is zero-shot, i.e., without the overhead of interacting with the testing environment.
The pseudocode is in Alg.
3 in Appendix.
E XPERIMENT
In our experiments, we aim at investigating the following questions: ( Q1 )
Whether PAnDR outperforms existing algorithms for fast adaptation? ( Q2 )
How do contrastive environment representation and MI-based refinement contribute to the performance of PAnDR?
( Q3 )
Whether nonlinear neural approximation of PDVF and gradient-based policy adaptation improve the quadratic counterpart?
4. S ETUP
We conduct experiments in four continuous control domains.
All the experimental details can be found in Appendix C.
Domains and Variations.
The fast adaptation domains used in our experiments is shown in Table 2 in Appendix A, where the Spaceship-Charge, Ant-Wind and Swimmer-Fluid are from (Raileanu et al., 2020) and the HalfCheetah-Mass is from (Lee et al., 2020). [Figure]
Offline Experiences and Adaptation Budget.
We follow the same data setting as (Raileanu et al., 2020).
For offline experiences, a policy is optimized by PPO (Schulman et al., 2017) in each training environments.
The obtained policies are training policies that serve as collectors.
In general, such collector policies can be obtained from other ways.
For each domain, 50-episode interaction trajectories are collected by each combination of training environment and policy, e.g., 15 ∗ 15 ∗episodes of experience for Ant-Wind.
For online adaptation, 50 steps of interaction are collected in the testing environment by one of the training policies.
For PAnDR, 100 steps of GA is used by default.
Evaluation Criteria.
The main evaluation criterion is adaptation performance , calculated as the undiscounted cumulative rewards of the policy obtained after online adaptation.
Moreover, for
PAnDR, we analyze the evolvement of adaptation performance against the number of GA adaptation steps (in Sec. 4.3).
The results reported in our experiments are means and error bars of a standard deviation calculated with 7 independent trials.
4. P ERFORMANCE C OMPARISON
In this section, we compare the adaptation performance between PAnDR and representative existing algorithms to answer Q1 within the domains introduced above.
Baselines.
We consider the following three baselines.
1) PPO:
We use the PPO policy which is trained online with 3 e 6 steps in testing environments as a powerful baseline.
2) MAML (Finn et al., 2017):
We use MAML as a representative of gradient-based Meta RL.
Note MAML (as well as most
Meta RL algorithms) performs online meta training.
This means in principle it is not applicable to the offline training experiences.
3) PDVF (Raileanu et al., 2020): Since we propose PAnDR based on the same paradigm of PDVF, we use PDVF as a closely related baseline.
Moreover, to our knowledge, PDVF is the state-of-the-art fast adaptation algorithm in offline-training-online-adaptation setting.
Note that in this section, we use PDVF to denote the method name of (Raileanu et al., 2020) rather than the extended value function.
For all above baselines, we use the official source codes provided by PDVF paper (Raileanu et al., 2020).
For PAnDR, we implement on these source codes with almost the same configuration of hyperparameters and structures for a valid comparison.
(A1) Results.
As shown in Fig.
3, for the simple Spaceship domain, the overall performance of
PAnDR is significantly better than PDVF and slightly outperfroms MAML.
Further in the three more difficult MuJoCo domains, the leading edge of PAnDR is much more evident and consistent compared with the two baseline methods, while MAML loses its competitiveness.
We consider that this is because the preferable representations obtained via self-supervised learning and the MI-based refinement are more effective and robust than those of PDVF.
Moreover, with less-constrained approximation, the more accurate value estimation achieved improves the optimality of policy obtained during adaptation.
It is worth noting that, perhaps somewhat surprisingly, PAnDR shows competitive and even better performance when comparing with the powerful PPO baseline trained in the testing environments.
Instead of requiring millions of interactions to optimize a policy, PAnDR only needs limited interacting experiences for fast adaptation.
This further reveals the superiority and potential of PAnDR.
[Figure]
4. A BLATION S TUDY AND A NALYSIS
In this section, we give the answers to Q2 and Q3 .
We provide main results below, and the complete results along with detailed analysis are given in Appendix B. [Table]
(A2-I) MI-based Representation Refinement.
To verify the contribution of the MI minimizing and maximizing in representation refinement of PAnDR, we compare PAnDR with its several variants including: 1) PAnDR without MAX MI: removes MI maximizing refinement; 2) PAnDR without MIN MI: removes MI minimizing refinement; 3) PAnDR without MI: removes both MI maximizing and minimizing refinements.
As shown in Fig.4, both MI minimizing and maximizing positively contribute to the improvement of PAnDR’s overall performance.
It demonstrates that both MI-based refinement components are indispensable in PAnDR.
(A2-II) Context Contrast for Environment Representation.
In Fig.4, we present the efficacy of context contrast in comparison with dynamics prediction upon both types of V θ in PDVF and
PAnDR (variants shown in Table 1).
As in Fig.4, PDVF with CER has superior performance than original PDVF while PAnDR without MI outperforms PAnDR with NA-GA.
We then conclude that context contrast is more favorable in both cases of V θ compared with dynamics prediction.
(A3-I) V θ Approximation & GA Adaptation.
Also with the variants in Table 1, we establish the comparsions: PAnDR v.s. PAnDR with QA, and PDVF v.s. PDVF with NA-GA.
The results shown in Fig.4 verify that there is a clear advantage of using NN nonlinear approximation of V θ along with
GA optimization over the quadratic counterpart.
(A3-II) Adaptation Performance against GA Steps.
Since there is no longer a closed-form solution to optimization available for V θ with NN nonlinear approximation, we investigate how consecutive steps of GA affect the adpatation performance of PAnDR.
The results for various choices of
GA step number are provided in Table 3-8 in Appendix B.3.
Moreover, we provide detailed analysis for useful insights.
C ONCLUSION AND D ISCUSSION
In this paper, we focus on fast adaptation problem with only offline experiences on training environments available.
Following the paradigm of (Raileanu et al., 2020), we propose a new algorithm called PAnDR that consists of effective environment and policy representations and better policy optimization during online adaptation.
In essence, policy adaptation in PAnDR (and PDVF) is a process of policy representation optimization and decoding.
Both the two parts are not well studied at present.
Optimization in policy representation space is non-trivial since there is a lack of knowledge on the landscape of the space; while the decoding of such latent representation can be invalid without carefully handling the uncertainty (Notin et al., 2021).
We consider that they are significant and there is much room for improvement.
In addition, advances in environment and policy representation learning are expected to further improve the performance of PAnDR.
A CKNOWLEDGMENTS
The work is supported by the National Science Fund for Distinguished Young Scholars (Grant No.: 62025602), the National Natural Science Foundation of China (Grant Nos.: 11931015, 62106172), the XPLORER PRIZE, the New Generation of Artificial Intelligence Science and Technology Major
Project of Tianjin (Grant No.: 19ZXZNGX00010) and the Science and Technology on Information
Systems Engineering Laboratory (Grant No. WDZC20205250407)
A PPENDIX A E NVIRONMENTS S ETTING D ETAILS
In each domain, we have a variety of environments controlled by corresponding variations.
These variations determine the dynamics and reward functions of environment.
Therefore, fast adaptation is expected to be achieved against these changes.
The number of training and testing environments are provided in Table 2.
[Table]
B C OMPLETE E XPERIMENTAL R ESULTS B. P ERFORMANCE ON T RAINING E NVIRONMENTS
Although the aim of meta-RL is the fast adaptation to new testing environments, we also present the results by testing the trained agents on the original training environments, as shown in Fig.5.
Obviously, PPO achieves the best performance in most of the environments thanks to re-learning in each environment.
Although our method is worse than PPO in most training environments, its overall performance is clearly better than MAML and PDVF.
In a few environments, PAnDR even surprisingly outperforms PPO algorithm, which further reveals the superiority and potential of PAnDR.
We find that PAnDR performs poorly in environments 8, 9, 10, 11 and 12.
This is because the learning of our algorithm is based on the offline dataset collected by PPO.
Due to the poor performance of
PPO, the collected offline experience is of low quality, and we cannot extract abundant and effective environment and policy information from it, thus limiting the performance ceiling of our algorithm
The reason for the poor performance of PDVF in the testing environment of HalfCheetah-Mass showed in Fig.3 can also be summarized from Fig.5.
PDVF can’t get satisfactory performances even in the training environments, so it can’t be expected to generalize in the testing environments and perform well.
[Figure]
B. I N - DEPTH A NALYSIS OF MI- BASED R EFINEMENT
Here we give the results on Swimmer and HalfCheetah in Fig.6.
As mentioned in Section 4.3, both MI minimizing and maximizing positively contribute to the improvement of PAnDR’s overall performance in all environment, which proves that both MI-based refinement components are indispensable in PAnDR.
We also observe a phenomenon from the experiments that the results of using
[Figure] [Figure]
only MI-maximization refinement are slightly worse than those of using only MI-minimization refinement in most of the environments.
This may mainly due to following two factors.
On the one hand, additional environment-policy joint representations are introduced into the refinement process compared with that of MI-minimization.
An inferior joint representation may result in limited or sometimes even negative improvements to the policy and environment representations during the refinement.
On the other hand, in contrast to MI-minimization, the current policy representation space is not further compressed in MI-minimization refinement.
As a result, the policy representation may still contains residual redundant information, which can negatively affect the subsequent value estimation and online adaptation
[Figure]
B. A DAPTATION P ERFORMANCE AGAINST GA S TEPS .
During the online adaptation phase, we use the gradient ascent (GA) to optimize the policy representation.
For this purpose [Figure] we conduct experiments of different numbers of steps of gradient ascent, the results are shown in Table 3 to Table 6.
It can be seen that as the number of steps increases, the performance of the optimized policy also rises gradually.
In Fig.8, we give the distribution of policy performance at each step when the number of steps = 200 in the SpaceShip environment, each policy is randomly sampled as described in Alg.3.
It can be concluded from the figures that different initial policies have different performance trends as the number of GA steps increases.
For example, as showed in Fig.8 Policy 3 Env.1, when the selected initial policy is inferior, a better result can potentially be obtained by gradient ascent.
To the opposite, when the selected initial policy is an approximated optimal one, GA with multiple steps may mislead the optimization towards the wrong direction so that the policy’s performance is even worse as showed in Fig.8 Policy 3 Env.4.
However, the more the number of gradients ascending, the more computation required, and Finally, considering the balance between the additional computation cost of more GA steps and the optimization of policy, we choose to stop the gradient ascend when the step reaches to 100 during the adaptation.
However, evaluating the updated policy at each step and then adopting the best-performing one will lead to heavy computational overhead.
To alleviate this problem, we evaluate the updated policy every K steps.
We give the results of different K as shown in Table 7 to Table 10.
We can see that the results of evaluating policy at different GA frequency are relatively acceptable when K ≤ 5 , which can reduce computational cost by more than 50%.
[Table] [Table]
B. R EMAINING A BLATION A NALYSIS
The remaining results of Q2 and Q3 are showed in Fig.6 and Fig.7.
The conclusions are consistent with the ones in Section 4.3: (1) Context contrast is more favorable in designing V θ both in (Raileanu et al., 2020) and PAnDR compared with dynamics prediction; (2) NN nonlinear approximation of V θ along with GA optimization has clear advantage over the quadratic counterpart.
C A DDITIONAL E XPERIMENTAL D ETAILS C. O THER A LGORITHMS E XPERIMENTAL SETTINGS
For PDVF, we use the official source codes from here 1 .
We follow the parameters settings used in the original paper.
After running three seeds on per environment, we found that except for the AntWind, PDVF cannot achieve the performance neither on Swimmer-Fluid nor Spaceship as reported in the original paper.
Moreover, the source codes take a lot of time in collecting new data during the second stage for data augmentation.
On the same device, the training time of PDVF is about three times that of PAnDR.
Therefore, in our experiments, we abandon the second stage of PDVF and use the results of the first stage training of 7 seeds.
For MAML, we use the codes from here 2 .
In MAML, in the meta training phase, the update number is set to 500 and during each update we sample 20 trajectories from each task; in the meta adaptation phase, the update number is set to 10 and during each update we sample 20 trajectories from each task.
C. E QUIPMENT AND S OFTWARE
All experiments are performed on a Linux server with Ubuntu 18.04 containing 256G of RAM,
Intel(R)
Xeon(R) CPU E5-2680 v4 @
2.40GHz CPU processor with 56 logical cores
, the server contains 4 Nvidia-2080Ti GPUs.
Our codes are implemented with Python 3.7, Torch 1.7.1, gym 0.15.7, and MuJoCo 2.0.2.
C. H YPERPARAMETERS S ETTING
Before training the encoder networks, we need first to train the PPO agents on the training environments, and then use these agents to collect data.
The hyperparameters setting of the PPO algorithm is shown in Table 11.
Table 12 shows the training parameters of the encoder networks.
The large difference between the weights of maximizing mutual information and minimizing mutual information is due to the
[Table] [Table]
different ways of calculating the upper and lower bounds of MI.
So we give a larger weight to the
MI minimizing weight to make the smaller loss work as well
[Table]
Table 13 gives the main parameters of the value approximation and policy adaptation stage.
D A LGORITHM P SEUDOCODE
Algorithm 1 describes the training process of encoders.
First, we trained the environment representation and policy representation by context contrast and policy recovery, respectively.
Then the MI loss is introduced to further refine the encoder networks.
Algorithm 2 and 3 correspond to the training process of the value function and the online adaptation process of the policy representation, respectively.
In Algorithm 2, we formulate the training of value function as a prediction problem using Monte Carlo return as the label.
In Algorithm 3, we describe
[Table] [Table]
the optimization of the policy representation by gradient ascent along the direction of increasing values of V θ .
E R ELATED W ORK E. M ETA -RL
Meta-RL is meta-learning in the field of RL that the training and testing tasks are drawn from the same family of problems while having different components such as reward probabilities, environmental dynamics and so on.
Common meta-RL methods are mainly divided into three classes.
The first class is the model-based meta-RL, Duan et al.
(2016) uses a RNN structure to record historical information in the agent model and incoporate task-related knowledge into the model parameters.
The second class is the optimization-based meta-RL methods.
A representative method is (Finn et al., 2017) that use base-learners to collect data on different subtasks and then use a meta-learner to learn from these data.
However, both model-based and optimization-based method requires onpolicy learning, of which the sampling efficiency is poor.
The third class is the context-based methods which propose to train an encoder in multi different environments to extract the environmental information.
The context of new environments is inferred using a small amount of data collected during the initial limited interactions.
Then the context is used as the input of the policy and value networks to help the agent quickly adjust its policy and adapt to the new environments.
The encoder is expected to capture the different task information.
In (Rakelly et al., 2019; Fakoor et al., 2020), the encoder networks are learned based on the reward signal.
In (Fu et al., 2021), the encoder is trained on the trajectories through contrastive learning.
However, training context encoders using reward signals maybe unstable while directly using trajectories may introduce redundant information as the encoder learned based on trajectory may contains policy information.
Recently, Raileanu et al.
(2020) uses a predictive approach to try to extract environment context and policy context separately, but they are not completely decoupled.
In contrast to these methods, our PAnDR can extract and refine policy and environment embeddings to obtain the decoupled representations.
[Table] [Table]
E. R EPRESENTATION L EARNING
Due to the high complexity of the policy space in RL
[Table]
it is difficult to directly optimize the policy in this space.
Therefore, a possible way is to map the original high-dimensional policy space to a lowdimensional representation space and optimizing the policy in this representation space.
In this way, one can greatly reduce the difficulty of searching for the optimal policy in the policy space.
Some recent works focus on the policy representation and use the policy representation as auxiliary input to the value function.
For example, PDVF (Raileanu et al., 2020) used this idea in meta-RL.
It’s trained by using state and policy representations to predict corresponding action.
Harb et al.
(2020) uses the actions that policy sampled in probing states as policy representations.
There are also some articles (Faccio et al., 2021; Tang et al., 2020) that use policy network parameters as policy representations.
Then given the policy representation, by gradient ascent, the policy representation can be updated along the direction of increasing the values of value function.
In this way, one can find an approximated optimal policy without the overhead of interacting with the environment.
However, how to characterize the policy in a reasonable and efficient way and whether the policy representation can improve the meta-RL is still an open challenge.
In contrast to policy representation, environment representation is usually applied in context-based meta-RL.
There are two common ways to extract envionment representations.
One way is to predict future information based on the information of current steps (Raileanu et al., 2020; Guo et al., 2020).
The other way is to extract representations based on contrastive learning (Fu et al., 2021).
The goal of contrastive learning is to learn an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart.
A metric is then given to measure the similarity of the feature representations of these samples and a discriminative encoder is trained to group data with similar embedding while distinguish data with dissimilar embedding.
Then the learned representations are used for downstream tasks.
PAnDR also use contrastive learning to extract representations.
We follow the training framework and the infoNCE loss used in CURL (Laskin et al., 2020).
But we take a different approach than CURL to construct positive and negative examples.
Details are given in Sec. 3.
[Equation]
[Table]
[-K i =
-][Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Figure] [Table]
[Table]
[Figure]
[Table]
[Figure]