Learning to Constrain Policy Optimization with Virtual Trust Region
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We introduce a constrained optimization method for policy gradient reinforcementlearning, which uses a virtual trust region to regulate each policy update.
In addition to using the proximity of one single old policy as the normal trust region, wepropose forming a second trust region through another virtual policy representinga wide range of past policies.
We then enforce the new policy to stay closer tothe virtual policy, which is beneﬁcial if the old policy performs poorly.
More importantly, we propose a mechanism to automatically build the virtual policy froma memory of past policies , providing a new capability for dynamically learningappropriate virtual trust regions during the optimization process.
Our proposedmethod, dubbed Memory-Constrained Policy Optimization (MCPO), is examinedin diverse environments, including robotic locomotion control, navigation withsparse rewards and Atari games, consistently demonstrating competitive performance against recent on-policy constrained policy gradient methods
Introduction
Deep reinforcement learning (RL) is the current workhorse in machine learning.
Using neural networks to approximate value and policy functions enables classical approaches such as Q-learning[25] and policy gradient [19] to achieve promising results on many challenging problems such as
Go, Atari games and robotics [17, 11, 8, 9].
Compared to Deep Q-learning, deep policy gradient(PG) methods are often more ﬂexible and applicable to discrete and continuous action problems.
However, these methods tend to suffer from high sample complexity and training instability sincethe gradient may not accurately reﬂect the policy gain when the policy changes substantially [5].
This is exacerbated for deep policy networks where numerous parameters need to be optimized, andminor updates in parameter space can lead to considerable changes in policy space.
To address this issue, one solution is to regularize each policy update by restricting the Kullback–
Leibler (KL) divergence between the new policy and the previous one, which can guarantee monotonic policy improvement [13].
However, jointly optimizing the approximate advantage functionand the KL term does not work in practice [13].
Therefore, Schulman et al.
(2015) proposed Trust
Region Policy Optimization (TRPO) to constrain the new policy within a KL divergence radius,which requires second-order gradients.
Alternatives such as Proximal Policy Optimization (PPO)[15] use a simpler ﬁrst-order optimization with adaptive KL or clipped surrogate objective whilestill maintaining the reliable performance of TRPO.
Recent methods recast the problem through anew lens using Expectation-Maximization or Mirror Descent Optimization, and this also results inﬁrst-order optimization with KL divergence term in the loss function [1, 18, 27, 20].
An issue with the above methods is that the previous policy used to restrict the new policy may besuboptimal and thus unreliable in practice.
For example, due to stochasticity and approximations,the new policy may fall into a local optimum even under trust-region optimizations.
Then in the next update, this policy will become the “previous” policy, and will continue pulling the next policyto stay in the local optimum, thus slowing down the training progress.
For on-policy methods usingmini-batch updates like PPO, the situation is more problematic as the “previous” policy is deﬁnedas the old policy to collect data, which can be either very far or close to the current policy.
There isno guarantee that the old policy deﬁnes a reasonable trust region for regulating the new policy.
In this paper, we propose a novel constrained policy iteration procedure, dubbed MemoryConstrained Policy Optimization (MCPO), wherein a virtual policy representing memory of pastpolicies regularizes each policy update.
The virtual policy forms a virtual trust region, attracting thenew policy more when the old policy performs badly, which prevents the optimization from fallinginto local optimum caused by the old policy of poor quality.
As such, we measure 2 KL divergencescorresponding to the virtual and old policy, and assign different weights to the two KL terms inbuilding the objective function.
The weights are computed dynamically based on the performanceof the two policies (the higher performer yields higher weights).
In contrast to prior works using heuristics (e.g. running average or mean of past policies) to formadditional trust regions [24], we argue that the virtual policy should be determined dynamically tomaximize the performance on current training data.
Thus we store policies in a policy memory, andlearn to extract the most relevant ones to the current context.
We sum the past policies in a weightedmanner wherein the weights are generated by a neural network–named the attention network , whichtakes the information of the current, the old and the last virtual policy as the input.
The attentionnetwork is optimized to maximize the approximate expected advantages of the virtual policy.
Wejointly optimize the policy and attention networks to train our system, alternating between samplingdata from the policy and updating the networks in a mini-batch manner.
We verify our proposed MCPO through a diverse set of experiments and compare our performancewith that of recent constrained policy optimization baselines.
In our experiment on classical controltasks, amongst tested models, MCPO consistently achieves better performance across tasks andhyperparameters.
Our testbed on 6 Mujoco tasks shows that MCPO with a big policy memoryis performant where the attention network plays an important role.
We also demonstrate MCPO’scapability of learning efﬁciently on sparse reward and high-dimensional problems such as navigationand Atari games.
Finally, our ablation study highlights the necessity of MCPO’s components suchas the virtual policy and the attention network.
Background: Policy Optimization with Trust Region
In this section, we brieﬂy review some fundamental constrained policy optimization approaches.
A general idea is to force the new policy π θ to be close to a recent policy π θ old .
In this paper, weusually refer to a policy via its parameters (i.e. policy θ means policy π θ ).
We also use ﬁnite-horizonestimators for the advantage with discount factor γ ∈ (0 , 1) and horizon T .
Conservative Policy Iteration (CPI)
This method starts with a basic objective of policy gradientalgorithms, which is to maximize the expected advantage ˆ A t .
[Equation]
where the advantage ˆ A t is a function of returns collected from ( s t , a t ) by using π θ old (see Appendix
A.2) and ˆ E t [ · ] indicates the empirical average over a ﬁnite batch of data.
To constrain policy updates,the new policy is a mixture of the old and the greedy policy: ˜ θ = argmax L CPI ( θ ) .
That is,θ = αθ old + (1 − α )
˜ θ where α is the mixture hyperparameter [5].
As the data is sampled from theprevious iteration’s policy θ old , the objective needs importance sampling estimation.
Hereafter, wedenote π θ ( a t | s t ) π θold ( a t
| s t ) as τ t ( θ ) for short.
KL-Regularized Policy Optimization To enforce the constraint
, one can jointly maximize the advantage and minimize KL divergence between the new and old policy, which ensures monotonicimprovement [13].
[Equation]
where β is a hyperparameter that controls the update conservativeness, which can be ﬁxed (KL
Fixed) or changed (KL Adaptive) during training [15].
Trust Region Policy Optimization (TRPO)
The method optimizes the expected advantage withhard constraint [13].
This is claimed as a practical implementation, less conservative than the theoretically justiﬁed algorithm using KL regularizer mentioned above.
[Equation]
where δ is the KL constraint radius.
Proximal Policy Optimization (PPO)
PPO is a family of constrained policy optimization, whichuses ﬁrst-order optimization and mini-batch updates including KL Adaptive and clipped PPO.
Inthis paper, we use PPO to refer to the method that limits the change in policy by clipping the lossfunction (clipped PPO)
The objective L PPO is deﬁned as
[Equation]
where (cid:15) is the clip hyperparameter.
In all the above methods, θ is the currently optimized policy, which is also referred to as the currentpolicy.
θ old represents a past policy, which can be one or many update steps before the currentpolicy.
In either case, the rule to decide θ old is ﬁxed throughout training.
If θ old is suboptimal, itis unavoidable that the following updates will be negatively impacted.
We will address this issue inthe next section.
Memory-Constrained Policy Optimization
In trust-region policy gradient methods with mini-batch updates such as PPO, the old policy θ old isoften the last “sampling” policy for collecting observations from the environment.
This policy is,by design, ﬁxed during updates of the main policy until the next interaction with the environment.
This means θ old may not immediately precede the current policy θ but may be from many (update)steps before.
Since θ old is not up-to-date and could possibly be not good, using it to constrain thecurrent policy θ can cause the suboptimal update of θ .
Similarly, the immediately preceding policycould also be poor in quality and thus, using it as θ old could be detrimental to the optimization.
To tackle this issue, we propose to learn to constrain θ towards a weighted combination of multiple past policies besides θ old .
We argue that we can optimize the attention network to ensure thecombined policy is optimal.
Below is formal description of our method.
3.1 Virtual Policy
Computing the virtual policy via past policies The virtual policy should be determined basedon the past policies and their contexts such as quality, distance or entropy.
A simple strategy such astaking average of past policies is likely suboptimal as the quality of these policies vary and some canbe irrelevant to the current learning context.
Let ψ be the weighted combination of M past policies{ θ i } Mi =1 , which we refer to as a “virtual” policy for naming convenience, ψ is computed as follows:
[Equation]
where f ϕ is a neural network parameterized by ϕ which outputs softmax attention weights over the
M past policies, v is a “context” vector capturing different relations among the current policy θ , thelast sampling policy θ old , and the last virtual policy ψ old .
We build the context by extracting speciﬁcfeatures: pair-wise distances between policies, the empirical returns of these policies, policy entropyand value losses (details in Appendix Table 4).
Intuitively, these features suggest which virtualpolicy will yield high performance (e.g., a virtual policy that is closer to the policy that obtainedhigh return and low value loss).
The details of f ϕ training will be given in Sec. 3.2.
Algorithm
Require: A policy buffer M , an initial policy π θ old .
T , K , B are the learning horizon, number of update epochs, and batch size, respectively.
Storing past policies with diversity-promoting writing We use a memory buffer M to store pastpolicies.
We treat M as a queue with maximum capacity N , which means a new policy θ will beadded to the end of M and if M is full, the oldest policy will be discarded.
However, we do notadd any new policy θ to M unconditionally but only when θ satisﬁes our “diversity-promoting”condition.
Let D ( a, b ) =
ˆ E t [ KL [ π a ( ·| s t ) , π b ( ·| s t )]] denote the “distance” between 2 policies π aand π b , the diversity-promoting writing is deﬁned as:
[Equation]
where D ( θ old , ψ ) serves as a threshold.
This condition makes sure that the policy to be added is farenough from ψ .
It is reasonable because if θ is too similar to ψ (in regard of θ old ), the advantage ofstoring multiples past policies in order to ﬁnd a good one will disappear.
We will elaborate more onthis in Sec. 4.5.
3.2 Policy Optimization with Two Trust RegionsOptimizing the policy parameters θ
During policy optimization, we make use of both θ old and 135 ψ to constrain θ .
Hence, θ old and ψ form 2 trust regions, and we aim to enforce the new policy to becloser to the better one.
To this end, we propose to learn a new policy θ by maximizing the followingobjective function:
[Equation]
where β is the weight balancing between the main objective and the policy constraints, α t is theweight balancing between constraining θ towards ψ and θ towards θ old .
The expectation is estimatedby taking average over t in a mini-batch of sampled data.
We note that in the early stage of learning,the attention network is not trained well, and thus the quality of the virtual policy ψ may be worsethan θ old .
In the long-term, ψ will get better and tend to provide a better trust-region.
Hence,we need to use both trust regions to ensure maximal performance at any learning stage.
We canalso prove that using our proposed two trust regions guarantees monotonic policy improvement (see
Appendix. C).
Below we introduce a mechanism to automatically determine the contribution of thetwo trust regions through computing α and β coefﬁcients.
Intuitively, if the virtual policy is better than the old policy, the new policy should be kept close tothe virtual policy and vice versa.
Hence, α t should be proportional to the contribution of ψ to theﬁnal performance with regard to that of θ old .
Thus, we deﬁne: α t ( ·| s t
[Equation]
= [Equation] ( R t ( θ old ))
[Table]
where R t ( ψ ) , R t ( θ old ) are the estimated returns corresponding to ψ and θ old , respectively.
Weestimate R t ( · ) via weighted importance sampling, that is, R t ( · ) =
τ t ( · ) ˆ A t where both are computedusing the same s t .
We also dynamically adjust β by switching between 2 values β min and β max( 0 < β min < β max ) as follows:
[Equation]
4) where D ( · , · ) is again the KL “distance”.
The reason behind this update of β is to encourage strongerenforcement of the constraint when θ is too far from θ old .
Unlike using a ﬁxed threshold d targ tochange β (e.g in KL Adaptive, if D ( θ old , θ ) >
d targ , increase β [15]), we make use of ψ as areference for selecting β .
This allows a dynamic threshold that varies depending on the currentlearning.
We name this mechanism as switching- β rule.
Optimizing the attention network parameters ϕ
To encourage the attention network f ϕ (Eq.
1)to produce the optimal attention weights over past policies, we maximize the following objective:
[Equation]
5) which is the approximate expected return w.r.t.
ψ ϕ .
We write ψ ϕ to emphasize that ψ is a functionof ϕ (as formulated in Eq. 1).
We use gradient ascent to update ϕ by backpropagating the gradient∂ L 2 ∂ϕ through the attention network.
By learning f ϕ , we can guarantee that the virtual policy ψ is the best combination of past policiesin terms of return.
Note that for on-policy learning setting, optimizing “soft” attention weights isusually more robust than searching for the best past policy in M because the estimation of theexpected return using current data samples can be noisy and is not always reliable.
Also, searchingis computationally more expensive than using the attention.
Final Objective We train the whole system by maximizing the following objective:
[Equation]
(6) where MCPO stands for Memory-Constrained Policy Optimization .
We optimize θ and ψ alternatelyby ﬁxing one and learning the other.
We implement MCPO using minibatch update procedure [15].
MCPO Pseudocode is given in in Algo 1.
For notational simpliﬁcation, the algorithm uses 1 actor.
Experimental results
In our experiments, we show our optimization scheme is superior in different aspects: hyperparameter sensitivity, sample efﬁciency and consistent performance in continuous and discrete actionspaces.
The main baselines are recent on-policy constrained methods that use ﬁrst-order optimization, in which most of them employ KL terms in the objective function.
They are KL Adaptive,
KL Fixed, PPO [15], MDPO [20], VMPO [18] and TRGPPO [23] .
We also include second-ordermethods such as TRPO [13] and ACKTR [26].
Across experiments, for MCPO, we ﬁx β max = 10 ,β min = 0 .
01 and only tune N .
More details on the baselines and tasks are given in Appendix B.1.
[Figure] [Figure]
e d S l
[Figure]
4.1 Classical Control
In this section, we compare MCPO to other ﬁrst-order policy gradient methods (KL Adaptive, KL
Fixed, PPO, MDPO and VMPO) on 3 classical control tasks:
Pendulum, LunarLander and BipedalWalker, which are trained for one, one and ﬁve million environment steps, respectively.
Here, weare curious to know how the model performance ﬂuctuates as the hyperparameters vary.
For eachmodel, we choose one hyperparameter that controls the conservativeness of the policy update, andwe try different values for the signature hyperparameter while keeping the others the same.
Forexample, for PPO, we tune the clip value (cid:15) ; for KL Fixed we tune β coefﬁcient and these possiblevalues are chosen following prior works.
For our MCPO, we tune the size of the policy memory
N (5, 10 and 40).
We do not try bigger policy memory size to keep MCPO running efﬁciently (see
Appendix B.2 for details).
Table 1 reports the results of MCPO and 5 baselines with the best hyperparameters.
For these simple tasks, tuning the hyperparameters often helps the model achieve at least moderate performance.
However, models like KL Adaptive and VMPO cannot reach good performance despite being tuned.
PPO shows good results on LunarLander and BipedalWalker, yet underperforms others on Pendulum.
Interestingly, if tuned properly, the vanilla KL Fixed can show competitive results comparedto PPO and MDPO in BipedalWalker.
Amongst all, our MCPO with suitable N achieves the bestperformance on all tasks.
Remarkably, its performance does not ﬂuctuate much as N changes from5 to 40, often obtaining good and best results.
We speculate that for simple tasks, either with smallor big M , the policy attention can always ﬁnd optimal virtual policy ψ , and thus, ensures stableperformance of MCPO.
On the contrary, other methods observe a clear drop in performance ashyperparameters change (see Appendix B.2 for learning curves and the full table).
4.2 Navigation
Here, we validate our method on sparse reward environments using MiniGrid library [3].
In particular, we test MCPO and other baselines (same as above) on Unlock and UnlockPickup tasks.
In thesetasks, the agent navigates through rooms and picks up objects to complete the episode.
The agentonly receives reward +1 if it can complete the episode successfully.
For sample efﬁciency test, wetrain all models on Unlock (ﬁnd key and open the door) and UnlockPickup (ﬁnd key, open the doorand pickup an object), for only 100,000 and 1 million environment steps, respectively.
The modelsuse the best conservative hyperparameters found in the previous task (more in Appendix B.3).
[Table]
Appendix’s Fig.
2 shows the learning curves of examined models on these two tasks.
For Unlocktask, except for MCPO and VMPO, 100,000 steps seem insufﬁcient for other models to learn usefulpolicies.
When trained with 1 million steps on UnlockPickup, the baselines can ﬁnd better policies, yet still underperform MCPO.
Here VMPO shows faster learning progress than MCPO at thebeginning, however it fails to converge to the best solution.
Our MCPO is the best performer, consistently ending up with average return of 0.9 (90% of episodes ﬁnished successfully).
To achievesample-efﬁciency, MCPO needs to store and search for good policies (rarely found in sparse rewardproblems), and adhere to it during optimization.
MCPO’s success may be attributed to utilizing thevirtual policy that has the highest return.
To illustrate how the virtual policy supports MCPO’s performance, we analyze the relationshipsbetween the old ( θ old ), the virtual policy ( ψ ) and the policies stored in M ( θ j ) throughout Unlocktraining.
Fig.
1 (a) plots the location of these policies over a truncated period of training (fromupdate step 5160 to 7070).
Due to diversity-promoting rule, the steps where policies are added to
M can be uneven (ﬁrst row-red lines), often distributed right after the locations of the old policy(second row-green lines).
We query at 10 -th step behind the old policy (fourth row-cyan lines) toﬁnd which policy in M has the highest attention (third row-yellow lines, linked by blue arrows).
As shown in Fig.
1 (a) (second and third row), the attended policy, which mostly resembles ψ , canbe further or closer to the query step than the old policy depending on the training stage.
Sincewe let the attention network learn to attend to the policy that maximizes the advantage of currentmini-batch data, the attended one is not necessarily the old policy.
The choice of the chosen virtual policy being better than the old policy is shown in Fig.
1 (b) wherewe collect several checkpoints of virtual and old policies across training and evaluate each of themon 10 testing episodes.
Here using ψ to form the second KL constraint is beneﬁcial as the newpolicy is generally pulled toward a better policy during training.
That contributes to the excellentperformance of MCPO compared to other single trust-region baselines, especially KL Fixed and
Adaptive, which are very close to MCPO in term of objective function style.
4.3 Mujoco
Next, we examine MCPO and some trust-region methods from the literature that are known tohave good performance on continuous control problems: TRPO, PPO, MDPO and TRGPPO (animproved version of PPO).
To understand the role of the attention network in MCPO, we design avariant of MCPO ( N = 40 ): Mean ψ , which simply constructs ψ by taking average over policyparameters in M .
This baseline represents prior heuristic ways of building trust-region from pastpolicies [24].
We pick 6 hard Mujoco tasks and train each model for 10 million environment steps.
We report the results of best tuned models in Table 2 (see Appendix B.4 for full results).
The results show that MCPO consistently achieves good performance across 6 tasks, where it outperforms others signiﬁcantly in HalfCheetah, Hopper, and HumanoidStandup.
In other tasks, MCPO isthe second best, only slightly earning less score than the best one in Walker2d and Ant.
The variant
Mean ψ shows reasonable performance for the ﬁrst 4 tasks, yet almost fails to learn on the last two.
Thus, mean virtual policy can perform badly.
To understand the effectiveness of the attention network, we visualize the attention pattern of MCPOon the last two tasks and on Hopper-a task that Mean ψ performs well on.
Fig.
1 (c) illustrates thatfor the ﬁrst two harder tasks, MCPO gradually learns to favor older policies in M ( j ∗ > 35) , whichputs more restriction on the policy change as the model converges.
This strategy seems criticalfor those tasks as the difference in average return between learned ψ and Mean ψ is huge in these
[Table]
cases.
On the other hand, on Hopper, the top attended slots are just above the middle policies in M( j ∗ ∼ 25) , which means this task prefers an average restriction.
As in most cases, MCPO with bigpolicy memory ( N = 40 , more conservativeness) is beneﬁcial.
For those tasks where MCPO doesnot show clear advantage, we speculate that conservative updates are less important.
We also visualize α t for HalfCheetah task, and observe an increasing trend across training steps (see
Appendix Fig. 3).
As the attention network gets trained, the virtual policy becomes better and cancomplement the old policy when the latter performs worse.
Hence, on average, the weight α t tendsto be bigger overtime.
4.4 Atari Games
As showcasing the robustness of our method to high-dimensional inputs, we execute an experimenton a subset of Atari games wherein the states are screen images and the policy and value functionapproximator uses deep convolutional neural networks.
We choose 9 typical games (6 were introduced in [10] and 3 randomly chosen) and benchmark MCPO against PPO, ACKTR, VMPO and
PRGPPO, training all models for only 10 million environment steps.
In this experiment, MCPOuses N = 10 and other baselines’ hyperparameters are selected based on the original papers (see
Appendix B.5).
As seen in Table 3, MCPO is signiﬁcantly better than other baselines in terms of both mean andmedian normalized human score.
The learning curves of all models are given in Appendix Fig.4.
To conﬁrm MCPO maintains the leading performance with more training iterations, we traincompetitive models for 40 million frames for the ﬁrst 6 games (Appendix Fig. 5).
The results showthat MCPO still outperforms other baselines after 40M frames.
4.5 Ablation Study
Finally, we verify MCPO’s 3 components: virtual policy (in Eq. 3), switching- β (Eq. 4) anddiversity-promoting rule (Eq. 2).
We also conﬁrm the role of choosing the right memory size
Nand learning to attend to the virtual policy ψ .
In the task BipedalWalkerHardcore (OpenAI Gym),we train MCPO with different conﬁgurations for 50M steps.
First, we tune N (5,10 and 40) usingthe normal MCPO with all components on and ﬁnd that N = 10 is the best.
Keeping N = 10 , weablate or replace our component with an alternative and report the ﬁndings as follows.
Virtual policy: To show the beneﬁt of pushing the new policy toward the virtual policy, we implement a variant of MCPO that does not use ψ ’s KL term in Eq.
3 (a.k.a.
α t = 0 ).
This variantunderperforms the normal MCPO by a margin of 100 return.
Switching- β : The results show thatcompared to the annealed β strategy adopted from MDPO, our switching- β achieves signiﬁcantlybetter results with about 50 return score higher.
Diversity-promoting writing
: We compare ourproposal with the vanilla approach that adds a new policy to M at every update step (frequent writing) and another version that writes to M every interval of 10 update steps (uniform writing).
Bothfrequent and uniform writing show slow learning progress, and ends up with negative rewards.
Perhaps, frequently adding policies to M makes the memory content similar, hastening the removal ofolder, yet maybe valuable policies.
Uniform writing is better, yet it can still add similar policies to
M and requires additional effort for tuning the writing interval.
Learned ψ : To benchmark, we tryalternatives: (1) using Mean ψ and (2) only using half of the features in v context to generate ψ (Eq.1).
The results conﬁrm that the Mean ψ is not a strong baseline for this task, only reaping moderaterewards.
Using less features for the context causes information loss, hinders generations of usefulψ , and thus, underperforms the full-feature version by a margin of 50 return.
For completeness, we also compare our methods to heavily tuned baselines and conﬁrm that the normal MCPO ( N = 10 )is the best performer (see Appendix B.6).
Related work
A framework for model-free reinforcement learning with policy gradient is approximate policy iteration (API), which alternates between estimating the advantage of the current policy, and updatingthe current policy’s parameters to obtain a new policy by maximizing the expected advantage [2, 19].
Theoretical studies have shown that constraining policy updates is critical for API [5, 13, 16, 22].
An early attempt to improve API is Conservative Policy Iteration (CPI), which sets the new policyas a stochastic mixture of previous greedy policies [5, 12, 21].
Our paper differs from these worksin three aspects: (1) we do not directly set the new policy to the mixture, rather, we use a mixture ofpreviously found policies (the virtual policy) to deﬁne the trust region constraining the new policyvia KL regularization; (2) our mixture can consist of more than 2 modes, and thus using multiplemixture weights (attention weights); (3) we use the attention network to learn these weights.
Also motivated by Kakade et al.
(2002), TRPO extends the theory to general stochastic policies,rather than just mixture polices, ensuring monotonic improvement by combining maximizing theapproximate expected advantage with minimizing the KL divergence between two consecutive policies [13].
Arguing that optimizing this way is too conservative and hard to tune, the authors reformulate the objective as a constrained optimization problem to solve it with conjugate gradientand line search.
To simplify the implementation of TRPO, Schulman et al.
(2017) introduces ﬁrstorder optimization methods and code-level improvement, which results in PPO–an API method thatoptimizes a clipped surrogate objective using minibatch updates.
Constrained policy improvement can be seen as Expectation-Maximization algorithms where minimizing the KL-term corresponds to the Expectation step, which can be off-policy [1] or on-policy[18].
From the mirror descent perspective, several works also use KL divergence to regularize policyupdates [27, 20, 16].
A recent analysis also points out the advantages of using KL term as a regularizer over a hard constraint [6].
Some other works improve PPO with adaptive clip range [23] oroff-policy data [4].
We instead advocate using 2 trust regions and apply our idea to a weaker backbone: PPO-penalty.
Our method is also different from [4] as we do not use off-policy data/gradientsto update the policy.
Overall, our approach shares similarities with them where we also jointly optimize the approximate expected advantage and KL constraint terms for multiple epochs of minibatchupdates.
However, we propose a novel dynamic virtual policy to construct the second trust region asa supplement to the traditional trust region deﬁned by the old or previous policy.
Discussion
We have presented Memory-Constrained Policy Optimization, a new method to regularize each policy update with two-trust regions with respect to one single old policy and another virtual policyrepresenting multiple past policies.
The new policy is encouraged to stay closer to the region surrounding the policy that performs better.
The virtual policy is determined online through a learnedattention to a memory of past policies.
Compared to other trust-region optimizations, MCPO showsbetter performance in many environments without much hyperparameter tuning.
Limitations Our method introduces several new components and hyperparameters such as N andβ .
Due to compute limit, we have not tuned β extensively and thus the reported results may not bethe best performance that our model can achieve.
Although we ﬁnd the default values β max = 10 ,β min = 0 .
01 work well across all experiments in this paper, we recommend adjustments if usersapply our method to novel domains.
Negative Societal Impacts Our work aims to improve optimization in RL to reduce the samplecomplexity of RL algorithms.
This aim is genuine, and we do not think there are immediate harmfulconsequences.
However, we are aware of potential problems such as unsafe exploration committedby the agent (e.g. causing accidents) in real-world environments (e.g. self-driving cars).
Finally,malicious users can misuse our method for unethical purposes, such as training harmful RL agentsand robots.
This issue is typical for any machine learning algorithm, and we will do our best toprevent it from our end.