Discovered Policy Optimisation
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The last decade has been revolutionary for reinforcement learning (RL) — itcan now solve complex decision and control problems.
Successful RL methodswere handcrafted using mathematical derivations, intuition, and experimentation.
This approach has a major shortcoming— it results in speciﬁc solutions to the
RL problem, rather than a protocol for discovering efﬁcient and robust methods .
In contrast, the emerging ﬁeld of meta-learning provides a toolkit for automaticmachine learning method optimisation, potentially addressing this ﬂaw.
However,black-box approaches which attempt to discover RL algorithms with minimal priorstructure have thus far not been successful.
Mirror Learning , which includes RLalgorithms, such as PPO, offers a potential framework.
In this paper we explorethe Mirror Learning space by meta-learning a “drift” function.
We refer to theresult as Learnt Policy Optimisation (LPO).
By analysing LPO we gain originalinsights into policy optimisation which we use to formulate a novel, closed-form
RL algorithm, Discovered Policy Optimisation (DPO).
Our experiments in Braxenvironments conﬁrm state-of-the-art performance of LPO and DPO, as well astheir transfer to unseen settings.
Introduction
Recent advancements in deep learning have allowed reinforcement learning (RL) algorithms tosuccessfully tackle large-scale problems.
Early deep RL algorithms, such as A2C [21] and DDPG[18] follow the scheme of generalized policy iteration (GPI), where the RL agent alternates betweenpolicy evaluation and policy improvement phases [30].
Namely, they train a critic neural network thatestimates the value function and an actor network which models the agent’s policy and use theseto estimate the update direction with policy gradients of the expected return.
Another approach isthat of trust-region learning (TRL) methods, which construct a surrogate objective that they optimisewithin a small region around the current policy [27], aiming at guaranteeing stability of the policyiterates.
Indeed, TRL-inspired algorithms, such as TRPO [27] and PPO [28], have been amongthe most widely used methods [2] and are known for their performance and stability.
Nevertheless,although these research threads have delivered a handful of successful techniques, their design relieson concepts handcrafted by humans, rather than discovered in a learning process .
As a possibleconsequence, these methods often suffer from various ﬂaws, such as the brittleness to hyperparametersettings [27, 11], and a lack of optimality guarantees.
Unfortunately, the most promising alternative approach, algorithm discovery , thus far has been atough nut to crack.
First of all, techniques that enable optimisation of algorithm components in theouter loop process—meta RL [26, 7]—are at an early stage of development, and are vastly morecompute intensive than the RL problem in the inner loop [34].
Second, to discover more complexalgorithms, one needs to solve a more sophisticated, and computationally taxing meta-optimisation problem.
As a result, most methods discover simple algorithms at best [22].
Recently, Mirror Learning [16], a new theoretical framework, introduced an inﬁnite space of provablycorrect algorithms, all of which share the same template.
In a nutshell, a Mirror Learning algorithm isdeﬁned by four attributes, but in this work we focus on the drift function .
A drift function guides theagent’s update, usually by penalising large changes.
Any mirror learning algorithm provably achievesmonotonic improvement of the return, and converges to an optimal policy [16].
Popular RL methodssuch as TRPO and PPO are instances of Mirror Learning.
In this paper, we use meta-learning to discover a new state-of-the-art (SOTA) RL algorithm withinthe Mirror Learning space.
Our algorithm thus inherits theoretical convergence guarantees byconstruction.
Speciﬁcally we parameterise a drift function with a neural network, which we thenmeta-train using evolution strategies [25, ES].
The outcome of this meta-training is a speciﬁc mirrorlearning algorithm which we name Learnt Policy Optimisation (LPO).
However, we take a stepfurther and learn from LPO about important RL concepts that it has discovered by visualising andanalysing its optimisation objective.
While having a neural network representation of a novel, high-performing drift function is a greatﬁrst step, our next goal is to understand the relevant algorithmic features of this drift function.
Building upon these insights we propose a new, closed-form algorithm which we name — Discovered
Policy Optimisation (DPO).
We evaluate LPO and DPO in the Brax [8] continuous control environments, where they obtain superior performance compared to PPO.
Importantly, both LPO and
DPO generalise to environments that were not used for training.
To our knowledge, DPO is the ﬁrsttheoretically-sound, deep RL algorithm that was discovered via meta-learning.
Related Work
In the last few years, signiﬁcant effort has been directed at developing effective RL algorithms throughboth algorithmic and implementational advances.
Fujimoto et al.
[9, TD3] combine DDPG policytraining with estimates of pessimistic Bellman targets from a separate critic.
Hsu et al.
[14] stabilisethe, previously unsuccessful [28], KL-penalised version of PPO and improve its robustness throughnovel policy design choices.
Haarnoja et al.
[12] introduce a mechanism that automatically adjuststhe temperature parameter of the entropy bonus in SAC.
However, none of these hand-crafted effortssucceeds in fully mitigating common RL pathologies, such as sensitivity to hyperparameter choicesand lack of domain generalisation [4].
This motivates radically expanding the RL algorithm searchspace through automated means.
Indeed, recent work explores applications of meta-RL techniques in guiding RL algorithm discoveryand design.
RL 2 equips a learning agent with a recurrent neural network conditioning on transitionsbetween tasks, and adapts the agent’s behaviour to the current environment [4].
Similarly, a MAMLagent learns policy meta-parameters which can adapt to any task with a few steps of gradient descent[7].
Neither RL 2 nor MAML, however, go beyond improving the robustness of ultimately handcrafted algorithms.
To overcome this, Xu et al.
[34, FRODO] introduce an actor-critic methodthat adjusts its hyperparameters online using meta-gradients that are updated with every few inneriterations.
Similarly, STAC [35] uses implementation techniques from IMPALA [5] and auxiliaryloss-guided meta-parameter tuning to further improve on FRODO.
Such advances have inspired extending meta-gradient RL techniques to more ambitious objectives,including the discovery of algorithms ab initio .
Notably, Oh et al.
[22] succeeded in meta-learning an
RL algorithm, LPG, that can solve simple tasks efﬁciently without explicitly relying on concepts suchas value functions and policy gradients.
Similarly, Evolved Policy Gradients [13, EPG] meta-trains apolicy loss network function with Evolution Strategies [25, ES].
Although EPG surpasses PPO inaverage performance, it suffers from much larger variance [13].
MetaGenRL [15] instead meta-learnsthe loss function for deterministic policies which are inherently less affected by estimators’ variance[29].
MetaGenRL, however, fails to improve upon DDPG [18] in terms of performance, despitebuilding up on it.
Neither EPG nor MetaGenRL have resulted in the discovery of novel analytical RLalgorithms, perhaps due to the limited interpretability of the loss functions learnt.
Lastly, Co-Reyeset al.
[3], Garau-Luis et al.
[10] and Alet et al.
[1] discover and improve standard RL conventions byevolving, symbolically, algorithms represented as graphs, which leads to improved performance insimple tasks.
However, none of those trained-from-scratch methods inherit correctness guarantees,limiting our certainty of the generality of their abilities.
In contrast our method, LPO, is metadeveloped in a Mirror Learning space [16], where every algorithm is guaranteed convergence to an optimal policy.
As a result to this construction, meta-training of LPO is easier than that of methodsthat learn “from scratch”, and achieves great performance across environments.
Furthermore, thanksto the clear meta-structure of Mirror Learning, LPO is interpretable, and lets us discover new learningstrategies.
This lets us introduce DPO—an efﬁcient algorithm with a closed-form formulation thatexploits the discovered learning concepts.
Background
In this section, we introduce the essential concepts required to comprehend our contribution—the
RL and Meta-RL problem formulations, as well as the Mirror Learning and Evolution Strategiesframeworks for solving them, respectively.
3.1 Reinforcement Learning
Formulation We formulate the reinforcement learning (RL) problem as a Markov decision process(MDP)
[30] represented by a tuple h S , A , r, P,  , d i which deﬁnes the experience of a learning agentas follows: at time step t 2 N , the agent is at state s t 2 S (where s 0 ⇠ d ) and takes an action a t 2 A 104 according to its stochastic policy ⇡ ( ·| s t ) , which is a member of the policy space ⇧ .
The environmentthen emits the reward r ( s t , a t ) and transits to the next state s t +1 drawn from the transition function,s t +1 ⇠ P ( ·| s t , a t ) .
The agent aims to maximise the expected value of the total discounted return,
[Equation]
The agent guides its learning process with value functions that evaluate the expected return conditionedon states or state-action pairs
[Equation]
The function that the agent is concerned about most is the advantage function , which computesrelative values of actions at different states,
[Equation]
Policy Optimisation
In fact, by updating its policy simply to maximise the advantage function atevery state, the agent is guaranteed to improve its policy, ⌘ ( ⇡ new )  ⌘ ( ⇡ old ) [30].
This fact, althoughrequiring a maximisation operation that is intractable in large state-space settings tackled by deep
RL (where the policy ⇡ ✓ is parameterised by weights ✓ of a neural network), has inspired a rangeof algorithms that perform it approximately.
For example, A2C [21] updates the policy by a step ofpolicy gradient (PG) ascent
[Equation]
estimated from a batch of B transitions.
Nevertheless, such simple adoptions of generalized policyiteration [30, GPI] suffer from large variance and instability [36, 29, 28].
Hence, methods thatconstrain (either explicitly or implicitly) the policy update size are preferred [27].
Among the mostpopular, as well as successful ones, is Proximal Policy Optimization [28, PPO], inspired by trustregion learning [27], which updates its policy by maximising the PPO-clip objective,
[Equation]
where the clip ( · , 1 ± ✏ ) operator clips (if necessary) the input so that it stays within [1  ✏ , 1 + ✏ ]interval.
In deep RL, the maximisation oracle in Equation (1) is implemented through a few steps ofgradient ascent on policy parameters.
Meta -RL The above approaches to policy optimisation rely on human-possessed knowledge, andthus are limited by humans’ understanding of the problem.
The goal of meta-RL is to optimise thealgorithm of the learning agent.
Formally, suppose that an RL algorithm alg
, parameterised by , trains an agent for K iterations.
Meta-RL aims to ﬁnd the meta-parameter  =
⇤ such that theexpected return of the output policy, E [ ⌘ ( ⇡ K ) | alg  ] , is highest.
3.2 Mirror Learning
A mirror-learning agent [16], in addition to value functions, has access to the following operators: thedrift function D ⇡ k ( ⇡ | s ) which, intuitively, evaluates the signiﬁcance of change from policy ⇡ k to ⇡at state s ; the neighbourhood operator N ( ⇡ k ) which forms a region around the policy ⇡ k ; as well assampling and drift distributions  ⇡ k ( s ) and ⌫ ⇡⇡ k ( s ) over states.
With these deﬁned, a mirror-learningalgorithm updates an agent’s policy by maximising the mirror objective
[Equation]
If, for all policies ⇡ and ¯ ⇡ , the drift function satisﬁes the following conditions:
It is non-negative everywhere and zero at identity D ⇡ k ( ⇡ | s )
D ⇡ k ( ⇡ k | s )
= 0 ,
Its gradient with respect to ⇡ is zero at ⇡ = ⇡ k ,
then the Mirror Learning algorithm attains the monotonic improvement property, ⌘ ( ⇡ k +1 )  ⌘ ( ⇡ k ) ,and converges to the optimal return, ⌘ ( ⇡ k ) !
⌘ ( ⇡ ⇤ ) , as k ! 1 [16].
A mirror-learning agent canbe implemented in practice by specifying functional forms of the drift function and neighbourhoodoperator, and parameterising the policy of the agent with a neural network, ⇡ ✓ .
As such, the agentapproximates the objective in Equation (2) by sample averages, and maximises it with an optimisationmethod, like gradient ascent.
It can be shown that PPO is a valid instance of Mirror Learning, withthe drift function:
[Equation]
Although in its formulation, PPO puts no explicit constraints on its update size [28], as its maximisation oracle (see Equation (2)) is N steps of gradient ascent, with learning rate ↵ and gradient clippingthreshold c , it implicitly employs a neighbourhood of an Euclidean ball or radius N ↵ c around ✓ k .
Different mirror-learning algorithms can display different properties, such as sample complexity,world-clock time efﬁciency, or they can aid fulﬁlling other purposes [16].
That is to say, dependingon a preference, a different algorithm is optimal, and PPO is not necessarily the one.
In thispaper, we specify the optimality criterion to be the return of the K th iterate, ⌘ ( ⇡ K ) , and discover amirror-learning algorithm that is superior to PPO with respect to it.
3.3 Evolution Strategies
Evolution Strategies [24, 25, ES] is a backpropagation-free approach to optimisation of stochasticfunctions.
At their core lies the following identity, which holds for any continuously differentiablefunction F of  , and any positive scalar
[Equation] [Equation]
where N (0 , I ) denotes the standard multivariate normal distribution.
By taking the limit  ! 0 , thegradient on the left-hand side recovers the gradient of r  F (  ) .
These facts inspire an approach ofoptimising F with respect to  without estimating gradients with backpropagation—for a randomsample ✏ 1 , . . .
, ✏ n ⇠ N (0 , I ) , the vector 1 n
P ni =1 F (  + ✏ i ) ✏ i is an unbiased gradient estimate.
To reduce variance of this estimator, antithetic sampling is commonly used [23].
In the context of
Meta-RL, where  is the meta-parameter of an RL algorithm alg
, the role of F (  ) is played by theaverage return after the training, F (  ) =
E [ ⌘ ( ⇡ K ) |  ] .
As oppose to the meta-gradient approachesdescribed in Section 2, ES does not require backpropagation of the gradient through the whole trainingepisode—a cumbersome procedure which, often approximated by the truncated backpropagation,introduces bias [32, 33, 22, 6, 19].
Methods
Our overall approach is to meta-learn a drift function to perform policy optimisation over a ﬁxedepisode length K .
Hence, our meta-objective is the expected ﬁnal return,
[Equation]
In Subsection 4.1 we describe the parameterisation of our learnt drift function, while in Subsection 4.2we provide a detailed description of our meta-learning approach.
[Figure]
4.1 Drift Function Network
The drift function that we learn takes form D ⇡ k ( ⇡ | s ) =
E a ⇠ ⇡ k [ f  ( x ) | s ] , where f  ( x ) as a fullyconnected neural network parameterised by  .
Our drift network is a function of the probability ratiobetween a candidate and the old policy, r = ⇡ ( a | s ) / ⇡ k ( a | s ) , and of the advantage
A =
A ⇡ k ( s, a )(which we assume to be normalised across each batch).
To ease learning complicated mappings, weinclude non-linear transformations of these arguments, ultimately forming the following input
[Equation]
In order to guarantee that the neural network is a valid drift function, it sufﬁces to impose thatf  ( x r,A ) = 0 and r r f  ( x r,A )
=
whenever r = 1 , and f  ( x r,A )  0 everywhere.
As in ourmodel, x r,A = 0 whenever r = 1 , the former condition is guaranteed by excluding bias terms fromthe network architecture.
To meet the latter two conditions, we apply the ReLU activation at the lastlayer with a slight shift, x
7! ReLU ( x  ⇠ ) , where ⇠ = 10  6 .
To alleviate the difﬁculty of the meta-training, the main variant of our implementation initialises thedrift function near the PPO one.
That is, before we pass the last layer’s output to shifted ReLU, we addto it the PPO drift function (more precisely, the input of expectation in Equation 3).
We have foundthat this operation leads to a better performance and generalisation across different environments.
Furthermore, such a setting directly tests for the optimality of the drift function of PPO.
If PPO wasoptimal, the network could simply learn the zero mapping.
In principle, the drift function is not restricted to operate with probability ratios and advantages only;it could also accept other arguments, such as algorithm hyperparameters or statistics measuring theprogress of training.
However, many of these vary greatly, in the dimensionality or scale, acrossdifferent instances or types of environments.
Furthermore, a large number of arguments wouldimpede the analysis of the learnt drift function, and comparison to PPO, which are also goals ofour work.
Hence, we work with simple and transferable ratios and normalised advantage estimates.
Nevertheless, for speciﬁc applications, it may be beneﬁcial to consider other, possibly domain-speciﬁc,information, and we do encourage it.
4.2 Meta-Training the Drift Function Network
The meta-objective we optimise is the performance of the learner policy at the end of training:
[Equation]
⌘ ( ⇡ ✓ K ) | alg  ] , where ✓ K is the K th (last) iterate of the RL training under the mirrorlearning
algorithm alg
.
The expectation is taken over the randomness of the initial parameter ✓ 0and stochasticity of the environment.
We solve this problem using ES with antithetic sampling.
Ateach generation (outer loop iteration), we sample a batch of perturbations of  , initialise the policyparameters ✓ 0 , and then train the policy under alg  , using the drift function’s parameter  , for Kiterations.
At the end of the inner-loop training, we estimate the return of the ﬁnal policy  ✓ K , anduse it to estimate the gradient of r  F (  ) as in Equation (4).
We meta-learn the drift function and evaluate policies trained by it in the Brax [8] physics simulatorenvironments.
Brax is designed to take advantage of parallel computation on accelerators, allowing us
[Figure] [Figure]
to roll out thousands of episodes in parallel, and train entire RL agents within minutes.
Thus, it is wellsuited for this type of optimisation problem.
Furthermore, we increase its efﬁciency by vectorisingthe policy optimisation algorithm itself , which lets us train hundreds of RL agents per minute usingaccelerators.
We implement our method on top of the Brax version of PPO, which provides a
Mirror Learning-friendly code template, keeping the policy architecture and training hyperparametersunchanged.
For meta-training we use both evosax [17] and the Learnt_optimization [20] libraries.
For full details of meta-training see Appendix B.
Empirical Studies
We consider two different meta-training setups.
First, as described in Subsection 5.1 we attempt tolearn a drift function completely from scratch to investigate how similar it is to existing algorithmslike PPO.
Second, in Subsection 5.2 we ask whether we can learn a drift function that successfullygeneralises to multiple environments, if it is initialised near PPO.
5.1 Learning drift functions from scratch
In this setting, f  is a neural network with two hidden layers of size 256 and a ReLU activationfunction.
We meta-train it across 5 Brax environments.
We name the resulting algorithm LPO-Zero,and visualise it in Figure 2.
Interestingly, LPO-Zero appears to have learnt a few PPO-like features, as can be observed on Figure2.
For example, it appears to have learnt to clip the update incentive at a speciﬁc ratio threshold,much like PPO; however, it only does so for negative advantages.
Nevertheless, LPO-Zero largelyunderperforms with respect to LPO, and possibly requires much more training to catch up.
We presentﬁgures with performance evaluation of LPO-Zero in Appendix C.
5.2 Learning with the PPO Initialisation
In this setting, f  is a small neural network, with a single hidden layer with 128 neurons, with biasterms removed, and tanh activation function.
Here, we add PPO to the output of the last hidden layer,before passing it to the shifted ReLU,
[Equation]
where ˜ f  ( x r,A ) is the output of the last hidden layer of the drift network.
As such, the the resultingdrift function similar to that of PPO at initialisation.
Surprisingly, we have found that meta-training in a single environment is sufﬁcient to generate driftfunctions whose abilities transfer to unseen tasks.
Moreover, we found that the learnt drifts generallydisplay similar characteristics.
For readability, we chose the drift function that was trained on Ant,whose induced algorithm we refer to as Learnt Policy Optimisation (LPO).
Visualisations and resultsfor drift functions trained on other environments can be found in the Appendix E.
The results on Figure 6 show that LPO, trained only on Ant, outperforms PPO in unseen environments.
Furthermore, the Brax PPO implementation uses different hyperparameters, such as the number ofupdate epochs and the total number of timesteps, for each of the tasks.
This means that LPO, whichwas trained on Ant with hyperparameters associated to it, is robust not only against new environments,but also against new hyperparameters.
We visualise the derivative of the LPO loss in Figure 3, whichenables us to derive an analytical version of it in Section 6.
Analysis of LPO
[Figure]
In this section, we analyse the two key featuresthat are consistently learnt and contribute mostto LPO’s performance.
We then interpret theireffect on policy entropy , and the update asymmetry discovered by LPO, through which it differslargely from PPO (recall Figure 3 for visualisation).
Rollback for negative advantage.
In thebottom-left quadrant of the heat map, which corresponds to A < 0 and r < 1 , we observe thatthe ratio derivative of the LPO objective is positive in a large region, roughly corresponding tor < 1  ✏ .
This implies that actions that fall intothis quadrant, although seemingly not appealing,are encouraged to be taken by the agent, whichcan be interpreted as a form of rollback [31].
Hence, LPO learns to decrease r down to 1  ✏ ,but unlike PPO, encourages r to stay preciselyaround that value.
By doing so, LPO preventsthe agent from giving up on actions that appearpoor at the moment, and encourages it to keepexploring them at a moderate frequency.
Cautious optimism for positive advantage.
The upper-right quadrant corresponds to A > 0 andr > 1 , which is induced by actions that seem the most appealing to update to.
Nevertheless, LPOis cautious in doing so, gradually decreasing the pace of its update towards them, and eventuallyabstaining from chasing the most extreme advantage values—these may come from critic errors.
Wewant to highlight that this view of LPO on positive advantages in much more sophisticated than thatof PPO, which simply removes any incentive from updating towards actions with r > 1 + ✏ , and thuscan be viewed as optimistic relative to PPO.
Implicit entropy maximisation.
Together, these two central features of LPO encourage the agentto spread its policy probability mass moderately over all actions, thus leading to larger entropy and
[Figure]
allowing for richer exploration.
Thus, LPO has implicitly discovered entropy maximisation , whichwe demonstrate in Figure 4.
Update asymmetry.
LPO learns asymmetric features that respect a natural asymmetry of behaviourchange in RL: increasing r for positive advantage A may encourage exploration of a newly-foundaction or strengthen a dominant action, whereas decreasing r for negative A will always discourageexploration of that action and strengthen a dominant action.
In this context, the two discussed featuresof LPO make it completely unlike PPO, which clips the update incentives symmetrically around theorigin.
Secondary Features.
LPO, but not LPO-Zero, appears to consistently learn objectives with gradientspikes around r = 1 in the upper left and lower right quadrants.
Nevertheless, adding them to ouranalytic model of LPO did not improve performance.
We speculate, therefore, that these spikes aremostly artifacts of the network parameterisation.
Discovered Policy Optimisation: A New RL Algorithm Inspired By LPO
In this section, advancing concepts that LPO has discovered, we introduce a novel algorithm—
Discovered Policy Optimization (DPO).
7.1 The Discovered Drift Function Model
Combining the key features identiﬁed in Section 6, we construct a closed-form model of LPO thatcan easily be implemented with just a few lines of code on top of an existing PPO implementation.
We name the new algorithm Discovered Policy Optimisation (DPO) because we have not derivedit—it was instead discovered in the meta-learning process.
DPO is a mirror-learning algorithm, with adrift function that takes different functional forms, depending on the sign of advantage A , as dictatedby the update asymmetry principle from the previous section.
Speciﬁcally, we have found that the(parameter-free) drift function
[Equation]
ReLU
[Equation]
[Equation]
faithfully reproduces the key features of LPO (cautious optimism and rollback) for appropriateconstants ↵ = 2 ,  = 0 .
6 (see Appendix E.1 for veriﬁcation of the drift conditions).
We visualise DPO in Figure 5 and note that even the “crossing-over” of gradient slices of LPO on Figure 3 isfaithfully reproduced.
7.2 Results
We compare DPO to PPO and LPO on a variety of Brax environments.
We use the PPO implementation provided by Brax, which we enhanced with advantage normalisation as we observed it to improve
[Figure]
performance across the majority of the environments.
Our methods also use this implementationtechnique.
While evaluating DPO, similarly to LPO, we do not re-tune any hyperparameters that wereoriginally selected for PPO in Brax.
The results on Figure 6 reveal that DPO matches the performanceof LPO and outperforms PPO on the evaluated environments, despite being a two-line analytic modelof LPO based on two key features .
This enables RL practitioners to implement DPO as easily as PPOwith a performance on par with our best learnt drift function.
Conclusion
In this paper, we approached the problem of algorithm discovery by restricting our meta-learning to thespace of valid Mirror Learning algorithms.
Speciﬁcally, we optimised a drift function parameterisedby a neural network, which we trained with Evolution Strategies .
We consider this work to bean example of the new, promising paradigm of RL algorithm discovery.
Namely, our strategywas to develop a high-performing RL algorithm by combining theoretical insights with large-scalecomputational techniques.
As a result of the training, we obtained a theoretically sound methodthat we named Learnt Policy Optimisation (LPO), which outperforms a state-of-the-art baseline(PPO) in unseen environments, and with unseen hyperparameter settings.
After analysing the learnedfeatures discovered by LPO, we introduced Discovered Policy Optimisation (DPO)—a closed-formapproximation to LPO.
Our experimental results show that DPO matches LPO in performance androbustness to hyperparameter settings.
In the future, we plan to expand the variety of inputs to thelearnt drift function, as well as to parameterise and meta-learn other attributes of mirror learning.
Weexpect these advancements to provide more insights into policy optimisation, ultimately resulting inmore robust and better performing RL algorithms.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We discuss this in the Introduction(1), and Discussion (Section 8)
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work
does not introduce any signiﬁcant new negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We do not violate any of the ethics concerns
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
Our results
are empirical.(b)
Did you include complete proofs of all theoretical results?
[N/A]
Our results are
empirical.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] They areattached in the supplementary material
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] They are in Appendix B
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] They are in the ﬁgures
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] They are in Appendix B4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite all codebasesused.
(b) Did you mention the license of the assets?
[N/A] We follow the codebase licenses, butdo not mention them.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We will include code in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We are not using anyone’s data.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A] We are not using anyone’s data.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]