Original: Extracting Strong Policies for Robotics Tasks from Zero-Order Trajectory Optimizers
Revised: Extracting Strong Policies for Robotics Tasks from Zero-Order Trajectory Optimizers

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Solving high-dimensional, continuous robotic tasks is a challenging optimization problem.
Revised: Solving high-dimensional, continuous robotic tasks is a challenging optimization problem.

Original: Model-based methods that rely on zero-order optimizers like the crossentropy method (CEM) have so far shown strong performance and are considered state-of-the-art in the model-based reinforcement learning community.
Revised: Model-based methods that rely on zero-order optimizers like the crossentropy method (CEM) have so far shown strong performance and are considered state-of-the-art in the model-based reinforcement learning community.

Original: However, this success comes at the cost of high computational complexity, being therefore not suitable for real-time control.
Revised: However, this success comes at the cost of high computational complexity, being therefore not suitable for real-time control.

Original: In this paper, we propose a technique to jointly optimize the trajectory and distill a policy, which is essential for fast execution in real robotic systems.
Revised: In this paper, we propose a technique to jointly optimize the trajectory and distill a policy, which is essential for fast execution in real robotic systems.

Original: Our method builds upon standard approaches, like guidance cost and dataset aggregation, and introduces a novel adaptive factor which prevents the optimizer from collapsing to the learner’s behavior at the beginning of the training.
Revised: Our method builds upon standard approaches, like guidance cost and dataset aggregation, and introduces a novel adaptive factor which prevents the optimizer from collapsing to the learner’s behavior at the beginning of the training.

Original: The extracted policies reach unprecedented performance on challenging tasks like making a humanoid stand up and opening a door without reward shaping.
Revised: The extracted policies reach unprecedented performance on challenging tasks like making a humanoid stand up and opening a door without reward shaping.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The general purpose of model-based and model-free reinforcement learning (RL) is to optimize a trajectory or ﬁnd a policy that is fast and accurate enough to be deployed on real robotic systems.
Revised: The general purpose of model-based and model-free reinforcement learning (RL) is to optimize a trajectory or ﬁnd a policy that is fast and accurate enough to be deployed on real robotic systems.

Original: Policies optimized by model-free RL algorithms achieve outstanding results for many challenging domains (Heess et al., 2017; Andrychowicz et al., 2020), however, in order to converge to the ﬁnal performance, they require a large amount of interactions with the environment and can hardly be used on real robots, which have a limited lifespan.
Revised: Policies optimized by model-free RL algorithms achieve outstanding results for many challenging domains (Heess et al., 2017; Andrychowicz et al., 2020), however, in order to converge to the ﬁnal performance, they require a large number of interactions with the environment and can hardly be used on real robots, which have a limited lifespan.

Original: Moreover, real robotic systems are high-dimensional and have a highly non-convex optimization landscape which makes policy gradient methods prone to converge to locally optimal solutions.
Revised: Moreover, real robotic systems are high-dimensional and have a highly non-convex optimization landscape which makes policy gradient methods prone to converge to locally optimal solutions.

Original: In addition, model-free RL methods only gather task-speciﬁc information which inherently limits their generalization performance to new situations.
Revised: In addition, model-free RL methods only gather task-speciﬁc information which inherently limits their generalization performance to new situations.

Original: On the other hand, recent advances in model-based RL show that it is possible to match model-free performances by learning uncertainty-aware system dynamics (Chua et al., 2018;
Revised: On the other hand, recent advances in model-based RL show that it is possible to match model-free performances by learning uncertainty-aware system dynamics (Chua et al., 2018;

Original: Deisenroth &
Revised: Deisenroth &

Original: Rasmussen, 2011; Du et al., 2019).
Revised: Rasmussen, 2011; Du et al., 2019).

Original: The learned model can then be used within a model-predictive control framework for trajectory optimization.
Revised: The learned model can then be used within a model-predictive control framework for trajectory optimization.

Original: Zero-order optimizers are gaining a lot of traction in the model-based RL community (Chua et al., 2018; Wang & Ba, 2020; Williams et al., 2015) since they can be used with any choice of model and cost function, and can be surprisingly effective in ﬁnding high-performance solutions (Pinneri et al., 2020)
Revised: Zero-order optimizers are gaining a lot of traction in the model-based RL community (Chua et al., 2018; Wang & Ba, 2020; Williams et al., 2015) since they can be used with any choice of model and cost function, and can be surprisingly effective in ﬁnding high-performance solutions (Pinneri et al., 2020)

Original: (close to a global optimum) in contrast to their gradient based counterparts, which are often highly dependent on hyperparameter tuning (Henderson et al., 2017).
Revised: (close to a global optimum) in contrast to their gradient-based counterparts, which are often highly dependent on hyperparameter tuning (Henderson et al., 2017).

Original: One of the most popular optimizers is the Cross-Entropy Method (CEM), originally introduced in the 90s by Rubinstein & Davidson (1999).
Revised: One of the most popular optimizers is the Cross-Entropy Method (CEM), originally introduced in the 90s by Rubinstein & Davidson (1999).

Original: Despite their achievements, using zero-order methods for generating action sequences ishighly time inefﬁcient in complex high-dimensional environments, due to the extensive sampling, making it impossible to deploy them for real-time applications.
Revised: Despite their achievements, using zero-order methods for generating action sequences is time consuming in complex high-dimensional environments, due to the extensive sampling, making it hard to deploy them for real-time applications.

Original: Extracting a policy from powerful zero-order optimizers like CEM would bridge the gap between model-based RL in simulation and real-time robotics.
Revised: Extracting a policy from powerful zero-order optimizers like CEM would bridge the gap between model-based RL in simulation and real-time robotics.

Original: As of today, this is still an open challenge (Wang & Ba, 2020).
Revised: As of today, this is still an open challenge (Wang & Ba, 2020).

Original: We analyse this issue and showcase several approaches for policy extraction from CEM.
Revised: We analyze this issue and showcase several approaches for policy extraction from CEM.

Original: In particular, we will use the sample-efﬁcient modiﬁcation of CEM (iCEM) presented in Pinneri et al.
Revised: In particular, we will use the sample-efﬁcient modiﬁcation of CEM (iCEM) presented in Pinneri et al.

Revised: Throughout the paper, we will call these optimizers “experts” as they provide demonstration trajectories.

Original: In order to isolate the problem of bringing policy performance close to the expert’s one, we consider the true simulation dynamics as our forward model.
Revised: To isolate the problem of bringing policy performance close to the expert’s one, we consider the true simulation dynamics as our forward model.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • pinpointing the issues that arise when trying to distill a policy from a multimodal, stochastic teacher; • introducing APEX, an Adaptive Policy EXtraction procedure that integrates iCEM with DAgger and a novel adaptive variant of Guided Policy Search; • producing an improving adaptive teacher, with higher performance than the original iCEM optimizer; • obtaining strong policies for hard robotic tasks in simulation (H UMANOID S TANDUP , F ETCH P ICK &P LACE ,H ALFCHEETAH R UNNING , D OOR ), where model-free policies would usually just converge to local optima.
Revised: • pinpointing the issues that arise when trying to distill a policy from a multimodal, stochastic teacher; • introducing APEX, an Adaptive Policy EXtraction procedure that integrates iCEM with DAgger and a novel adaptive variant of Guided Policy Search; • our speciﬁc integration of methods produces an improving adaptive teacher, with higher performance than the original iCEM optimizer; • obtaining strong policies for hard robotic tasks in simulation (H UMANOID S TANDUP , F ETCH P ICK &P LACE , D OOR ), where model-free policies would usually just converge to local optima.

Original: Videos showing extracted policies’ performances can be found at https://sites.google.
Revised: Videos showing extracted policies’ performances can be found at https://sites.google.

Original: com/view/adaptive-policy-extraction .
Revised: com/view/adaptive-policy-extraction .

Original: The source code will be made available with the ﬁnal version of the paper.
Revised: The source code will be made available with the ﬁnal version of the paper.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our objective is to extract high-performing policies from CEM experts that can operate with few planning samples in order to make iterative learning fast.
Revised: Our objective is to extract high-performing policies from CEM experts that can operate with few planning samples in order to make iterative learning fast.

Original: Other kinds of zero-order optimizers have been used to generate control sequences (Williams et al., 2015; Lowrey et al., 2019) but they still have to evaluate thousands of trajectories for each time step.
Revised: Other kinds of zero-order optimizers have been used to generate control sequences (Williams et al., 2015; Lowrey et al., 2019) but they still have to evaluate thousands of trajectories for each time step.

Original: Even simple random shooting has been used as a trajectory optimizer to bootstrap a model-free policy (Nagabandi et al., 2018).
Revised: Even simple random shooting has been used as a trajectory optimizer to bootstrap a model-free policy (Nagabandi et al., 2018).

Original: In order to train policies from optimal control solutions, it was shown that the expert optimizers need to be guided towards the learning policy – known as guided policy search (GPS) (Levine & Koltun, 2013; Levine & Abbeel, 2014).
Revised: To train policies from optimal control solutions, it was shown that the expert optimizers need to be guided towards the learning policy – known as guided policy search (GPS) (Levine & Koltun, 2013; Levine & Abbeel, 2014).

Original: In our work, the expert does not come from optimal control, but is the stochastic iCEM optimizer, which we will also refer to as teacher.
Revised: In our work, the expert does not come from optimal control but is the stochastic iCEM optimizer, which we will also refer to as teacher .

Original: We apply GPS in a model model-predictive control setting, as done in Levine & Koltun (2013); Mordatch & Todorov (2014);
Revised: We apply GPS in a model model-predictive control setting, as done in Levine & Koltun (2013); Mordatch & Todorov (2014);

Original: Mordatch et al.
Revised: Mordatch et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: ; Kahn et al.
Revised: ; Kahn et al.

Original: (2017); Sun et al.
Revised: (2017); Sun et al.

Original: (2018) using local trajectory optimization to generate a dataset for training a global policy through imitation learning.
Revised: (2018) using local trajectory optimization to generate a dataset for training a global policy through imitation learning.

Original: These approaches alternate between training a policy and creating new data with a model-based supervisor guided towards the learner, which was formalized in Sun et al.
Revised: These approaches alternate between training a policy and creating new data with a model-based supervisor guided towards the learner, which was formalized in Sun et al.

Original: Stochastic experts require particular guidance strategies, such as an adaptive cost formulation that we propose here and warm-starting, as done in Wang & Ba (2020).
Revised: Stochastic experts require particular guidance strategies, such as an adaptive cost formulation that we propose here, together with expert warm-starting via distribution initialization and additional samples from the policy. A simple form of warm-starting was already done in Wang & Ba (2020).

Original: Recently, approaches like simple point-to-point supervised training as Behavioral Cloning (BC), or
Revised: Recently, approaches like simple point-to-point supervised training as Behavioral Cloning (BC), or

Original: Generative Adversarial Network training (GAN) have been explored (Wang & Ba, 2020) for policy distillation from CEM, but only largely sub-optimal policies could be extracted.
Revised: Generative Adversarial Network training (GAN) have been explored (Wang & Ba, 2020) for policy distillation from CEM, but only largely sub-optimal policies could be extracted.

Original: When the policy is used alone at test time and not in combination with the MPC-CEM optimizer, its performance drops signiﬁcantly, becoming almost random for some environments.
Revised: When the policy is used alone at test time and not in combination with the MPC-CEM optimizer, its performance drops signiﬁcantly, becoming almost random for some environments.

Original: We argue that the reason behind the difﬁculty in distilling a policy from CEM expert data is the multimodality of the CEM solution space and its inherent stochasticity due to the sampling, which we address below.
Revised: We argue that the reason behind the difﬁculty in distilling a policy from CEM expert data is the multimodality of the CEM solution space and its inherent stochasticity due to the sampling, which we address below.

Original: Another possibility to train higher-performance policies from experts is DAgger (Ross & Bagnell, 2010), which is an on-policy method, asking the expert to relabel/correct policy actions.
Revised: Another possibility to train higher-performance policies from experts is DAgger (Ross & Bagnell, 2010), which is an on-policy method, asking the expert to relabel/correct policy actions.

Original: Nevertheless, as we will show in the following sections, DAgger alone is not sufﬁcient to extract high-performing policies in our setting.
Revised: Nevertheless, as we will show in the following sections, DAgger alone is not sufﬁcient to extract high-performing policies in our setting.

Original: In order to solve this problem, we use a guiding cost in combination with DAgger.
Revised: To solve this problem, we use a guiding cost in combination with DAgger.

Original: A combination of GPS and DAgger-style relabeling was proposed in PLATO (Kahn et al., 2017), however for the purpose of creating unbiased training data from iLQG experts.
Revised: A combination of GPS and DAgger-style relabeling was proposed in PLATO (Kahn et al., 2017), however to create unbiased training data from iLQG experts.

Original: Since unguided DAgger is not appropriate with CEM, PLATO is not successful in our setting either.
Revised: Since unguided DAgger is not appropriate with CEM, PLATO is not successful in our setting either.

Original: The components of our algorithm will be explained in the following sections.
Revised: The components of our algorithm will be explained in the following sections.

Original:  M ETHODS
Revised:  M ETHODS

Original: The aim of trajectory optimization is to ﬁnd a suitable action sequence (cid:126)a t = ( a t , a t +1 , . .
Revised: Trajectory optimization aims to ﬁnd a suitable action sequence (cid:126)a t = ( a t , a t +1 , . .

Original: . , a t + h ) of horizon h that minimizes a cost function f ( (cid:126)a t , s t ) , where s t is the current state of the system.
Revised: . , a t + h ) of horizon h that minimizes a cost function f ( (cid:126)a t , s t ) , where s t is the current state of the system.

Original: i.e.
Revised: i.e.

Original: [Equation]
Revised: [Equation]

Original: The cost function f encodes the task.
Revised: The cost function f encodes the task.

Original: Optimal control is obtained if f is the trajectory cost until step h plus the cost-to-go under the optimal policy for an inﬁnite-horizon problem, evaluated at the last state in the ﬁnite horizon trajectory. 
Revised: Optimal control is obtained if f is the trajectory cost until step h plus the cost-to-go under the optimal policy for an inﬁnite-horizon problem, evaluated at the last state in the ﬁnite horizon trajectory. Typically, the cost-to-go is replaced by a proxy cost such as the distance to a target. In our case, the cost f is given by the sum of negative environment rewards up to the planning horizon.

Original: 3. I MPROVED C ROSS -E NTROPY METHOD FOR MPC
Revised: 3. I MPROVED C ROSS -E NTROPY METHOD FOR MPC

Original: To optimize Eq.
Revised: To optimize Eq.

Original: 1 we use a variant of the Cross-Entropy Method.
Revised: 1 we use a variant of the Cross-Entropy Method.

Original: Although originally introduced as an adaptive importance sampling procedure to estimate rare-event probabilities, it was recently employed in model-based RL (Chua et al., 2018; Wang & Ba, 2020) as a trajectory optimizer.
Revised: Although originally introduced as an adaptive importance sampling procedure to estimate rare-event probabilities, it was recently employed in model-based RL (Chua et al., 2018; Wang & Ba, 2020) as a trajectory optimizer.

Original: A practical implementation of CEM uses a Gaussian proposal distribution over the optimization variables (cid:126)a , N ( µ, σ ) , evaluates the cost function f
Revised: A practical implementation of CEM uses a Gaussian proposal distribution over the optimization variables (cid:126)a , N ( µ, σ ) , evaluates the cost function f

Original: ( (cid:126)a ) , and reﬁts this distribution iteratively by using the top- k samples.
Revised: ( (cid:126)a ) , and reﬁts this distribution iteratively by using the top- k samples.

Original: It ﬁnds low-cost regions of f and high-performing samples of the variable (cid:126)a .
Revised: It ﬁnds low-cost regions of f and high-performing samples of the variable (cid:126)a .

Original: The cost function can be either expressed through a learned value function or through the dynamics model of the system, which in turn can be learned from data or in analytical form.
Revised: The cost function can be either expressed through a learned value function or through the dynamics model of the system, which in turn can be learned from data or in analytical form.

Original: In the model-predictive control (MPC) framework, CEM can be used at every time-step to generate the optimal action plan, of which only the ﬁrst action is executed, and the whole procedure is repeated for the next steps until task completion.
Revised: In the model-predictive control (MPC) framework, CEM can be used at every time-step to generate the optimal action plan, of which only the ﬁrst action is executed, and the whole procedure is repeated for the next steps until task completion.

Original: In this paper we will make use of iCEM, a sample-efﬁcient improvement of CEM by Pinneri et al.
Revised: In this paper we will make use of iCEM, a sample-efﬁcient improvement of CEM by Pinneri et al.

Original: (2020) that makes use of colored-noise and memory.
Revised: (2020) that makes use of colored-noise and memory .

Original: In particular, it generates correlated action sequences with a non-ﬂat frequency spectrum, differently from the ﬂat Gaussian noise of CEM.
Revised: In particular, it generates correlated action sequences with a non-ﬂat frequency spectrum, differently from the ﬂat Gaussian noise of CEM.

Original: This, together with the memory addition, results in an order of magnitude sample reduction.
Revised: This, together with the memory addition, results in an order of magnitude sample reduction.

Original: 3.1. U SING A POLICY TO INFORM THE OPTIMIZATION : I CEM π
Revised: 3.1. U SING A POLICY TO INFORM THE OPTIMIZATION : I CEM π

Original: Unguided search for action sequences can be hard, in particular if the cost function has large ﬂat regions (sparse rewards).
Revised: Unguided search for action sequences can be hard, in particular, if the cost function has large ﬂat regions (sparse rewards).

Original: Since we aim at extracting a policy π ( s ) , we can expect that after some examples the policy can be used to guide the search into the right region.
Revised: Since we aim at extracting a policy π ( s ) , we can expect that after some examples the policy can be used to guide the search into the right region.

Original: Thus, we warm-start the mean µ of the iCEM Gaussian distribution with the policy actions.
Revised: Thus, we warm-start the mean µ of the iCEM Gaussian distribution with the policy actions.

Original: In particular, at time t = 0 where no prior information exists, the mean is initialized from rolling out the trajectory with the policy until the planning horizon: µ ← ( π ( s 0 ) , π ( s 1 ) , . . .
Revised: In particular, at time t = 0 where no prior information exists, the mean is initialized from rolling out the trajectory with the policy until the planning horizon: µ ← ( π ( s 0 ) , π ( s 1 ) , . . .

Original: π ( s h )) .
Revised: π ( s h )) .

Original: Whenever the action for a new step is computed, iCEM uses shift initialization of the mean, and only the mean-action at the end of the horizon is initialized from the policy. 
Revised: Whenever the action for a new step is computed, iCEM uses shift initialization of the mean, and only the mean-action at the end of the horizon is initialized from the policy. Since iCEM is sample-based we also provide it with samples from the policy directly.

Revised: More concretely, the actions performed by the policy are added in the last iteration if iCEM.

Original: The warm-started iCEM algorithm is called iCEM π and is shown in Alg. 1.
Revised: The policy-informed iCEM algorithm is called iCEM π and is shown in Alg. 1.

Original: 3. O FF - AND O N -P OLICY I MITATION L EARNING
Revised: 3. O FF - AND O N -P OLICY I MITATION L EARNING

Original: How difﬁcult is  to clone a policy from iCEM π ?
Revised: How difﬁcult is it to clone a policy from iCEM π ?

Original: Using a basic off-policy method like Behavioral
Revised: Using a basic off-policy method like Behavioral

Original: Cloning (BC), which simply does point-to-point L2 loss minimization, does not work out-of-the-box.
Revised: Cloning (BC), which simply does point-to-point L2 loss minimization, does not work out-of-the-box.

Original: This is because during test time, the policy visits state-space regions for which it never received any expert feedback, and it consequently fails.
Revised: This is because during test time, the policy visits state-space regions for which it never received any expert feedback, and it consequently fails.

Original: This problem is also known as “covariate shift”.
Revised: This problem is also known as “covariate shift”.

Original: The performance gap (difference in cost to go) between the expert and the policy scales quadratically in time due to the covariate shift (Ross & Bagnell, 2010).
Revised: The performance gap (difference in cost to go) between the expert and the policy scales quadratically in time due to the covariate shift (Ross & Bagnell, 2010).

Original: A classical way to address this problem is with on-policy imitation learning.
Revised: A classical way to address this problem is with on-policy imitation learning.

Original: DAgger (Dataset
Revised: DAgger (Dataset

Original: Aggregation) by Ross et al.
Revised: Aggregation) by Ross et al.

Original: (2011) is the most popular example.
Revised: (2011) is the most popular example.

Original: DAgger works by iteratively rolling

Original: [Figure]

Original: iCEM (

Original: [Figure] ) guided iCEM [Figure] out the current policy, querying the expert for the correct actions (also called relabeling) on the states visited by the policy, and training the policy on the aggregate dataset across all iterations.
Revised: DAgger works by iteratively rolling out the current policy, querying the expert for the correct actions (also called relabeling) on the states visited by the policy, and training the policy on the aggregate dataset across all iterations.

Original: The covariate shift is alleviated by populating the data with states visited by previous iterations of the policy.
Revised: The covariate shift is alleviated by populating the data with states visited by previous iterations of the policy.

Original: I SSUES OF STOCHASTIC TEACHERS :
Revised: I SSUES OF STOCHASTIC TEACHERS :

Original: Nevertheless, some problems arise when considering population-based optimizers as experts, both for BC alone or in combination with DAgger.
Revised: Nevertheless, some problems arise when considering population-based optimizers as experts, both for BC alone or in combination with DAgger.

Original: The optimized action sequence of CEM might converge to different solutions every time it is asked to relabel the policy actions.
Revised: The optimized action sequence of CEM might converge to different solutions every time it is asked to relabel the policy actions.

Original: Naturally,there are many actionsthat can result in an equally high-performing solution.
Revised: Naturally, many actions can result in an equally high-performing solution.

Original: As an example: in the Fetch Pick & Place task, well before grasping the box, the action of opening or closing the gripper can be arbitrary, as long as the action of opening the gripper is executed on time.
Revised: As an example: in the Fetch Pick & Place task, well before grasping the box, the action of opening or closing the gripper can be arbitrary, as long as the action of opening the gripper is executed on time.

Original: To illustrate this problem we analyze the variance of the relabeled actions produced by the same expert on a ﬁxed state sequence.
Revised: To illustrate this problem we analyze the variance of the relabeled actions produced by the same expert on a ﬁxed state sequence.

Original: The variance is high, spanning over a third of the action-space, as can be seen in Fig. 2(a).
Revised: The variance is high, spanning over a third of the action-space, as can be seen in Fig. 2(a).

Original: 3. G UIDED P OLICY S EARCH
Revised: 3. G UIDED P OLICY S EARCH

Original: In order to lift some of the aforementioned problems the expert can be guided using the current policy.
Revised: To lift some of the aforementioned problems, the expert can be guided using the current policy.

Original: For high-dimensional control tasks, in addition to the high variance (Fig. 2(a)), zero-order optimizers may also ﬁnd multiple different solutions, leading to multi-modal training data.
Revised: For high-dimensional control tasks, in addition to the high variance (Fig. 2(a)), zero-order optimizers may also ﬁnd multiple different solutions, leading to multi-modal training data.

Original: This prohibits a successful policy extraction through supervised learning.
Revised: This prohibits a successful policy extraction through supervised learning.

Original: To address these issues, the cost function f in Eq.
Revised: To address these issues, the cost function f in Eq.

Original: 1 becomes the main cost J (task) plus a guidance cost to bias the optimizer’s solution towards the policy:
Revised: 1 becomes the main cost J (task) plus a guidance cost to bias the optimizer’s solution towards [Figure] i [Figure]

Revised: n

Original: [Equation]
Revised: [Figure] ( d i m [Figure] [Figure]

Original:  which is also known as Guided Policy Search (GPS) (Levine & Koltun, 2013).
Revised: the policy: [Equation] which is also known as Guided Policy Search (GPS) (Levine & Koltun, 2013).

Original: When competing solutions arise, the one closer to the policy is preferred, tackling in this way the problem of multimodality and creating much more consistent training data.
Revised: When competing solutions arise, the one closer to the policy is preferred, tackling in this way the problem of multimodality and creating much more consistent training data.

Original: As a remark, in our experiments we use deterministic policies which reduces the KL-divergence to a squared difference between the actions.
Revised: As a remark, in our experiments, we use deterministic policies which reduces the KL-divergence to a squared difference between the actions.

Original: In addition to Eq. 2, we further guide iCEM by warm-starting it from the policy, as implemented in iCEM π (Alg. 1).
Revised: In addition to Eq. 2, we further guide iCEM by warm-starting it from the policy, as implemented in iCEM π (Alg. 1).

Original: As a consequence, both together lead to a reduced action variance, as seen in
Revised: As a consequence, both together lead to a reduced action variance, as seen in

Original: Fig. 2(b).
Revised: Fig. 2(b).

Original: For an illustrative state sequence, the standard deviation of expert actions drops from σ = 0 .
Revised: For an illustrative state sequence, the standard deviation of expert actions drops from σ = 0 .

Original: 30 without guidance to σ = 0 .
Revised: 30 without guidance to σ = 0 .

Revised: The data for training the policy comes from the expert interacting with the environment, that we call expert rollouts .

Original: When combining GPS with DAgger we use the same cost-function (Eq. 2) for both expert rollouts and actions relabeling.
Revised: When combining GPS with DAgger we use the same cost-function (Eq. 2) for the normal expert rollout and for asking the expert to perform action relabeling.

Original: This is in contrast to PLATO (Kahn et al., 2017), designed for iLQG experts, where the guidance cost is only  for the expert.
Revised: This is in contrast to PLATO (Kahn et al., 2017), designed for iLQG experts, where the guidance cost is only used for the expert rollouts.

Original: Also, in PLATO the policy is not executed to collect new data, but only the guided expert  is relabeled using theexpert with an unmodiﬁed cost to collect optimal training data.
Revised: Also, in PLATO the policy is not executed to collect new data, but relabeling of guided expert data is done using the unmodiﬁed cost to collect optimal training data.

Original: For zero-order optimizers, unguided solutions are not helpful as shown above.
Revised: For zero-order optimizers, unguided solutions are not helpful as shown above.

Original: Without relabeling the actual policy rollouts, we found PLATO not to work in our settings.
Revised: Without relabeling the actual policy rollouts, we found PLATO not to work in our settings.

Original: The hyperparameter λ in Eq. 2 is difﬁcult to choose.
Revised: The hyperparameter λ in Eq. 2 is difﬁcult to choose.

Original: It might be adapted according to learning progress or other heuristics.
Revised: It might be adapted according to learning progress or other heuristics.

Original: However, when using the guided cost in CEM other considerationsalso have to be made which lead to a simple adaptation scheme introduced in the next section.
Revised: However, when using the guided cost in CEM other considerations have to be made which lead to a simple adaptation scheme introduced in the next section.

Original: 3. A DAPTIVE AUXILIARY COST WEIGHTING
Revised: 3. A DAPTIVE AUXILIARY COST WEIGHTING

Original: The purpose of λ is to trade-off the potential loss in cost by staying close to the policy.
Revised: The purpose of λ is to trade-off the potential loss in cost by staying close to the policy.

Original: Generalizing
Revised: Generalizing

Original: Eq. 2, we can phrase the cost function f in Eq.
Revised: Eq. 2, we can phrase the cost function f in Eq.

Original: 1 as the sum of the main cost J and a set of auxiliary costs C aux
Revised: 1 as the sum of the main cost J and a set of auxiliary costs C aux

Original: j :
Revised: j :

Original: [Equation]
Revised: [Equation]

Original: Examples of the auxiliary costs are the guidance cost as in Eq.
Revised: Examples of the auxiliary costs are the guidance cost as in Eq.

Original: 2 and the action norm (cid:107) (cid:126)a (cid:107)
Revised: 2 and the action norm (cid:107) (cid:126)a (cid:107)

Original: to prefer small action magnitudes.
Revised: to prefer small action magnitudes.

Original: We follow the philosophy that the auxiliary costs are subordinate and should only bias the solutions without causing a large performance drop.
Revised: We follow the philosophy that the auxiliary costs are subordinate and should only bias the solutions without causing a large performance drop.

Original: This leads to the idea that the auxiliary costs should never dominate the optimization.
Revised: This leads to the idea that the auxiliary costs should never dominate the optimization.

Original: Since the  costs might vary by orders of magnitude,  we opt for a formulation where the auxiliary costs can maximally lead to a ﬁxed fractional loss in performance.
Revised: Since the main costs might vary by orders of magnitude, for instance in sparse reward settings, we opt for a formulation where the auxiliary costs can maximally lead to a ﬁxed fractional loss in performance.

Original: As a motivating example, let us consider hard tasks that present ﬂat regions in the cost function: in this case a non-zero λ can lead to a failure of guided CEM to ﬁnd any good solution.
Revised: As a motivating example, let us consider hard tasks that present ﬂat regions in the cost function: in this case, a non-zero λ can lead to a failure of guided CEM to ﬁnd any good solution.

Original: How is that possible?
Revised: How is that possible?

Original: Let us consider the Fetch Pick & Place task as shown in Fig.
Revised: Let us consider the Fetch Pick & Place task as shown in Fig.

Original: When none of the sampled action-sequences moves the box (which is quite likely in the ﬁrst CEM iteration) then all sequences are equivalent under the main cost.
Revised: When none of the sampled action-sequences moves the box (which is quite likely in the ﬁrst CEM iteration) then all sequences are equivalent under the main cost.

Original: Elites (Alg. 1 line 17) are only selected based on the auxiliary costs which leads to a quick reduction in sampling variance and no good solution is ever found.
Revised: Elites (Alg. 1 line 17) are only selected based on the auxiliary costs resulting in a quick reduction in sampling variance and preventing a good solution to be ever [Figure] i [Figure]

Original:  To illustrate this phenomenon, Fig.
Revised: found. To illustrate this phenomenon, Fig.

Original: 3 displays the sampling distribution of iCEM π over the course of the optimization iterations (line 9 in Alg. 1).
Revised: 3 displays the sampling distribution of iCEM π throughout the optimization iterations (line 9 in Alg. 1).

Original: Using a ﬁxed λ (iCEM π -ﬁxed λ ) typically converges to the policy action which is often unsuccessful in a new situation or when the policy is still weak.
Revised: Using a ﬁxed λ (iCEM π -ﬁxed λ ) typically converges to the policy action which is often unsuccessful in a new situation or when the policy is still weak.

Original: We propose the following adaptation of λ j :
Revised: We propose the following adaptation of λ j :

Original: [Equation]
Revised: [Equation]

Original: [Figure] where c j is the new hyperparameter, 0 < (cid:15) is a small regularization to avoid amplifying tiny auxiliary costs, and R ( C ) represents the cost-range of C in the elite set:
Revised: where c j is the new hyperparameter, 0 < (cid:15) is a small regularization to avoid amplifying tiny auxiliary costs, and R ( C ) represents the cost-range of C in the elite set:

Original: [Equation]
Revised: [Equation]

Original: This formulation ensure that λ i is zero if the main cost J cannot be improved by the preliminary elite-set (as in the example above).
Revised: This formulation ensures that λ i is zero if the main cost J cannot be improved by the preliminary elite-set (as in the example above).

Original: When an actual improvement is possible and the cost landscape is not ﬂat ( R ( J ) > 0 ), the auxiliary cost inﬂuences up to a ﬁxed fraction ( c j ) of the cost-range.
Revised: When an actual improvement is possible and the cost landscape is not ﬂat ( R ( J ) > 0 ), the auxiliary cost inﬂuences up to a ﬁxed fraction ( c j ) of the cost-range.

Original: Thus, the sampling distribution for the adaptive case will converge only if a good solution is found, as visualized in Fig.
Revised: Thus, the sampling distribution for the adaptive case will converge only if a good solution is found, as visualized in Fig.

Original: 3 (a) (iCEM π -GPS adaptive λ ) for one auxiliary cost being the KL-term as in Eq.
Revised: 3 (a) (iCEM π -GPS adaptive λ ) for one auxiliary cost being the KL-term as in Eq.

Revised: The new hyperparameter c is easier to tune than λ .

Revised: We use the same c for all experiments, although the main costs are 3 orders of magnitude apart.

Revised: Another aspect of the ratio in Eq.

Revised: 4 is that, when none of the elite sequences is close to the policy network, then the denominator of Eq.

Revised: 4 is small, which heavily steers the sampling towards the policy, however, only if the numerator is not zero ( R ( J ) (cid:54) = 0 ), i.e. a reward signal from the environment is detected.

Original:  T EACHER IMPROVEMENT : Another effect ofthe adaptive guidance  is the improvement of the same teacher, which is not common nor expected in standard imitation learning.
Revised: Figure 9 in appendix D shows how the adaptive λ parameter evolves over time in the F ETCH P ICK &P LACE environment for a weak and medium policy. T EACHER IMPROVEMENT : Another effect of adaptive guidance in combination with iCEM π is the improvement of the expert/teacher, which is not common nor expected in standard imitation learning.

Original: The increased success-rate is shown in Fig. 3 (b).
Revised: The increased success-rate is shown in Fig. 3(b).

Original: As we can see from the plot, the guidance alone is not sufﬁcient to reach the original iCEM expert performance: when the auxiliary cost becomes adaptive, the iCEM beneﬁts from the policy and is able to exploit it only in crucial moments.
Revised: As we can see from the plot, the guidance alone is not sufﬁcient to reach the original iCEM expert performance: when the auxiliary cost becomes adaptive, the iCEM beneﬁts from the policy.

Original: 3. P UTTING THE PIECES TOGETHER : APEX
Revised: 3. P UTTING THE PIECES TOGETHER : APEX

Original: Our method, named Adaptive Policy EXtraction (APEX), uses adaptive policy-guided iCEM π and
Revised: Our method, named Adaptive Policy EXtraction (APEX), uses adaptive policy-guided iCEM π and

Original: DAgger using the same expert to create data for successful training of policies in an iterative fashion.
Revised: DAgger using the same expert to create data for successful training of policies in an iterative fashion.

Original: The main steps are: create data from  guided expert, update the policy, rollout the policy and relabel the actions using the same expert (DAgger), add the relabeled data to the dataset and update the policy.
Revised: The main steps are: create data from the guided expert, update the policy, rollout the policy and relabel the actions using the same expert (DAgger), add the relabeled data to the dataset, and update the policy.

Original: This iterates until the desired performance is reached.
Revised: This iterates until the desired performance is reached.

Original: More details are found in thepseudo-code in
Revised: More details are found in the

Revised: Input: iCEM π ; π θ : policy network; n : # rollouts per iteration 1 init θ randomly; D ← ∅ ; i ←2 while not converged do f

Original: Alg. 2.
Revised: ( (cid:126)a, s ) ← J t ( (cid:126)a, s t ) + λ 1 D KL ( π θ || (cid:126)a ) + λ 2 (cid:107) (cid:126)a (cid:107) ; // see Eqns. 2, 3, and τ CEM ← n Rollout with iCEM π ( f ( (cid:126)a, s ) , π θ ) ; // τ is the resulting trajectory add τ CEM to D θ ← train policy π θ on D τ π ← Rollout with π θ τ DAgger ← relabel actions in τ π with iCEM π

Revised: ( f ( (cid:126)a, s ) , π θ ) ; // DAgger add τ DAgger to D θ ← train policy π θ on D i ← i + 1 ; // one APEX iteration 12 return π θ

Original:  R ESULTS
Revised:  R ESULTS

Original: Can strong policies be obtained from model-based planning and imitation learning?
Revised: Can strong policies be obtained from model-based planning and imitation learning?

Original: By considering high-dimensional challenging robotics environments, some of them with sparse rewards, we test
Revised: By considering high-dimensional challenging robotics environments, some of them with sparse rewards, we test

Original: APEX and ﬁnd that the answer is yes.
Revised: APEX and ﬁnd that the answer is yes.

Original: In some cases we considerably improve the state-of-the-art currently held by model-free methods.
Revised: In some cases, we considerably improve the state-of-the-art currently held by model-free methods.

Original: We perform our experiments on a selection of 4 environments, listed below, which use the MuJoCo (Todorov et al., 2012) physics engine:
Revised: We perform our experiments on a selection of 4 environments, listed below, which use the MuJoCo (Todorov et al., 2012) physics engine:

Original: Input: iCEM π ; π θ : policy network; n : # rollouts per iteration 1 init θ randomly 2 D ← ∅ 3 i ←4 while not converged do f

Original: ( (cid:126)a, s ) ← J t ( (cid:126)a, s t ) + λ 1 D KL ( π θ ||

Original: (cid:126)a ) + λ 2 (cid:107) (cid:126)a (cid:107) ; // see Eqns.

Original: 2, 3, and τ CEM ← n Rollout with iCEM π ( f ( (cid:126)a, s ) , π θ )

Original: ; // τ is the resulting trajectory add τ CEM to D θ ← train policy π θ on D τ π ← Rollout with π θ τ DAgger ← relabel actions in τ π with iCEM π

Original: ( f ( (cid:126)a, s ) , π θ ) ; // DAgger add τ DAgger to D θ ← train policy π θ on D i ← i + 1 ; // one APEX iteration 14 return π θ

Original: H UMANOID S TANDUP : (OpenAI Gym Brockman et al.
Revised: H UMANOID S TANDUP : (OpenAI Gym Brockman et al.

Original: (2016) v2)
Revised: (2016) v2)

Original: A humanoid robot is initialized in a laying position.
Revised: A humanoid robot is initialized in a laying position.

Original: The goal is to stand-up without falling, i.e. reaching as high as possible with the head.
Revised: The goal is to stand-up without falling, i.e. reaching as high as possible with the head.

Original: We use a task horizon of 500.
Revised: We use a task horizon of 500.

Original: F ETCH P ICK &P LACE (sparse reward): (OpenAI Gym v1)
Revised: F ETCH P ICK &P LACE (sparse reward): (OpenAI Gym v1)

Original: A robotic manipulator has to move a box, randomly placed on a table, to a randomly selected target location.
Revised: A robotic manipulator has to move a box, randomly placed on a table, to a randomly selected target location.

Original: The reward is only the negative Euclidean distance between box and target location, so without moving the box there is no reward.
Revised: The reward is only the negative Euclidean distance between box and target location, so without moving the box there is no reward.

Original: D OOR (sparse reward): (DAPG1 project Rajeswaran et al.
Revised: D OOR (sparse reward): (DAPG project Rajeswaran et al.

Original: A simulated 24 degrees of freedom ShadowHand has to open a door with a handle.
Revised: A simulated 24 degrees of freedom ShadowHand has to open a door with a handle.

Original: The reward is the sum of door opening, a quadratic penalty on the velocities, and a bounty for opening the door (in contrast to Rajeswaran et al.
Revised: The reward is the sum of door opening, a quadratic penalty on the velocities, and a bounty for opening the door (in contrast to Rajeswaran et al.

Original: there is guidance of the hand to the handle).
Revised: (2018) where a guidance of the hand to the handle was used).

Original: H ALFCHEETAH R UNNING : (OpenAI Gym v3)
Revised: H ALFCHEETAH R UNNING : (OpenAI Gym v3)

Original: A half-cheetah agent should maximize its velocity in the positive x-direction.
Revised: A half-cheetah agent should maximize its velocity in the positive x-direction.

Original: In contrast to the standard setting, we prohibit a rolling motion of the cheetah by heavily penalizing large absolute angles of the root joint.
Revised: In contrast to the standard setting, we prohibit a rolling motion of the cheetah by heavily penalizing large absolute angles of the root joint.

Original: In the standard setting numeric instabilities in the simulator are exploited by iCEM.
Revised: In the standard setting numeric instabilities in the simulator are exploited by iCEM.

Original: In Fig.
Revised: In Fig.

Original: 4, we compare APEX against several imitation learning baselines:
Revised: 4, we compare APEX against several imitation learning baselines:

Original: Behavioral Cloning (BC) (iCEM BC), DAgger (iCEM DAgger), and BC from iCEM π with guidance cost (ﬁxed λ ) and warmstarting (iCEM π -GPS).
Revised: Behavioral Cloning (BC) (iCEM BC), DAgger (iCEM DAgger), and BC from iCEM π with guidance cost (ﬁxed λ ) and warm-starting (iCEM π -GPS).

Original: For reference, we also provide  performance of SAC 2 (Haarnoja et al., 2018) as a model-free RL baseline, to get an idea of the difﬁculty of the learning tasks.
Revised: For reference, we also provide the performance of SAC 1 (Haarnoja et al., 2018) as a model-free RL baseline, to get an idea of the difﬁculty of the learning tasks.

Original: Note that we use partially sparse rewards settings in F ETCH P ICK &P LACE (only box to goal reward) and D OOR (sparse reward) (no hand-to-handle reward) which makes them particularly challenging.
Revised: Note that we use partially sparse rewards settings in F ETCH P ICK &P LACE (only box to goal reward) and D OOR (sparse reward) (no hand-to-handle reward) which makes them particularly challenging.

Original: However, for F ETCH P ICK &P LACE , a specialized method for goal-reaching tasks using hindsight relabeling (DDPG+HER) (Andrychowicz et al., 2017) is able to solve the task.
Revised: However, for F ETCH P ICK &P LACE , a specialized method for goal-reaching tasks using hindsight relabeling (DDPG+HER) (Andrychowicz et al., 2017) can solve the task.

Original: In H UMANOID S TANDUP the performance of SAC marks the behavior of just sitting.
Revised: In H UMANOID S TANDUP the performance of SAC marks the behavior of just sitting.

Original: This is a local optima and is hard to escape for gradient based methods.
Revised: This is a local optima and is hard to escape for gradient based methods.

Original: APEX manages to stand up but cannot balance for long, presumably because there is a large variety in the ways the robot can fall.
Revised: APEX manages to stand up but cannot balance for long, presumably because there are many ways the robot can fall.

Original: Notably, the D OOR  environment, with its 24 DoF Shadow hand, is solved by our method with  high success-rate.
Revised: Notably, the D OOR (sparse reward) environment, with its 24 DoF Shadow hand, is solved by our method with a high success-rate.

Original: A very interesting effect of our approach is that the iCEM π expert working inside of APEX improves with the policy, as seen in Fig.
Revised: [Figure]

Original: 4 (right).
Revised: [Figure]

Original: As expected, with raising compute-budget of iCEM, the unguided experts get high performance.
Revised: A very interesting effect of our approach is that the iCEM π expert working inside of APEX improves with the policy, as seen in Fig.

Original: However, when guided with a policy the performance raises (dashed blue lines) with the policy performance, even if the policies themselves are not very strong yet.
Revised: When guided by a policy, the performance raises ( ) with the policy performance, even if the policies themselves are not very strong yet.

Original: Together, APEX is able to shrink the gap between expert and policy performance and yields strong results, see also the illustrative behaviors in Fig.
Revised: Together, APEX is able to shrink the gap between expert and policy performance and yields strong results, see also the illustrative behaviors in Fig.

Original: [Figure]
Revised: The policy can achieve signiﬁcantly higher performance than iCEM, as prominent in the low-budget case in H ALFCHEETAH R UNNING presented in Fig.

Original: [Figure]
Revised: 4. A BLATIONS

Original: Are all components of APEX required?
Revised: Are all components of APEX required?

Original: We perform several ablations to investigate this question.
Revised: We perform several ablations to investigate this question.

Original: Figure 5 shows the performance when removing each individual component from APEX. Using the adaptive guidance cost instead of a ﬁxed λ is only important in the sparse reward environments, as expected. 
Revised: Figure 6 shows the performance when removing each individual component from APEX, as detailed in Appendix B. Using the adaptive guidance cost instead of a ﬁxed λ is only important in the sparse reward environments, as expected. In case of dense rewards, the λ adaptation does perform identically.

Original: Removing warm-starting has a drastic effect in all environments.
Revised: Removing warm-starting has a drastic effect in all environments.

Original:  Not adding policy samples to the optimization has counter-intuitive effects in H ALFCHEETAH R UNNING : it performs initially better than with adding the samples, but the experts do not improve with the policyin this case such that the asymptotic performance is lower.
Revised: APEX without DAgger is also much worse. Not adding policy samples to the optimization has an interesting effect. At ﬁrst glance the policy performance is higher, however, asymptotically the full APEX is better or on par. The reason is shown in Fig.

Original:  APEX without DAgger is also much worse.
Revised: 7 where we report the expert performance of APEX and APEX without policy samples. The expert in the latter case only marginally improves upon the standard iCEM.

Original: Not using GPS is not shown, but has very low performance.
Revised: Thus, the policy is limited to the iCEM performance, whereas in APEX it can go beyond, see Fig.

Original: C ONCLUSIONS
Revised: H ALFCHEETAH

Revised: [Figure]

Revised: UNNING

Original:  We study the problem of policy extraction from model-based trajectory optimizers using CEM – a zero-order blackbox method – popular in recent model-based RL.
Revised: [Figure] [Figure]  C ONCLUSIONS We study the problem of policy extraction from model-based trajectory optimizers using CEM – a zero-order blackbox method – popular in recent model-based RL.

Original: Our method (APEX) is able to extract strong policies for hard robotic tasks which are especially challenging for model-free RL methods.
Revised: Our method (APEX) is able to extract strong policies for hard robotic tasks which are especially challenging for model-free RL methods.

Original: This is achieved by imitation learning from a guided CEM expert, where both policy and expert mutually propel themselves to higher performances.
Revised: This is achieved by imitation learning from a guided CEM expert, where both policy and expert mutually propel themselves to higher performances.

Original: With this work, we want to propose a promising stepping stone towards learning high-performing policies for real robots, where speed and minimal interaction with the system are more important than asymptotic performance.
Revised: With this work [Figure] we want to propose a promising stepping stone towards learning high-performing policies for real robots, where speed and minimal interaction with the system are more important than asymptotic performance.

Original: Model-based RL methods have been shown to be very sample efﬁcient (Chua et al., 2018;
Revised: Model-based RL methods are very sample efﬁcient (Chua et al., 2018;

Original: Wang & Ba, 2020) in terms of actual environment interactions, but are too slow to run realtime yet.
Revised: Wang & Ba, 2020) in terms of actual environment interactions, but are too slow to run in real-time yet.

Original: With the recently proposed improved CEM (Pinneri et al., 2020), the possibility for real-time planning became feasible, but still needs huge computational resources.
Revised: With the recently proposed improved CEM (Pinneri et al., 2020), the possibility for real-time planning became feasible but still needs huge computational resources.

Original: Extracting high performing policies is a promising route to success, where we provide here an important ingredient.
Revised: Extracting high performing policies is a promising route to success, where we provide here an important ingredient.

Original: Although we are using the simulator as  internal model to study the policy extraction problem in isolation, we believe our method will transfer to the model-based RL setup with learned models.
Revised: Although we are using the simulator as an internal model to study the policy extraction problem in isolation, we believe our method will transfer to the model-based RL setup with learned models and actually beneﬁt from it.

Original: For  real systems, models and policiescould also be pretrained in simulation and then adapt to the real system.
Revised: For one, the core problems with stochastic optimizers are identical. Another reason is that a powerful optimizer like CEM ends up ﬁnding brittle solutions that are overﬁtted to the dynamics of the simulator and make imitation hard. Learned models can prevent this from happening. Another route to real robot applications is to pretrain models and policies in simulation and then adapt to the real system and extracting policies with APEX.

Original:  A PPENDIX 
Revised: Similarly the sim-to-real transfer can be done without learned models but using the ground-truth simulations as done here to get a strong initialization for ﬁne-tuning on the real hardware. One might ask: why adopting a model-based planning method if we are using only a simulator? Because planning with population-based optimizers can produce nearly-optimal solutions and the extracted policy (if able to match expert’s performance) can beat model-free baselines like SAC, which can get stuck in local minima in high-dimensional systems, as happened in the complicated H UMANOID S TANDUP task. A PPENDIX A P ERFORMANCE T ABLES

Original: We report the performance numbers for our experiments in Tab. 1, while the expert/teacher settings are indicated in Tab. 2, and policy settings are in Tab. 3.
Revised: We report the performance numbers for our experiments in Tab. 1, while the expert/teacher settings are indicated in Tab. 2, and policy settings are in Tab. Wall-clock times are in Tab. 4. [Table]

Revised: for different environments are taken from Pinneri et al.

Original: [Table]
Revised: [Table]

Revised: h

Revised: [Table]

Original: [Table] 
Revised: [Table] − [Table] [Table]

Revised: (H ALFCHEETAH R UNNING ), 100 (H UMANOID S TANDUP )

Original: [Table]
Revised: [Table]

Revised: 25 (F ETCH P ICK &P LACE ), 100 ·

Revised: [Table]

Revised: (D OOR )

Revised: [Table]

Revised: B A BLATION EXPERIMENTS

Revised: In order to understand which components of APEX are affecting the performance of the policy extraction and the expert

Revised: [Table]

Revised: several ablations were carried out.

Revised: The results of this ablation studies are shown in Fig.

Revised: 6 and Fig.

Revised: In the following

Revised: [Table]

Revised: the implementation details of the different ablation studies are discussed.

Revised: APEX ( λ -ﬁxed): Instead of using the adaptive scheme for λ j that were proposed in Eq.

Revised: 4, its value is set to a constant value, with a value chosen to work well in the respective environment.

Revised: APEX w/o DAgger: Instead of using DAgger for policy extracting, plain behavioral cloning is used.

Revised: In DAgger, states visited by the policy are relabeled with actions from the expert and added in addition to the expert data to the training dataset of the policy.

Revised: In our ablation, only the data from the expert is added to the training dataset of the policy.

Revised: In case of DAgger, the policy gets twice the amount of data as in the case of behavioral cloning because in each iteration data from the expert and relabeled data from the policy is added to the dataset.

Revised: In both cases, the same number of gradient-steps are performed during training of the policy.

Revised: APEX w/o warmstarting: Instead of initializing the means of all action dimensions along the planning horizon with the actions from the policy at the beginning of each rollout, means along action dimensions and the planning horizon are initialized with zero.

Revised: After each planning step, the mean of action t is not initialized with the action from the policy but repeats the last action at time-step t − 1 .

Revised: APEX w/o policy samples: No additional sample trajectory with actions from the policy is added to the other samples during planning (see Alg. 1).

Revised: To clarify in the full version with policy samples one of the random samples is overwritten to keep N samples.

Revised: C E XPERT AND P OLICY I NTERPLAY

Revised: In Fig.

Revised: 8 we report the performance of the policy and the experts, for different compute budgets.

Revised: As discussed in the main paper, it is interesting that the expert inside APEX improves with the policy.

Revised: The can lead to a higher policy performance than the original iCEM with the same compute budget.

Revised: D A DAPTIVE λ IN GPS COST

Revised: In Fig.

Revised: 9, we report how the adaptive λ parameter in the GPS cost of APEX changes during planning for a weak and medium policy, respectively, in case of F ETCH P ICK &P LACE .

Revised: In both runs, we ﬁxed the goal and target locations.

Revised: In case of the week policy, no successful solution is found in the ﬁrst few time steps, meaning the min cost is the same as the max cost among elites.

Revised: Hence, λ is zero allowing the optimizer to freely explore without being hampered by the weak policy.

Revised: In case of the

Revised: H ALFCHEETAH R UNNING

Revised: H ALFCHEETAH R UNNING

Revised: H ALFCHEETAH R UNNING (

Revised: [Figure]

Revised: better policy, a solution is found early on possibly because optimizer

Revised: [Figure]