UC-D IFF OSI: U NIVERSAL C ONTROLLERS WITH D IFFERENTIABLE P HYSICS FOR O NLINE S YSTEM I DENTIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Creating robots that can handle changing or unknown environments is a critical step towards real-world robot applications.
Existing methods tackle this problem by training controllers robust to large ranges of environment parameters (Domain
Randomization), or by combining “Universal” Controllers (UC) conditioned on environment parameters with learned identiﬁcation modules that (implicitly or explicitly) identify the environment parameters from sensory inputs (Domain Adaptation).
However, these methods can lead to over-conservative behaviors or poor generalization outside the training distribution.
In this work, we present a domain adaptation approach that improves generalization of the identiﬁcation module by leveraging prior knowledge in physics.
Our proposed algorithm, UC-DiffOSI, combines a UC trained on a wide range of environments with an Online System
Identiﬁcation module based on a differentiable physics engine (DiffOSI).
We evaluate UC-DiffOSI on articulated rigid body control tasks, including a wiping task that requires contact-rich environment interaction.
Compared to previous works,
UC-DiffOSI outperforms domain randomization baselines and is more robust than domain adaptation methods that rely on learned identiﬁcation models.
In addition, we perform two studies showing that UC-DiffOSI operates well in environments with changing or unknown dynamics.
These studies test sudden changes in the robot’s mass and inertia, and they evaluate in an environment (PyBullet) whose dynamics differs from training (NimblePhysics).
I NTRODUCTION
In order for robots to shine in real-world applications, they need to handle ever-changing and unpredictable situations in real environments.
For instance, a robot waiter should be able to serve a new type of dish without spilling food, and an autonomous vehicle should take a person safely to an unvisited destination.
Creating artiﬁcial agents that can operate in changing and unknown environments is a longstanding problem in the robotics community.
The collective wisdom of the robotic research community in recent years indicates that enabling learning agents to work in changing and unknown environments is not about making one big breakthrough, but rather making many small but informed decisions.
One general approach advances control policies such that they can operate more robustly (Tan et al., 2018) or more adaptively (Cully et al., 2015) in testing environments.
However, these methods usually exhibit sub-optimal task performance or require additional ﬁne-tuning in the target environment.
Alternative approaches advance simulation techniques to bring the training environment closer to the testing one prior to learning a control policy, such as training a dynamics model (Jiang et al., 2021) or identifying simulation parameters (Tan et al., 2018) from data.
These methods are often used in ofﬂine settings as learning or identifying an accurate simulation model can be time consuming.
Thus, they cannot handle changing environments naturally.
Work such as Yu et al.
(2017) have also investigated methods that learn models to perform online system identiﬁcation.
However, a learned model often does not generalize well to unknown environments or those not seen during training.
Recent developments in differentiable physics simulation potentially offer a more effective way to address these challenges by advancing both control and simulation techniques.
By utilizing fast computation of analytical gradients, one can devise more computationally and sample efﬁcient optimal control and system identiﬁcation algorithms.
Recent differentiable physics simulators, such as
NimblePhysics (Werling et al., 2021), provide fast computation of analytical gradients in the face of constraint satisfaction and non-differentiable contact handling.
These enable generic gradient-based optimizers to solve contact-rich optimal control problems.
While promising, differentiable physics simulation does not solve the fundamental problem of multiple local minima due to ill-conditioned cost functions, often exacerbated by long-horizon and highly nonlinear differential equations.
This paper introduces a new approach for creating resilient and adaptive agents by combining differentiable physics simulation for online system identiﬁcation and reinforcement learning for ofﬂine policy training.
Online system identiﬁcation can be formulated as a short-horizon, local optimization problem, but must be solved fast.
This plays to the strength of differentiable physics simulation which provides analytical gradients efﬁciently, while avoiding the pitfalls of poor cost function landscapes.
On the other hand, for challenging control problems with long-horizon cost functions, we resort to a reinforcement learning approach leveraging samples (“rollouts”) generated ofﬂine at scale to train a control policy.
We explore many possible situations the agent might encounter when operating in the testing environment by varying the simulation parameters during training and learning a Universal Controller (UC) conditioned on the simulation parameters.
At test time, we use differentiable physics simulation to continuously optimize the simulation parameters based on the most recent history of observations (DiffOSI).
The optimal simulation parameters will “modulate” the universal policy to output the optimal action for the currently identiﬁed environment.
We evaluate our approach on two robotic control tasks, a cartpole balancing problem and a robot arm table wiping task involving rich contact phenomena.
We show that our proposed approach combining a Universal Controller and a Differentiable physics-based Online System Identiﬁcation module (UC-DiffOSI) can outperform pure learning-based or traditional system identiﬁcation methods.
Finally, we demonstrate that our approach can be applied to environments with changing dynamics or un-modeled effects.
R ELATED W ORK
Deep Reinforcement Learning and Domain Randomization.
Deep reinforcement learning has been proven to be effective in learning complex motor skills for simulated robots, such as running (Yu et al., 2018)
, parkour (Heess et al., 2017), and dressing (Clegg et al., 2018).
However, these controllers often perform poorly on real robot hardware due to the discrepancies between the simulated and real environment, also known as the sim-to-real gap (Neunert et al., 2017).
Domain randomization of the simulation physics parameters has been extensively explored to help the simulation-trained controller transfer to a different target environment, where a robust control policy is trained to perform well for a wide variety of simulated environments (Peng et al., 2017; Tan et al., 2018;
Hwangbo et al., 2019; Exarchos et al., 2020; OpenAI et al., 2019).
However, policies trained from domain randomization often exhibit over-conservative behaviors, leading to sub-optimal performance (Tan et al., 2018).
Different from these methods, we develop a domain adaptation approach by training adaptive controllers that can adjust behavior for different environments using an estimation of the environment parameters.
This enables our controller to achieve better performance than a domain randomization controller.
Domain Adaptation.
To achieve better task performance in novel situations, researchers have developed adaptive controllers that can adjust behavior for different environments.
Szita et al.
(2003) showed that Q-learning, using event-learning, can ﬁnd near-optimal policies in varying environments.
Heess et al.
(2015) demonstrated that control policies modiﬁed to use recurrent networks are also capable of dealing with unknown kinematic parameters such as link lengths.
Xu et al.
(2020) presented a deep reinforcement learning method that encodes the dynamic context online to achieve a stable non-planar pushing task controller.
Yu et al.
(2017) proposed a system using a Universal
Policy and Online System Identiﬁcation (OSI) function to explicitly incorporate model parameters to adapt to varying environments.
These methods usually identify the environment parameters (explicitly or implicitly) and then adjust the controllers to adapt to the new environment.
Differentiable Simulation.
In recent years, researchers have built more efﬁcient and featurecomplete differentiable physics engines.
These engines support 3D rigid body and contact constraints between spheres and planes (Degrave et al., 2016), analytic differentiation of a linear com-
[Figure]
plementarity problem (de Avila Belbute-Peres et al., 2018), modeling soft bodies via a differentiable real-time differentiable Material Point Method (Hu et al., 2018), support differentiable cloth simulation (Liang et al., 2019), optimize for large numbers of objects and contact interactions (Qiao et al., 2020), and support articulated rigid bodies with contact (Werling et al., 2021).
Prior work, such as Toussaint et al.
(2018) and Heiden et al.
(2019), also showed that differentiable physics can be integrated for end-to-end controller learning, in addition to parameter learning.
Jatavallabhula et al.
(2021) further integrated differentiable rendering to remove dependency on 3D vision in an end-to-end learning pipeline.
M ETHODS
Our goal is to design a system that can handle changing or unknown dynamics in the environment.
The true dynamics in the target environment can be described by x t +1 =
f µ ( x t , τ t ) , where x = ( q , ˙ q ) denotes the robot’s sensed states and their time derivatives, and τ denotes the control actions.
f µ evolves the system from timestep t to t + 1 with dynamics parameters µ .
We aim to predict optimal controls τ ∗ t which maximize task performance.
The controls are predicted in the ﬁrst part of our system, the universal controller (UC): ( x , µ ) (cid:55)→ τ .
The second part is a differentiable physics engine that performs online system identiﬁcation (DiffOSI): { ( x i , x i +1 , τ i ) } (cid:55)→ µ . The overview of the system is shown in Figure 1.
Together, they form a robust controller capable of handling unknown or changing environment dynamics.
3. L EARNING A U NIVERSAL C ONTROLLER
Universal Controller (UC) augments a regular robotic controller by conditioning it on parameters of the environment µ , such as friction coefﬁcient or robot payload.
This information is crucial for the controller to select appropriate actions for different environments, yet are non-trivial to infer directly from sensory input.
By providing this additional information to the UC, we expect it to outperform a regular policy given the true environment parameters.
A successful UC should perform near-optimally for a wide range of µ ’s. Given that the best way to obtain a control policy can be different across tasks, the training of UC largely depends on the task to be performed.
In this work, we tailor the training of UC to two control tasks of interest: cart pole balancing and table wiping.
For the cart-pole balancing problem, we want to obtain a controller that directly sends torque commands to the robot at high-frequency.
Thus, we directly apply a reinforcement learning approach to obtain a Universal Control Policy as done in Yu et al.
On the other hand, for the table wiping problem, we adopt a hierarchical control structure where the learned UC needs to modulate the parameters of a low-level admittance controller per wiping motion.
A black-box optimization technique is more suitable for this low-frequency problem.
More details on how we train our UPs can be found in Section 4.
Note that we train our UC with a set of training environments g ˆ µ , which approximates the target environment f ¯ µ .
¯ µ and ˆ µ need not represent the same set of parameters, and neither do f ¯ µ
and g ˆ µ need to represent the same model, as the exact governing equations of f ¯ µ is usually unknown.
We aim to use a g ˆ µ diverse enough to train a robust UC and expressive enough to approximate all possible trajectories evolved with f ¯ µ .
3. D IFFERENTIABLE P HYSICS FOR O NLINE S YSTEM I DENTIFICATION
We use differentiable physics to identify some unknown physics parameters ˆ µ ∈ R g that parameterize the dynamics of the system.
The numerical modeling, x t +1 =
g ˆ µ ( x t , τ t ) , in the differentiable physics engine should be the same as in UC training approximating the target environment dynamics, x t +1 =
f ¯ µ ( x t , τ t ) .
This way, the nature of the UC’s inputs stays consistent.
To perform system identiﬁcation, DiffOSI requires ﬁrst collecting a small number of samples
¯ X = { ¯ x t , ¯ τ t } from target environment f ¯ µ .
DiffOSI uses these samples to optimize for a ˆ µ that minimizes the differences between the resulted trajectory { ˆ x t , ˆ τ t } and the target state-action history { ¯ x t , ¯ τ t } .
DiffOSI requires a minimum of two samples, but if the problem is nondeterministically underconstrained (e.g., in the presence of contact), more samples (e.g., 30-50) may be required to exercise all dynamics of the system.
At the beginning (ﬁrst iteration k = 0 ), we initialize ˆ µ = µ 0 (e.g., mean of expected distribution).
For each iteration of DiffOSI optimization k , we execute the UC for a certain number of steps
T k ≤ | ¯ X k | using actions predicted with UC ( ¯ x k , ˆ µ k ) .
With the collected samples ¯ X k , we use
DiffOSI to optimize for the ˆ µ k that minimizes the following objective function:
[Equation]
t +1 ) , (1) where (ˆ q t +1 , ˆ˙ q t +1 )
= g ˆ µ k
(¯ q t , ¯˙ q t , ¯ τ t ) (2) and φ is any differentiable distance function.
A differentiable physics engine enables the computation of gradients of L with respect to the unknown parameters µ :
[Equation]
Our system is agnostic to the choice of the differentiable physics engine.
We use the Nimble differentiable physics engine (based on DART) by Werling et al.
(2021), which has the advantage of being able to handle articulated rigid bodies and differentiate through contact.
E XPERIMENTS 4. T ASK E VALUATION O VERVIEW
We compare our proposed algorithm (UC-DiffOSI) to six baseline methods:
Domain Randomization (DR): Optimize a controller in an environment where dynamics parameters are randomized.
UC-Random: UC given random parameters as input.
UC-Average: UC given the middle parameter of the training range as input.
UC-MLP (Yu et al.
UC given parameters predicted with an MLP.
UC-CMA-ES: UC given parameters optimized using CMA-ES (Hansen, 2016).
UC-Oracle: UC given ground truth parameters as input.
to address the following questions:
• Does UC-DiffOSI outperform Domain Randomization (DR)?
• Is DiffOSI more robust than MLP, in generalization to new environments?
• Is DiffOSI more efﬁcient than CMA-ES? [Table]
We compare our proposed algorithm with baseline methods in both task performance (total toward over one episode) as well as prediction accuracy for the environment parameters (mean absolute error and mean absolute relative error)
[Table]
4. C ARTPOLE
Task Description.
We ﬁrst evaluate our algorithm on the cartpole task from OpenAI Gym (Brockman et al., 2016), where the goal is to balance a pole on a cart-on-track.
The state space of the cartpole task consists of the position of the cart and pole q
= ( x, θ ) , as well as their velocities ˙ q
= ( ˙ x, ˙ θ ) .
To create different environment variations, we offset the center of mass of the pole from its geometric center by a random displacement ( µ , 0 .
2 µ ) in 2D, with µ uniformly sampled within the range [ − 0 .
6 m, 0 .
6 m ] .
This is to mimic attachments of different objects on the pole We randomly initialize the q and ˙ q of the cart and the pole, with x ∈ [ − 0 .
05] , ˙ x ∈ [ − 0 .
05] , θ ∈ [ − 0 . 05] , ˙ θ ∈ [ − 0 .
The controller can apply an impulse of [ − 500 N, 500 N ]∆ t to the cart at 50 Hz.
We deﬁne the task performance metric as the number of simulation steps where | θ | ≤ π/ 2 (where θ = 0 means the pole is upright) and | x | ≤ 2 .
5 are satisﬁed.
Controller Details.
As a classic control problem, there a numerous ways to design a controller for the cartpole task.
In this work, we leverage a reinforcement learning-based approach.
Speciﬁcally, we use Proximal Policy Optimization (PPO)
(Schulman et al., 2017) to train a control policy (MLP with 2 layers, 64 hidden units each) that takes the cartpole state s = ( q , ˙ q ) as input, and predicts the appropriate control force applied to the cart.
For the domain randomization (DR) baseline, we train the control policy with ˆ µ randomly selected at the beginning of each training episode.
We then augment the policy input with ˆ µ to obtain the universal controller (UC).
By providing this critical information about the environment, the policy can better decide the optimal action to take, which is demonstrated in our evaluation results.
System Identiﬁcation.
For all methods, we use the observations containing ( q t , ˙
q t , τ t , q t +1 , ˙ q t +1 ) .
For the UC-MLP baseline, we train an MLP (3 hidden layers with 256, 128, and 64 hidden units respectively) to predict ˆ µ from the states and actions in the history.
We use a dataset collected by executing ˆ τ =
UC ( x , ˆ µ ) in the cartpole simulation environment g ˆ µ for uniformly sampled µ .
Both UC-CMA-ES and UC-DiffOSI optimize ˆ µ to minimize the MSE between the result and target trajectories.
Results.
Results for the cartpole task can be found in Table 1.
Our approach achieves better results than the baseline methods in terms of task performance and is close to the upper bound (UC-Oracle).
This suggests our algorithm can successfully infer the environment parameters from the input history of past observations and actions.
This is further supported by the low prediction error of our algorithm.
We observe that UC-MLP can achieve low mean absolute error in predicted ˆ µ , yet performs worse in relative prediction error.
This is possibly due to the model learning to focus on regions where the true ˆ µ has higher magnitude, which also leads to inferior task performance.
This can possibly be mitigated by tuning the loss function or training data further, yet these are task spe- [Figure] ciﬁc, non-generalizable processes.
In addition, we also evaluate MLP-Narrow, which trains an MLP model to predict the environment parameter, but with ˆ µ sampled from a narrower range ( [ − 0 .
As shown in Table 1, MLP-Narrow does not generalize to parameters outside the training range. 4. H ALF C HEETAH
In this section, we evaluate our method on a 7-link, 6-DoF 2D half cheetah task (Figure 3a). Task Description. We adapt the half cheetah environment from OpenAI Gym (Brockman et al., 2016), and replace the simulator with the NimblePhysics simulator. The mass of the half cheetah’s torso is unknown, and uniformly sampled within the range [1 , 50] kg (where the original torso mass is 4.9 kg).
Controller Details.
Similar to cartpole, we use the PPO algorithm to train a control policy (MLP with 2 layers, 256 hidden units each).
We train 6 different seeds for each approach, DR and UC, where UC is trained with ground truth mutrue , corresponding to the UC-Oracle results.
System Identiﬁcation.
Both the MLP and DiffOSI methods for system identiﬁcation use the same inputs as those used in the cartpole task in Section 4.2.
The MLP (2 hidden layers with 512 and 128 hidden units respectively) is trained on a dataset of 600K examples generated from rolling out trajectories from the trained UC-Oracle models over uniformly sampled µ .
The DiffOSI module uses NimblePhysics as its simulator, and optimizes ˆ µ (initialized to the mean of the distribution) to minimize the L2 norm between the predicted and target next state.
Results.
In Figure 3b, we show that UC-Oracle is able to outperform pure domain randomization.
This shows that observation of physical parameters is beneﬁcial for this task.
Evaluations of various approaches using the trained DR and UC policies are shown in Table 2.
We show that UC-DiffOSI is able to perform comparably to the upper bound, UC-Oracle, due to its low parameter estimation errors.
UC-MLP has, on average, higher parameter estimation errors than UC-DiffOSI, but lower parameter estimation errors than random or average baselines (Figure 3c).
However, we found that for some trajectory rollouts, UC-MLP errors tend to magnify over time (Figure 3d is one such example).
This shows the weakness of UC-MLP – the UC depends on a perfect MLP for system identiﬁcation, and any slight mistake the MLP makes can quickly cause errors to compound over time.
These examples can lead to an even larger scale of failure at the task performance level (relative to the scale of µ estimation errors).
Thus the overall task performance of UC-MLP is worse than
UC-Random and UC-Average.
We also analyze the runtime of UC compared to an MPC baseline.
To achieve reasonable task performance, MPC baseline needs to have a planning horizon of more than 200 timesteps and replan every 40 timesteps.
Each trajectory optimization in MPC takes 100-300 milliseconds to solve while evaluating UC for 40 timesteps only takes roughly 40 milliseconds.
As such, MPC is not a suitable method for a setting like ours where realtime performance is important.
[Figure]
[Table]
4. T ABLE W IPING
Task Description.
In this task, we wipe a tabletop using a wiping tool attached to a robot arm’s end effector, as shown in supplementary Figure 5a.
The wiping tool joint has unknown stiffness and damping coefﬁcients µ = { ( k p , k d ) } .
To perform a wipe, we track a scripted wiping trajectory { q trajt } with an admittance controlled robot arm.
The goal is to maintain a target contact force on the contact normal direction f z,goal during the whole wiping process, i.e., to minimize the MSE between f z,sensed and f z,goal for all observed states when the wiping tool is in contact with the tabletop. The system overview can be found in supplementary Figure 5b with details as follows. [Table]
Controller Details.
The task controller is deﬁned by the combination of the admittance control law and the admittance control parameters θ =
( k τ , k ξ ) .
For the DR baseline [Table] we optimize for the optimal admittance control parameters θ ∗ in the simulation environment g ˆ µ .
The goal of the UC is to apply admittance control with the optimal control parameters mapped from the identiﬁed ˆ µ .
We collect a dataset (85 examples) to train the DNN-parameterized UC (2 hidden layers, 512 units each with ReLU activation functions and a linear FC layer).
The dataset contains paired examples of ˆ µ true and corresponding optimal control parameters θ ∗ =
( k ∗ τ , k ∗ ξ ) .
System Identiﬁcation.
Our goal with system identiﬁcation for the wiping task is to predict dynamics parameters ˆ µ to condition the UC.
For all approaches with UC, we ﬁrst collect a trajectory in the environment (parameterized by some unknown ¯ µ ) using canonical admittance control parameters ˜ θ .
From the collected trajectory, we uniformly sample 50 segments from the history to perform system identiﬁcation, where each segment contains ( q t , q t +1 , q trajt ) .
Both UC-CMA-ES and UC-DiffOSI optimize for the ˆ µ that minimize the MSE between the resulted and target sampled segments from the collected trajectory.
For UC-MLP, we feed the same 50 segments to the MLP and predict ˆ µ .
The MLP (same architecture as UC) is trained on a dataset mapping trajectory segments to ˆ µ true .
The ˆ µ true is uniformly sampled, and the trajectories are generated by executing the UC with ˆ µ true
(cid:55)→ ˆ θ .
Results.
We evaluate all baselines on the wiping task and report our results in Table 3.
For reference, we also include UC using random or average ˆ µ as baselines.
UC with average ˆ µ can be seen as an ofﬂine system identiﬁcation method.
For simulation, we use a timestep of 1e-3s.
For all our experiments, we set F z,goal = 3 N and uniformly sample ¯ µ ∈ [1 , 100] .
DR performs worse than all UC-based approaches, which indicates that UCs provide useful physics-awareness for the wiping task.
UC-MLP performs slightly better.
Both UC-CMA-ES and UC-DiffOSI outperform all baselines and are close to UC-Oracle.
Evaluating Higher Number Dimensions.
To further compare UC-CMA-ES and UC-DiffOSI, we evaluate on the same wiping task where µ is higher dimensional.
We increase the number of DoFs with unknown stiffness and damping coefﬁcients.
The convergence curves can be found in supplementary Figure 6.
In both the 2D and 3D optimization cases, UC-DiffOSI converges more quickly than UC-CMA-ES.
4. R OBUSTNESS TO V ARYING D YNAMICS
In this example, we evaluate whether our proposed algorithm can handle varying dynamics throughout an episode.
During each episode, we change ¯ µ at t = 250 (halfway through the episode) to a randomly sampled µ (cid:48) ∼ [ − 0 .
As shown in Table 4, our algorithm achieves notably better prediction accuracy than UC-MLP.
However, the task performance is not as good as the oracle version.
This is because in order to detect abrupt changes in the system dynamics, we need to collect sufﬁcient data with the new system, which leads to a delay in identifying the correct parameters. For [Table]
[Table] abrupt and large changes in µ such as the one we used, the task performance can be sensitive to the accuracy of the prediction [Table]
4. R OBUSTNESS TO N OVEL D YNAMICS M ODELING
To evaluate the ability of our approach to generalize to environment variations beyond the training range [Table] we apply our cartpole controller to the same task implemented in PyBullet physics engine.
Due to differences in how the two physics engines solves the equation of motion and performs integration, a policy trained in UC-DiffOSI (Nimble) does not transfer directly to PyBullet.
As shown in Figure 2b [Table] our method still outperforms baselines.
The gap is even larger between our method and DR, which suggests that in cases where the dynamics gap is larger, UC-DiffOSI can offer larger improvements.
C ONCLUSION
UC-DiffOSI is a learning-based approach for training control policies that can operate in changing and unknown environments.
Our method combines domain adaptation with a differentiable physics simulator by ﬁrst training a Universal Controller that is conditioned on the environment parameters and then using a differentiable physics engine, NimblePhysics, to identify the environment parameters from a recent history of robot sensory inputs.
By using a differentiable physics engine, we achieve efﬁcient and generalizable system identiﬁcation compared to prior methods based on learned models or traditional system identiﬁcation.
We evaluate our method on two robotic control problems: cartpole balancing and table wiping with a robot arm.
Our method achieves superior performance than the baseline methods and is able to handle changing or un-modeled dynamics.
There are promising directions that further extend our work.
For example, our algorithm currently assumes that the Universal Controller can optimally handle different environments given an accurate estimation of the environment parameters.
However, in real applications, the robot might encounter situations that are beyond the capability of a pre-trained UC.
Determining how to efﬁciently ﬁnetune a UC with limited data is thus important future work, where differentiable physics engines could play a pivotal role.
In addition, this work focuses on rigid body environments, while real-world environments are ﬁlled with objects that can deform.
An interesting direction could be to extend our approach to handle deformable objects while retaining high efﬁciency in the online system identiﬁcation.
Finally, we plan to apply our approach to real robot hardware, requiring bridging the sim-to-real gap and running the end-to-end control pipeline in real time.
R EPRODUCIBILITY S TATEMENT
To maximize reproducibility, we describe our methodology in detail in Section 3 and our experimental setup in Section 4.
The code, based upon on the open-source physics simulator (NimblePhysics (Werling et al., 2021)), will be released upon publication to facilitate future research. A L ATENCY
We additionally analyze the effect of the number of sampled used by DiffOSI on the prediction accuracy of ˆ µ .
Results are shown in Figure 4.
We compare different context window sizes (2, 3, and 5) and plot the absolute µ error as a function of the episode timestep for the half cheetah task.
We ﬁnd that a single sample (corresponding to a single step, with context window size of 2 containing states before and after the step) is sufﬁcient to obtain an accurate prediction of µ ; more samples does not help.
A single optimization step of DiffOSI takes 2-3 milliseconds.
B W IPING
In this section we show additional diagrams and results on the wiping task.
Figure 5a and Figure 5b depict the wiping task as well as an overview of the system we implemented.
Figure 6 shows a comparison between CMA-ES and DiffOSI in terms of absolute error µ predictions.
[Figure]
[Figure]
[Figure]
[Figure]