Learning Behaviors through Physics-driven Latent Imagination
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Model-based reinforcement learning (MBRL) consists in learning a socalled world model, a representation of the environment through interactions withit, then use it to train an agent.
This approach is particularly interesting in the context of ﬁeld robotics, as it alleviates the need to train online, and reduces the risksinherent to directly training agents on real robots.
However, in such approaches, theworld encompasses both the part related to the robot itself and the rest of the environment.
We argue that decoupling the environment representation (for example, images or laser scans) from the dynamics of the physical system (that is, the robot andits physical state) may be beneﬁcial: it can increase the ﬂexibility of world modelsand open the door to further robustiﬁcation.
In this paper, we apply this concept to astrong latent-agent, Dreamer.
We then showcase the increase of ﬂexibility by transferring the environmental part of the world model from one robot (a boat) to another(a rover), simply by adapting the physical model in the imagination.
We also demonstrate the robustness of our method through real-world experiments on a boat.
Introduction
Early forms of controllers, such as classic optimal control or dynamic programming, make use ofa supposedly known model of the environment.
On the learning side, Reinforcement Learning (RL)can also learn a model and use it to train an agent.
It is only during the last decade that Model-Based
RL (MBRL) algorithms capable of controlling systems using high dimension inputs, such as rawimages were developed.
These advances allowed their use on robots for solving complex tasks, onwhich traditional methods failed [1, 2].
In robotics, it is of paramount importance to learn quickly, asacquiring samples is not only ludicrously time-consuming, but can also be dangerous for both the robotand the operator overseeing the agent.
Hence, due to their high sample efﬁciency and their potentialresort to ofﬂine learning [3, 4], world models make for a compelling choice in robotics applications.
Yet, because most of the agents are meant to be used on classic benchmarks such as Atari [5], OpenAI
Gym [6] or the DeepMind control suite [7], they process either images or low proprioceptive variables(for example, position or speed), but rarely both.
In robotics, most of the works that used MBRL in thecontext of solving high-dimensions task discarded the proprioceptive variables [1, 2, 8], even thoughthey could have been acquired fairly easily.
Other works, such as [9, 10], make use of both but embedthem as a single model.
Unfortunately, neural networks being black-box models, this means that thereis no way of accessing the dynamics of the system, as it is directly integrated inside the environment.
All in all, this means that the world model learned by an agent will be unique to this system.
Unlike these studies, we argue that a world-model should be made of two components: an environment, which encompasses everything that the agent can sense using laser-scanners or cameras, and thedynamics, i.e. the set of physical equations and variables that rule the movement of the mobile robotin the world.
To do so, within the paradigm of Markov Decision Processes (MDPs), we decomposethe total state of the system in two parts, one related to the environment and one to the dynamics of therobot, S tot =
( S env t , S dyn t ) , and we assume that the transition from state to state can be decomposedas a change of the dynamics in response to the applied action, and a change of the environment in response to the new dynamics, P ( S tot t +1 | S tot t , a t )
=
P ( S dyn t +1 | S dyn t , a t ) P ( S env t +1 | S env t , S dyn t ) .
A concreteexample of why such representations are powerful is to think of car manufacturers.
All manufacturersmake different cars with different physical properties, yet those cars all share the same environment:the road.
Hence, a method like ours could allow the cars to share their environment while havingtheir own dynamics, speeding up the training process and potentially allowing the distributed trainingof heterogeneous agents.
Furthermore, it makes sense in robotics applications to use proprioceptiveinformation, as most real robotic tasks involve low level physical constraints coupled to high levelperception constraints.
In the end, MBRL is a very promising solution to bring more autonomy toreal-robots, yet we think it would be better suited to mobile robotics application if it was to nativelyaccount for the robots’ dynamic.
In this paper, we modify one of the strongest latent-model agent, Dreamer [11], and integrate thedynamics as an independent part of the world model.
As such, our world model will feature twostates: an environment state that uses low dimensions proprioceptive variables to transition fromone state to another ( P ( S env t +1 | S env t , S dyn t +1 ) ), and a physical state, that uses the action sent by the agentto transition from one state to another ( P ( S dyn t +1 | S dyn t , a t ) ).
While it might seem fairly similar to theoriginal world model with an observation of the proprioceptive input in addition to the images, itis not.
Decoupling the dynamics from the environment allows for interesting manipulations withinthe imagination process used to learn the actor.
Because the dynamics is no longer attached to theenvironment, we can now swap the dynamic model of a robot for the dynamic model of another robotas long as their proprioceptive states are of the same size..
Moreover, our method ensures that theimagination roll-out have environment-states that are consistent with the physical-states.
Eventually,we can use an analytic approximation of the robot dynamic model to alleviate the need to rely on alearned model.
Our contributions can be summarized as follows: (1) we propose a latent-state representation thatdecouples the environment from the dynamics to better ﬁt mobile robots; (2) we test this method insimulation and on a real robot and show that it robustiﬁes the agents compared to the baseline; and (3)we demonstrate that this method can also be used to transfer environment from one robot to anotherin a zero-shot setup.
All the code to reproduce those experiments is open-source and available ongithub (Appx. D.1).
Related Work
Model-Based Reinforcement Learning.
The earliest form of Model-Based Reinforcement Learning (MBRL) is probably optimal control.
In optimal control, the model of the system is typicallyknown, and an optimal controller is computed using methods such as LQR.
Until recently MBRL algorithms such as Pilco [12], among others [13], were designed to process and model low dimensionalinputs.
These methods were applied to robotic applications [14, 15] with impressive results, but theywere limited and struggled to control systems using high dimension sensory inputs such as images.
However, with the advent of deep-neural-network, MBRL was applied successfully to high dimensionsinputs, as shown in the comprehensive survey of [16].
As of today, MBRL performances are nearing ifnot exceeding the performances of model-free agents [11, 17].
The main advantage of MBRL relies onits higher sample efﬁciency when compared to the model-free agents.
Methods based on Variational
Auto-Encoders (VAE) [18], such as latent state methods [19, 11, 17], can learn robust policies thanksto the VAE stochastic states.
Unfortunately, as these methods started processing higher dimensioninputs, they also started to disregard proprioceptive information as it made solving the task easier.
Dreaming applied to real-world.
Latent-state methods have shown impressive results when applied on real robots.
[8] applies Dreamer on a 1/10th racecar with a LIDAR and compares it toother model-free methods in simulation.
They show that, on their task, Dreamer performs betterthan other model free RL algorithms, such as DDPG [20] or PPO [21].
They also demonstrate that
Dreamer transfers well from simulation to real robots, in well controlled environments.
[2] alsoapplies Dreamer on an Unmanned Surface Vessel (USV) with a LIDAR and trains it in simulation ona shore following task.
They then deploy it on the real robot, without retraining it, and evaluate therobustness of the approach by both changing the dynamics of the real systems and evaluating it indifferent weather conditions.
Both of these works also use Dreamer as their backbone, but unlikeours, they completely disregard the physical state of the system.
[1] shows an application of latentmodels to a real world system using cameras.
In [9], they rely on latent state model and integrate Φ
[Table]
[Equation]
[Table] [Table] [Table]
[Figure]
both the physical state of the system and high level distance sensors to move their robot around. However, they still consider the systems dynamics to be part of the whole model. A work that iscloser to ours is [22], in which they learn from images the Lagrangian dynamic of a system, and thenuse it to reconstruct the observed images.
In its current state, [22] is not applicable to real robots:the examples are simplistic, and make strong approximations about the dynamic model. [23] alsoshares similarities with the work presented here: they use Graph Neural Networks to perform systemidentiﬁcation as in [24], and then apply a Model Predictive Controller and RL on the predictedphysical states. The main difference with our work, is that we don’t only have a dynamic module butalso a perception module. In our method, the outputs of the dynamic module are used to predict thenext perception state, and both of their latent states are then used by the actor to decide on whichaction to apply next.
Method
[Figure] We now show how our model differs from the original implementation of Dreamer[11], which westarted from.
As in Dreamer, we solve a Partially Observable Markov Decision Process (POMDP),with discrete time steps t , for which we have continuous actions a t , high dimension sensory inputso t , low dimension proprioceptive inputs x t , and rewards r t generated by the environment.
Similarly to Dreamer, our version is articulated around three main concepts: the learning of theworld model from previous experiences (Fig. 1a), the learning of the behavior inside the imaginationprocess (Fig. 1b), and ﬁnally the application of the policy using observations from the environment tocollect new samples (Fig. 1c).
Unlike the original Dreamer, our agent does not rely on a single latent state.
Instead, weconsider that this single state can be decomposed into two states S tot t =
( S env t , S dyn t ) , where
P ( S tot t +1 | S tot t , a t ) =
P ( S dyn t +1 | S dyn t , a t ) P ( S env t +1 | S env t , S dyn t ) .
In practice, we consider that we canlearn a function q φ ( x t | S dyn t ) , this means that we can further decompose P ( S tot t +1 | S tot t , a t ) =
P ( S dyn t +1 | S dyn t , a t ) P ( S env t +1 | S env t , x t ) as we will be able to estimate x t from S dyn t within the imagination. In [11, 17], the latent dynamic model is composed of three modules: a representation modulep η ( s t | s t - 1 , a t - 1 , o t ) , a transition module q η ( s t | s t - 1 , a t - 1 ) , and a reward module q η ( r t | s t ) .
Withthis setup, both the representation and transition modules embed the dynamics of the system and theunderstanding of the environment.
This limits the capacities of the world model.
Hence, we proposeto separate the dynamics of the system from the environment.
To do so, we change the latent dynamicmodel of Dreamer to include two states: S envt the environmental state, and S dynt the physical state.
We then add two extra modules and slightly modify the inputs of the original latent dynamic model.
The new latent dynamic model is
Dynamics representation module:
[Equation]
Dynamics transition module:
[Equation]
Environment
[Equation]
[Equation]
Environment transition module:
[Equation]
Learning dynamics & Reconstructing Observations.
To learn both the dynamics and the environment, we rely on Recurrent State Space Models (RSSM) [19].
RSSMs can be seen as non-linear
Bayesian ﬁlter [25], with an observation step and a prediction step.
During the observation step, the
RSSM is given its previous state and an observation, and outputs a compact latent state.
During theprediction step, the RSSM is given its previous state and command, and outputs a compact latentstate.
This prediction step of the RSSM is the transition function, and the combination of a predictionand an observation is the representation module.
We consider two RSSMs: one learns the dynamics of the system and the other learns the environment.
( ) The RSSM that learns the dynamics takes as input both the raw proprioceptive variables and theactions.
To teach the network to embed within its latent state relevant information, we also traina decoder to reconstruct the proprioceptive variables associated with that state.
Fig.
2 showsexamples of reconstructed states.
Dynamics observation module: [Equation] Dynamics transition module: [Equation]
Dynamic reconstruction module:
[Equation]
The RSSM that learns the environment takes as input a high dimension sensory observation and the latent dynamic state.
We then train another network to reconstruct from its latent state theinputs or a projection of it.
Environment observation module:
[Equation]
Environment transition module
[Equation]
Using these two latent states, S env and S dyn , we also learn to predict the reward that will be givenby the environment q η ( r t | S envt , S dynt ) .
Regarding the optimization of these networks, we use thesame method as Dreamer [11, Sec. 4] with a caveat: the dynamics is not trained jointly with theenvironment and the reward.
To train the dynamics, we propose two options.
In the ﬁrst option, wedo not learn the dynamic model.
For instance, one could use an already learned dynamic model andkeep it ﬁxed for the whole of the training.
This is interesting as the physical model remains constantthroughout the training.
It helps learning the value and the reward model faster than if we were tolearn it from scratch.
Also, it offers the possibility to completely remove the learning and neuralnetwork components from the dynamic estimation, by using an analytical model, for instance.
Inrobotics, this makes sense, as good analytical/learned models are often already known.
If we don’tknow the dynamics, the second option is to learn it as we discover our environment.
However, to ease the optimization of the actor, the reward, and the value, we do not train the dynamics at the same rateas the rest of our agent’s neural networks.
We train the dynamics less often, but for more steps thanthe other components.
While this technique does not offer as much stability as the ﬁrst method, itstill increases the overall stability of the learning process.
Learning behaviors.
With the learning of the system dynamics and environment covered, we cannow use them in the imagination process.
The imagination generates imagined trajectories from whichthe actor is learned.
In the original Dreamer, imagined trajectories start from latent states s t obtainedfrom actual observations, and follow the predictions of the transition model s τ ∼ q ( ·| s τ - 1 , a τ - 1 )and the policy a τ ∼ q ( ·| s τ ) .
In our case, we use the dynamic transition model as a physic enginefor the imagination, and the environment transition model as the rendering engine.
This meansthat to imagine a trajectory, we start from a pair of latent states ( S envτ , S dynτ ) obtained from actualobservations, and iteratively apply the transitions models given in Eqs.
2 and 3, the policy beinggiven by a τ ∼ q ( ·| S envτ , S dynτ ) .
As this process goes along, the proprioceptive variables x τ arereconstructed from the physical state S dynτ using q φ ( ˆ x t | S dynt ) .
They are then used inside theenvironment transition model.
An important note is that, since RSSMs are VAEs, if we samplefrom the stochastic state to reconstruct the physics, then the imagined proprioceptive variables willbe noisy.
To prevent that, we take the mode of the distribution which generates smooth physicalpredictions.
Once we have a set of trajectories, we can get each pair of states ( S envτ , S dynτ ) and usethem to estimate the reward and value associated with them.
Using these, we train the policy and thevalue estimator using the same technique as the original Dreamer [11, Sec. 3].
Environment Transfer.
Unlike Dreamer, our method allows transferring environments from onerobot to another.
In this context, we consider that both robots share the same sensors and samestate-space.
We also consider that the environments in which the robots are going to evolve aresimilar.
Let us consider robot A, with already known dynamics modules p Aφ ( S dynt | S dynt - 1 , a t - 1 , x t ) ,q Aφ ( S dynt | S dynt - 1 , a t - 1 ) .
Let us consider robot B, with already known environment modules p
Bη ( S envt | S envt - 1 , x t - 1 , o t ) , p Bη ( S envt | S envt - 1 , x t - 1 ) , q Bη ( r t | S envt , S dynt ) , and a pre-collected set of samples ofrobot B interacting with its environment.
To learn the actor of robot A, using the environment of robot B, we learn entirely ofﬂine the actor, the reward, and the value, using the set of experiences collectedon robot B. It should be noted that we would not need to retrain the reward if it was rewritten asq
Bη ( r t | S envt , x t ) .
The main disadvantage of this method is that because we are using experiencescollected on robot B, the observation of the dynamics will match the dynamics of robot B and not theone of robot A.
However, as the imagination progresses the dynamics will match the one of robot
A. Another issue can arise if the robots are radically different.
In this case, the samples collectedon robot B will most likely not explore the state-space of robot
A enough to control it perfectly.
Nonetheless, as we will demonstrate later, it allows learning good preliminary policies to furtherreﬁne the agent from.
The exact algorithm used to learn the agent policy is given in Appx.
C. Besides,the state-space of both robots must be the same, which makes this method mostly aimed towardsmobile robots.
Examples of the state-space of our robots are given in Appx.
D.2.
Robots, Task & Evaluation
Robots.
To evaluate our approach we use two robots (Appx. A.1, Appx. A.2): an Unmanned Surface Vehicle (USV) and an Unmanned Ground Vehicle (UGV).
The USV is a Clearpath Heron,a small catamaran with a turbine in each hull.
There are two action dimensions: the ﬁrst elementcontrols the left turbine, the second element controls the right one.
This means that to drive in astraight line, the agent must send the same value to both turbines.
The UGV is a Clearpath Husky, askid-steer ground robot controlled with a twist input.
The size of the action of the Husky is 2, butunlike the Heron, the ﬁrst element controls the forward velocity, while the second controls the angularvelocity.
Both robots share similar sensors: a 2D laser-scanner, an RTK GPS and a 9DoF IMU.
Toprevent damaging our robots’ actuators, we saturate their acceleration before applying the output ofthe policy.
More information regarding the exact speciﬁcations of the USV and UGV can be foundin Appx.
A.1 and A.2.We teach our agent to solve a sensor-based navigation task: the robots mustfollow the shore of a lake at a ﬁxed distance while maintaining a set forward velocity.
Hence, thereward generated by the environment is made of two components: a low-level constraint, the velocity,and a high level constraint, the distance from the shore. [Table] [Figure] Task.
The exact description of the reward is available in Appx. B.
Fig. 3 illustrates the USV solving its task. To fulﬁl their tasks, the robots rely on a 2D laser-scanner and ontheir own velocity, estimated from the fusion of the RTK GPS andthe IMU through an Extended Kalman Filter (EKF).
Using the EKF,we provide to the agent three proprioceptive variables: its forwardvelocity, its lateral-velocity, and its angular-velocity.
The actionspace of both robots is [ − 1 , 1] 2 .
Evaluation.
We test our method both in simulation and on a real USV.
In simulation, to evaluatethe performance of our method, we used the different components of the reward as metrics.
Eachmodel ran for two hours and at the end of each run we collected the standard deviation, the mean, the10% quantile and 90% quantile of the rewards.
In the real world, we cannot compute the distancereward accurately: our estimation of the distance to the shore is uncertain because of the high level ofnoise of the laser data in a natural environment.
This is particularly true because the experimentswere carried out in late spring, where the luxurious vegetation makes laser-scans highly unreliable.
For this reason, we will not be evaluating the agent distance from the shore, but we will visuallyinspect the followed trajectories.
The training settings are given in Appx.
D
Simulation Results
We compare our method to the original version of Dreamer, and illustrate the beneﬁts of having aseparated state for the dynamics.
All models trained with our approach have been trained with a ﬁxeddynamic model.
This means that the dynamics was learned ahead of time and could not be reﬁnedduring the training process.
This was done to show that we could have used analytical models instead.
Unmanned Surface Vehicle.
To show the beneﬁts of using a separated state for the dynamics, weran a benchmark using the USV.
To verify that the agents perform optimally, we ﬁrst run a test wherethey are deployed in their original training environment.
Then, to evaluate how the agents behave whenthe dynamics change, we multiply the simulated robots damping factor by two.
Increasing this factormakes the robots’ dynamics softer: for the same inputs, the acceleration will be slower.
To test the robustness to perturbations in the dynamics, we also run a test with the original dynamics and a constantwater current with a speed of 0 .
4 m/s .
Finally, to further stress the agents, we multiply the dampingfactor by two and add 0 .
4 m/s of current.
The results of this benchmark can be found in Fig.
Our method is more robust than Dreamer to both dynamics change and perturbations.
Most notably,it maintains a correct velocity reward across the whole benchmark, when Dreamer fails as soon as thedynamics changes.
This suggests that images are not sufﬁcient to approximate the velocity correctly,and that by using our separated physical state, the actor is able to adapt its behavior to match thedesired velocity, despite having never seen such events in the training.
Moreover, having access to aphysical state seems to help to solve the high-level constraints, as our method outperforms the vanilla [Figure]
Dreamer on this part of the task too.
The only exception is when the damping factor is multiplied bytwo.
In this case, the lower velocity of the robot allows Dreamer to follow the shore better.
Overall,these experiments show that splitting the single latent state into two, one for the physics and one forthe environment, is not only a viable concept but also increases the robustness of the agent.
USV to UGV transfer.
To showcase the transfer capacities of our method, we learn an actorcompletely ofﬂine, using the environment of the USV and the dynamics of the UGV.
This is possibleas we separate the latent-state into those two parts, dynamics and environment.
As a reminder, the
USV and the UGV do not react to actions in the same way.
The USV is driven by sending commandsto the left and right turbines, when the UGV is driven by sending forward velocity and angularvelocity commands.
Furthermore, the dynamics of both systems are radically different: the USVglides, has a huge inertia and has a signiﬁcant amount of lateral velocity whereas the UGV has almostno slippage, no lateral velocity and no inertia.
To train our actor, we ﬁrst trained our USV to fulﬁllthe shore following task, and we then took its environment model (including the reward) along withthe samples of its interaction with the simulation.
We secondly learned the dynamics of the UGVfrom a set of past interactions with the simulation.
Finally, using the environment of the USV andthe dynamics of the UGV, we trained the actor entirely ofﬂine.
We then deployed it in simulationand compared it to an agent trained on the UGV and on the environment for an equivalent amount oftraining steps (300k).
[Figure]
Figs.
5a and .5b show the performance of the agent trained ofﬂine with environment transfer (ofﬂinetransfer, ours) compared to the agent trained online with its own environment (Dreamer).
It is worthnoting that we did not include the results of the USV actor applied directly on the UGV: it completelyfailed to solve the task because of the complete difference in the commands mapping.
From theseresults, we can see that the agent trained online achieves almost perfect results, with an averagedistance reward of 2.48.
On the other hand, our agent achieves an average reward of 2.4, whichamounts to about ± 30 cm of error on the 10 m distance it must keep from the shore.
Fig.
5c showsthe trajectory of the agents in the hardest spot of the simulation: a narrow hairpin-turn.
We can seethat the trajectories of the agent learned online are closely packed together.
On the contrary, thetrajectories of the agent learned ofﬂine are more loosely grouped, indicating that this policy is lessreliable and most likely less overﬁtted to the environment.
These approximate trajectories can beexplained by the fact that the imagination is inaccurate at its beginning.
Since the samples used togenerate the starting states of the imagination are taken from the USV, these starting states containdynamics that are not feasible on the UGV.
Yet, despite these shortcomings, we demonstrated that wecan easily transfer environments in a fully ofﬂine fashion even on robots with fundamentally differentdynamics and action mappings.
Real World Results
Finally, we evaluate our method on a real USV.
To do so, we train two agents in simulation, one with
Dreamer and one with our method, and deploy them directly on the real robot in a zero-shot setup.
Thus, the dynamics learned by our agents is only approximately matching the one of the system.
Weevaluate the performances of both agents on a whole lap around the lake.
These laps are shown in
Fig.
6a, a lap is about 1.6 km long and takes approximately 20 minutes.
During their lap, neither of theagents collided with their environment.
Regarding the trajectories, Fig.
6b shows that the trajectoryfollowed by our method is much less winding than the one followed by Dreamer.
The reason for that seems to be that the agent using Dreamer tends to overshoot more than the agent trained usingour method.
This overshooting behavior can easily be spotted on the cropped image of the lake.
On the middle right part of the image, we can clearly see the Dreamer agent overcompensating andnavigating toward trees.
This behavior cannot be seen on the agent trained using our method, whichfollows the shoreline at a much more constant distance.
It further demonstrates that accessing thedynamics makes the policy naturally robust to changes in the dynamics.
As for the velocity, the right most picture of Fig.
6c shows that the agent with a physical state (ours)is the closest from the desired forward velocity.
Additionally, our method shows a smaller spreadwhen compared to the original Dreamer.
It is interesting to note that this model can fulﬁl its taskdespite having noise on the physical inputs provided by the EKF, which is something it had neverencountered before.
Furthermore, during the experiments, we observed that our agent was makingless brutal accelerations and seemed to have a more ﬂuid way of steering the boat.
All in all, thisconﬁrms the simulation results shown in Fig.
4: our method is more robust and better matches theforward velocity target.
[Figure]
Conclusion
In this paper, we showed through both simulated and real world results that splitting the latentstate of MBRL into two sub-latent-states, one for the environment and one for the dynamics, is aviable concept.
In all our experiments, our method was trained with a ﬁxed dynamic model whichseems to conﬁrm that similar results could be achieved with analytical models, or any other systemidentiﬁcation method.
Through our experiments, we showed that explicitly adding information aboutthe dynamics of the system make the agent more robust to both perturbations and changes of thedynamics.
Finally, we showed that our method could achieve environment swaps from one robot toanother, allowing to train robots fully ofﬂine.
The learned policy could then be deployed on the targetsystem and be reﬁned.
Even though our method shows strong results, some limitations currently remain.
Most notably, thetransfer between systems will only be possible if they share the same state-space.
Another limitationis that, with the current setup, the dynamics does not explicitly depends on the environment.
Thiscould be addressed by providing either exteroceptive inputs or an access to the environment stateto the physics module.
Future work will focus on integrating this component, along with how toapply domain randomization on the robots’ dynamics within the imagination process.
This is possiblebecause we can manipulate the dynamics independently of the environment within the imaginationwhile being sure that the imagined environment will be consistent with imagined physics.
In doingso, we hope to signiﬁcantly speed up the learning process of robust actors.
We will also investigatethe integration of a physical model that uses physics as a model prior as in [26, 22, 27].