C ONTROL OF T WO - WAY C OUPLED F LUID S YSTEMS WITH D IFFERENTIABLE S OLVERS
A
BSTRACT
We investigate the use of deep neural networks to control complex nonlinear dynamical systems, specifically the movement of a rigid body immersed in a fluid.
We solve the Navier Stokes equations with two way coupling, which gives rise to nonlinear perturbations that make the control task very challenging.
Neural networks are trained to act as controllers with desired characteristics through a process of learning from a differentiable simulator.
Here we introduce a set of physically interpretable loss terms to let the networks learn robust and stable interactions.
We demonstrate that controllers trained in a canonical setting with quiescent initial conditions reliably generalize to varied and challenging environments such as previously unseen inflow conditions and forcing.
Further, we show that controllers trained with our approach outperform a variety of classical and learned alternatives in terms of evaluation metrics and generalizing capabilities.
I NTRODUCTION
Control of tasks of physical systems are a ubiquitous challenge in science.
In particular, fluids create very difficult environments which manifest themselves in simulations via the nonlinearities arising from the Navier Stokes (NS) equations.
However, advancements in this field are important for society, and impact areas such as energy, transportation and biology (Barlas & Kuik, 2007;
Ho et al., 2003; Lord et al., 2000).
Traditionally, open and closed loop control techniques were investigated (Collis et al., 2004).
The latter have clear advantages thanks to their conditioning on state measurements.
We investigate and analyze a novel way to train closed loop controllers, namely via deep neural networks recurrently trained in a differentiable simulation environment with physicsbased losses.
This approach is motivated by the classical challenges of closed loop control for
Navier-Stokes environments: fluid flows are typically complex and chaotic.
Moreover, the number of degrees of freedom in numerical solvers often is too large, making heavily reduced representations necessary (Sipp & Schmid, 2016; Noack et al., 2004; Bergmann & Cordier, 2008; Proctor et al., 2016).
Instead, training with a differentiable simulator provides access to the full, unmodified physical environment, and provides reliable and diverse training feedback in the form of gradients.
More specifically, we investigate steering an actuated rigid body immersed in fluid systems with two way coupling, i.e. the rigid body influences the fluid around it and vice-versa.
We focus on objectives that require the rigid body to reach specific target configurations, i.e. center of mass location and orientation.
In this context, the differentiable simulations makes it possible to learn controllers without providing ground truth control signals.
An ubiquitous challenge for neural network approaches is generalization to conditions beyond the training distribution (Goodfellow et al., 2016).
We show that although training takes place in a quiescent flow condition, i.e. a fluid initially at rest, the networks trained via differentiable simulators are able to find control strategies that reliably handle more complex setups than those seen at training time.
Their control characteristics are dictated by a set of physically interpretable loss terms, making it possible to favor desired aspects of the control, e.g., the amount of overshoot, tracking speed or maximum control effort.
The performance of our networks is assessed in four different test scenarios with increasing levels of complexity.
We show their advantages over a range of base- line algorithms, from linear controllers such as PID and loop shaping (McFarlane & Glover, 1990;
Kwakernaak, 2002), to supervised and reinforcement learning algorithms (Haarnoja et al., 2018a).
R ELATED W ORK
Many recent works have been investigating different ways of coupling control and deep learning.
Since neural networks are good universal approximators (Hornik et al., 1989), many have investigated using them as a reduced order model of a complex dynamical system (Eivazi et al., 2020;
Hasegawa et al., 2020; Nair & Goza, 2020), which can then be used as an inexpensive solver for known closed loop control techniques such as model predictive control (Bieker et al., 2020; Morton et al., 2018; Chen et al., 2021).
Achieving linear-to-nonlinear mappings through learned Koopman operators has also been studied in recent works (Yeung et al., 2019; Li et al., 2020).
Another way of using deep learning for control purposes is through reinforcement learning (Verma et al., 2018; Paris et al., 2021;
Ren et al., 2021; Novati et al., 2019; Ma et al., 2018).
In this case a neural network typically receives a representation of the multi dimensional state describing the system at a given time, e.g. velocity probes and scalar variables, and outputs the control efforts.
This is achieved by training the network to maximize a reward function that describes a control objective.
In recent years, a variety of refined reinforcement learning variants were proposed (Schulman et al., 2015; Ho & Ermon, 2016; Schulman et al., 2017; Haarnoja et al., 2018b).
Recently, differentiable solvers for numerous fields, such as robotics (Toussaint et al., 2019) and biology (Ingraham et al., 2019), were constructed to take advantage of deep learning tools via automatic differentiation.
Since the gradients regarding a cost function are available, it is possible to directly find solve for approporiate control efforts of a given task.
This task could be placing a piece of cloth into a target container (Liang et al., 2019), pouring liquids (Schenck & Fox, 2018), moving a fluid to a specified region (Holl et al., 2020) or generating a specified velocity field from an immersed body (Takahashi et al., 2021).
To accomplish the control task, a full optimization needs to be performed for every timestep of a simulation to compute a suitable control signal.
However, this is typically much too slow for practical applications with real time requirements.
In this work we only use the gradients from the differentiable solver to train a network to act as a controller, which only relies on a sparse set of measurements from the environment and can be evaluated very efficiently.
M ETHODOLOGY
In physics and engineering the evolution of a physical system η ( x, t ) is often described by a partial differential equation (PDE) as [Equation]
[Equation]
where F models the physical behavior of the system and ω ( t, η ) represents variables that influence it such as boundary conditions.
If the system depends only on time as ξ ( t ) then equation 1 reduces to an ordinary differential equation (ODE) as
[Equation]
[Equation] Given a generic control policy ˆ P ( t | θ ) parametrized by θ , an external actuation can be inserted into a system described by equation 2 according to [Equation]
[Equation]
In this work, we target a coupled PDE-ODE system, interacting via boundary conditions ω and the control policy ˆ P .
Integrating equation 3 over time yields a state modified by the policy, ξ ( t, ˆ P ) , and the control task to reach ξ obj is given by the minimization problem
[Equation]
More specifically, we use the incompressible Navier Stokes equations, which is a form of equationwith n = 1 , that describes how a velocity field evolves given specified boundary conditions as the following
[Equation]
where u is the velocity field, p is the pressure, ρ is the density and ν =
ˆ u ˆ L Re is the kinematic viscosity, where Re is the Reynolds number and ˆ u and ˆ L are a reference velocity and length, respectively.
We additionally target rigid objects immersed in the fluid.
Their linear and angular movement can be described by equation 2 with n = 2 as
[Equation]
where x r is the body position, m the body mass, α the body angle
and I its moment of inertia.
The terms (cid:80) F and (cid:80) T denote the forces and torques that are acting upon the body, respectively.
When using both equation 6 and equation 7 the system has 3 degrees of freedom (DOF).
In a few cases below we will omit equation 7, yielding a 2 DOF scenario.
When dealing with a rigid body immersed in a fluid, these terms reduce to
[Equation]
where S is the body surface, ⃗n is the surface normal, ⃗r maps the surface location to the local coordinate system of the body, with the origin being the center of mass.
Dirichlet boundary conditions for the NS simulation are imposed on the velocity field via u = ∂x r /∂t at rigid body cells.
Together, equation 6 and equation 7 represent a coupled dynamical system subjected to nonlinear perturbations derived from the interaction between rigid body and fluid.
Control efforts after exerted via forces:
[Equation]
The specific control problem can then be formulated by finding the control efforts through [ F control , T control ]
T = P ( t | θ )
so that
[Equation]
where x obj and α obj are an objective position and angle, respectively.
Therefore the control task investigated can be summarized as controlling an ODE (rigid body movement equations) with highly nonlinear disturbances that emerge from a PDE (NS equations), which is also influenced by the ODE solution.
N EURAL N ETWORKS AS C ONTROL P OLICIES
We investigate how to use neural networks to represent a policy P ( z ( t ) | θ ) for the control task described by equation 12, where z ( t ) is a set of discrete state variables, which we denote with z for brevity, and θ denotes the weights of the network.
The network acting as a policy receives the current and previous n p states as input, and has the objective of infering appropriate control efforts for a given learning objective.
Each state consists of the spatial error e xy , angular error e α , rigid [Figure]
For clarity, we will refer to the latter simply as F and T .
Also ∂x r ∂t and ∂α∂t will be referred to as ˙ x and ˙ α , respectively.
e xy , ˙ x and F are expressed in the local reference frame of the rigid body.
We denote states at a discrete time t with a superscript t .
In this way, the input for the networks at a time t can be expressed as z = [ w t , w t − 1 , . . .
, w t − n p ] , where w t =
[ e txy , ˙ x t , F t , e tα , ˙ α t , T t ] T .
In the following, a range of different learning procedures are investigated: differentiable physics, supervised and reinforcement learning.
4. L EARNING VIA A D IFFERENTIABLE S OLVER
Our method employs a fully differentiable solver.
Hence, based on the following loss formulations, we can provide gradients to the neural network policy P diff ( z | θ ) about reactions of the physical system from policy actions and its temporal evolution.
This policy can be trained without the need to pre-compute potentially sub-optimal training data.
Rather, the network is left to discover the best possible policy over the course of the training.
Our loss formulation include a time horizon of l time steps as a central parameter.
The evaluations across this time interval leads to training signals that take into account how outputs of the policy network influence the future states of the environment.
Via the differentiable solver, the loss signals are recurrently backpropagated to the policy, allowing it to ”plan ahead”, e.g., to avoid overshooting.
This process is illustrated in figure 1.
Instead of pre-computing training data, we make use of a loss function that combines three objectives.
The objective term O typically dominates, and ensures that the body reaches the target state:
[Equation]
where the β are hyperparameters that weigh the different terms.
However, a loss function with this term alone results in a controller with tendencies of overshoot since it does not account for the rigid body velocity.
Hence we introduce a velocity term V :
[Equation]
Far away from the target, larger spatial and angular errors in the denominators lead to smaller values of V .
Closer to the target objective these errors approach zero [Equation] and hence V becomes an L2 norm of the linear and angular velocities.
As a consequence the optimization guides the policy to slow down the body near the target, thus reducing overshooting effects.
Following previous work (Bieker et al., 2020)
[Equation]
we additionally include a term to avoid large control efforts as well as abrupt changes
[Equation]
Finally the combined loss function for the differentiable solver training can be written as
[Equation]
Above, the β hyperparameters regulate the relative impact of each term for the controller.
Because of the direct physical impact of each hyperparameter, adjusting them is straight forward.
For example, spatial tracking can be more precise by increasing β xy or the angular tracking can be accelerated by decreasing β ˙ α .
However, overly large values can result in overshooting.
The hyperparameters for this work were chosen aiming for an overall balance of the objectives .
4. S UPERVISED L EARNING
As a baseline for learning, we include a fully supervised learning approach.
Due to a lack of optimal, ground-truth control policies, we construct a dataset in the following manner: we manually prescribe velocities to the rigid body so that it reaches an arbitrary target.
We then compute the fluid forces acting upon the body and calculate the control efforts to cancel them and yield the acceleration of the prescribed trajectory.
In this way, we obtain a set of states z with paired, expected control efforts [ ˆ F ( z ) , ˆ T ( z )] T .
Training with these precomputed forces can be performed in a fully supervised way with the loss
[Equation]
4. R EINFORCEMENT L EARNING
Additionally, we include a reinforcement learning algorithm that works without making use of the solver gradients.
We use the Soft Actor Critic (SAC) algorithm (Haarnoja et al., 2018b), a recent state-of-the-art approach.
SAC is a model-free variant that has the added benefit of being off-policy.
Thus, past experiences can be stored in a replay buffer and are not invalidated by policy updates.
As the simulation process is computationally very expensive, this increase in sample efficiency is highly beneficial.
The actor, which is also represented by a neural network, has the objective of maximizing the predicted Q values.
Additionally, SAC introduces an entropy term into the objective of the policy as regularization.
It discourages overly confident decisions while also controlling the trade-off between exploration and exploitation inside the action space.
In our setting, the actor represents a control policy, and hence we refer to it as P RL ( z | θ ) .
Following equation 12, the reward at a time t is calculated using the squared distance objective
[Equation]
with g ( e txy ) yielding a constant G if || e txy || ≤ ϵ , and zero otherwise.
We normalize r t by the starting error e 0 from the beginning of a sequence in order to reduce the variance between different trajectories and make the reward independent from the initial state.
We present in this paper preliminary results and other reward functions are still being investigated.
E XPERIMENTS
We perform a series of experiments with increasing degrees of complexity to assess the generalization capabilities of the considered approaches.
Below we explain the default parameters which are applicable unless noted otherwise.
Details and deviating parameters for all experiments are provided in the appendix. Source code used in this work can be found on https: //github.com/brenerrr/PhiFlow/tree/two_way_coupling .
Datasets and Test Scenarios
The training data consists of simulations with only one objective and parameters Ba2 or Ba3 from Table 1 for systems with 2 and 3 DOF, respectively, using a quiescent flow ( u = 0 ) as initial conditions.
Since the fluid is initially at rest, all the fluid perturbations are created from the rigid body movement.
The validation datasets for Ba2 and Ba3 consist ofsimulations with the same parameters as training but different objectives.
To evaluate generalization of 2 DOF networks, we employ Buo2 , which increases the difficulty by introducing a lighter fluid source that disrupts the flow through buoyancy.
For 3 DOF networks trained on Ba3 , we use the [Table] scenarios In3 , InBuo3 and InBuoAg3 .
For In3 , the fluid is less viscous (higher Re ) and ∆ t is smaller.
Correspondingly, the controllers are sampled once every two timesteps.
Setup InBuo introduces buoyancy forces as well an inflow while a second rotating obstacle is additionally present in InBuoAg3 .
These test environments were designed to deviate more and more strongly from the quiescent training conditions.
Unless stated otherwise, the test datasets are comprised ofsimulations with different trajectories, created by changing the objectives after ∆ t = 100 .
The supervised learning approach uses a separate, pre-computed dataset for training.
It is comprised of 100 simulations from which 80 are used for training and 20 for validation and each simulation has 500 time steps.
Neural Network Representation and Training The underlying network architecture for all of them is kept constant: two dense layers with ReLU activation, followed by a third dense layer.
For the latter, the networks trained via differentiable physics and reinforcement learning have a hyperbolic tangent activation to ensure that the control efforts are bounded.
As we achieved a better performance for the supervised case without the activation of the last layer, it was omitted there.
Training the differentiable physics networks starts with a simulation with quiescent initial condition, gathering objectives from an uniform random distribution.
After advancing the simulation so that n p past states exist, the network training is activated and its outputs are used as control efforts.
Once l = 16 time steps are available, we compute a loss and update θ .
A discussion about how we choose l can be found in section C.1.3.
After 1000 simulation steps we restart the simulation until n i training iterations are performed.
We choose n i = 1000 and n i = 5000 when training a network for 2 and 3 DOF systems, respectively.
For supervised training, in total n i = 150 ,and n i = 200 , 000 training iterations are performed for the network used for systems with 2 and
DOF, respectively.
The reinforcement learning approach uses the same simulation environment as the differentiable training.
After each simulated time step a batch of 128 samples is drawn from the replay buffer and used for training.
In total, n i = 300 , 000 training iterations are performed.
Evaluation Metrics As error metrics for comparing results we primarily use absolute errors in position and orientation, ||
e xy ||
and || e α || .
We also compute an average steady state error ∥
e ∥ ss that can assess the steady-state performance of the controllers without their initial transient phases.
It is calculated using the last 3 / 4 of the time interval in which an objective was being tracked.
For bar plots of ∥ e ∥ ss we also display one standard deviation for illustrating the spread of the error values.
R ESULTS
We first perform a broad comparison of the different algorithms, before moving on to more complex cases and generalization tasks.
6. A LGORITHMIC C OMPARISON
Baseline Algorithms Two types of linear controllers are tested as a reference to help assess the performance of the learned versions: a classic proportional-integrator-derivative (PID) P PID and a loop shaping controller P LS designed through a blend of mixed-sensitivity design and the GloverMcFarlane method.
All
2 DOF Validation First, we consider a validation with the setup Ba2 (same simulation parameters as training but with different targets).
The goal is to make sure all the controllers are functioning correctly for the conditions seen at training time.
All the approaches considered are able to achieve a low error given sufficient time, which confirms they work as intended in conditions they were designed to operate in.
Details can be found in the appendix.
Increased Difficulty Next we perform a test with the setup Buo2 , which includes perturbations from a lighter fluid and a longer time window.
The interaction between buoyancy and rigid body creates oscillatory flow structures and the control task becomes more challenging.
As a consequence, ∥ e xy ∥ ss is much larger for all approaches except for P diff .
The latter exhibits an error more than three times smaller than the second best (PID), as shown in figure 2.
The trajectories clearly show the amount of undesirable oscillations and overshooting present in the P LS , P sup and P PID .
Both P LS and
P RL exhibit a poor performance, with the latter still being preliminary results and not able to finish all test simulations.
Hence, we will omit these two methods for further tests.
6. I NCREASED C OMPLEXITY AND G ENERALIZATION
We now evaluate the performance of the remaining methods for a rectangular body with orientation (3 DOF).
The additional degree of freedom adds a significant amount of complexity to the control task, and we use a set of more complex test cases to evaluate generalization.
In line with the
DOF case, we first perform an evaluation with validation cases, i.e. from Ba3 .
All three remaining methods fare well for these environments, and yield stable controllers (details are provided in the appendix).
Generalization Tests Modified Inflow Conditions.
Next, a test with the In3 setup is performed to assess the generalization capabilities of the considered controllers.
Here the environments are changed substantially by introducing an inflow from the left side of the domain and utilizing a smaller viscosity (higher Re ).
The inflow together with the rigid body movement creates unsteady flow structures that vary depending on the box angle.
P diff is able to maintain a smaller ∥ e xy ∥ ss compared to the other controllers as can be seen in figure 3.
Very noticeable oscillations around the objective locations are present in the trajectory of P sup and P PID .
Instead, P diff produces a stable orientation, showing that this policy successfully counteracts the perturbations caused by the strongly varying flow.
Inflow and Buoyancy.
Next, we introduce a source of lighter fluid that rises due to buoyancy at the bottom of the domain with parameters InBuo3 .
This test has the goal of assessing the robustness
Buoyancy worsens the performance of controllers but P diff still maintains low steady state errors
Although the flow is fairly chaotic, P diff is able to maintain low steady state errors and good tracking.
of the controllers further, since the fluid source tends to create higher frequency oscillations, which makes the control task harder. While the P sup and P PID have a considerable worsening in their performance with higher steady state errors, they also display undesirable oscillations.
On the other hand, P diff maintains low values for ∥ e xy ∥ ss and ∥ e α ∥ ss while rejecting most of the perturbations as shown in figure 4.
Agitated Inflow with Buoyancy.
The last test introduces a second rigid body which spins close to the left boundary, together with a higher Re = 5000 .
These characteristics contribute to a more chaotic flow, with stronger high frequency perturbations resulting in a very challenging test scenario.
Especially when considering the spatial error, it is clear that P diff is able to stabilize most perturbations and maintain a low steady state error as shown in figure 5.
The approach can maintain a stable path despite the changed flow conditions, while the other approaches fail to do so.
6. D ISCUSSION
Taken together, the previous set of tests show the advantages of the proposed training via differentiable simulations.
It is able to provide a neural network with feedback to control a dynamical system subjected to nonlinear perturbations for long periods of time, even when training took place in a different, simplified environment.
P diff is able to handle new situations without the strongly deteriorated tracking performance of the other controllers, with our tests indicating that this is a consequence of the differentiable solver, which provides a varied learning signal and results in a neural network that robustly handles a large variety of conditions.
It is also appealing to be able to use the original set of equations of a system, even if they are as nonlinear as the NS equations because reduced representations often do not portrait well nonlinear behaviors.
Instead, the differentiable simulation approach allows training the controller using the full set of model equations, such that it can adapt to the subtleties of the environment.
C ONCLUSIONS
We have studied the use of differentiable simulations to train neural networks acting as controllers for complex dynamical systems.
The considered system describes the movement of a rigid body subjected to nonlinear perturbations derived from the Navier Stokes equations, posing a very challenging control task.
The proposed approach, which introduced a set of physically interpretable loss terms, is able to train robust controllers without the need to provide reference data for training.
It is able to produce controllers that generalizes very well to substantially different, challenging flow conditions.
Numerous interesting venues for future research exist based on our results, such as exploring the transfer of synthetically trained controllers to real-world environments.
A CKNOWLEDGMENTS
This work was supported by the ERC Consolidator Grant SpaTe (CoG-2019-863850).
A L INEAR C ONTROLLERS A. PID
The P gain was adjusted so that the maximum control effort from the PID controller is in the same order of magnitude as the one from the network controllers.
The D gain is then tuned to be as low as possible while still avoiding overshoot and the same is done for the I gain.
The gains for the controllers for each setup are displayed in Table 2 and the control effort U is obtained by
[Equation]
[Table]
A. L OOP S HAPING
Loop Shaping design consists of finding a controller K so that the open loop response K ∗ P , where P is the system plant.
It should be as close as possible to the open loop response of a transfer function P ′ , which is chosen by the user.
A common choice for P ′ is P ′ ( s )
=
ω
b s where ω b is the control bandwidth and s is a complex frequency.
This function has the property of having high gains for low frequencies and low gains for high frequencies.
In other words, an input signal with frequency higher than w b (noise) will be dampened and an input signal with frequency lower than w b (perturbations) will be amplified.
We use a combination of two loop-shaping methods: mixed-sensitivity-design, which favors performance, and the Glover-McFarlane method, which favors robustness to plant uncertainty, as implemented by the Matlab loopsyn() function.
It features a parameter α :
For α = 0 the controller has the best performance and while α = 1 favors robustness.
From a range of experiments with our physical environment, we choose α = 0 .
95 and w b = 0 .
After converting the found controller from the Laplace domain to the discrete one, we obtain the coefficients shown in Table 3 and the control effort U is calculated according to
[Equation]
DL
ReLU
DL
DL ReLU tanh t t-np texyx ̇ e αα̇
FT exyx ̇ e αα̇
FT ... exyx ̇ e αα̇
FT t 𝒫 where i is the index of the current time step, e is the error and n and d are the controller coefficients. [Table]
B N ETWORKS A RCHITECTURE D ETAILS
The dimensions of the considered neural network layers are given in Table 4, and its architecture is visualized in figure 6.
The network of the 2 DOF setup has a total of 2206 trainable parameters while the one used in the 3 DOF setup has 2243 trainable parameters.
[Table]
C T RAINING D ETAILS C. D IFFERENTIABLE P HYSICS
The network training parameters for all setups investigated with differentiable physics can be found in Table 5.
A learning rate of 0.01 was used in our tests.
Additionally, learning rate decay is used so that the learning rate drops to half of its initial value every 100 and 1000 training iterations for the 2 DOF and 3 DOF setups, respectively.
The networks inputs and outputs are normalized based on a set of measured simulation statistics.
C.1. E FFECT OF T IME H ORIZON L
The effect of training with different time horizons l is also investigated.
A test consisting of one simulation with parameters InBuoAg3 and a N shaped trajectory is conducted and the performance of networks trained with different values of l can be seen in figure 7.
Small values of l mean that loss and backpropagation are performed for short physical timespans during training, i.e. a small temporal lookahead, which produces a controller with a deteriorated performance.
In addition, performance does not change significantly beyond l = 16 .
Since the same number of iterations were conducted for all runs, using larger time horizons only increases training time.
Therefore, we choose l = 16 since it provides a good balance between performance and training time.
[Table]
Training with small time horizons l produces poor performant controllers.
When using a very large time horizon, such as l = 32 , the gains are not worth the doubled amount of training time.
In order to evaluate how the performance from the network trained with differentiable physics is influenced by the networks weights initial values, we perform three training runs with different initial seeds.
We run a test simulation with parameters from InBuoAg3 and objectives that describe a N shape trajectory.
Despite minor differences the error norms for all seeds have a similar tracking performance.
This is especially apparent when comparing it to other controllers, as shown in figure 8.
C.1. A BLATION S TUDIES
We present an evaluation study with InBuo3 to assess the influence of the terms introduced in section 4.1.
One simulation is performed for each combination of loss terms.
When using only the O term, the training is severely under constrained, and as a consequence the network learns to exert overly large forces.
This destabilizes the simulations preventing a successful training.
Therefore we compare the performance of networks trained with the combinations: OVE (all terms), OV (objective and velocity terms) and OE (objective and effort terms).
Networks trained with OVE and OV exhibit similar error curves and trajectories as shown in figure 9.
The network trained with OE is also able to achieve low error values but it exhibits an uneven decay, which is an undesirable behavior.
When analyzing the control efforts it can be seen that OE provides a network that produces maximum control efforts more often, which means it uses more energy.
When considering OV , there are no [Figure] constraints on control efforts and consequently it learns a control policy that modulates the control efforts with a high frequency - an undesirable behavior in practice
[Figure]
C. S UPERVISED L EARNING
A learning rate of 0.01 is used with learning rate decay.
Every 15 , 000 iterations it drops to half of its size.
Training with more data is examined for assessing if the performance of the controller acquired via supervised learning could be improved.
A dataset with 200 simulations (double the size of the original one) is generated with 180 simulations being used for training and 20 for validation.
Then a test simulation with parameters from InBuoAg3 and objectives describing a N shaped trajectory is conducted.
It can be seen that adding more data did not result in clear improvements, as shown in figure 10.
C. R EINFORCEMENT L EARNING
We use the SAC implementation from stable-baselines3 (Raffin et al., 2021).
We use a fixed learning rate of 0.0003 and a reward discount factor γ of 0.99.
The parameter τ controlling the Polyak
Averaging of the two Q-Functions within the critic is set to 0.05.
C. H ARDWARE AND S OFTWARE
All optimization procedures were conducted utilizing the PyTorch framework (Paszke et al., 2019) on a GeForce RTX 2080 Ti.
Approximate training times are displayed in Table 6.
[Table]
D A DDITIONAL R ESULTS
In the following we present a collection of additional results that were not shown in the main body of the paper.
For all simulations, the cylinder has m = 11 .
78 and radius ˆ r = 5 while the box has m = 36 , I = 4000 , width w = 20 and height h = 6 .
[Table]
[Table]
x   y   x   y
Simulation −
[Figure] [Figure]