N EURAL F IELD D ISCOVERY D ISENTANGLES E QUIVARIANCE IN I NTERACTING D YNAMICAL S YSTEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
Systems of interacting objects often evolve under the inﬂuence of underlying ﬁeld effects that govern their dynamics, e.g. electromagnetic ﬁelds in physics, or map topologies and trafﬁc rules in trafﬁc scenes.
While the interactions between objects depend on local information, the underlying ﬁelds depend on global states.
Pedestrians and vehicles in trafﬁc scenes, for example, follow different trafﬁc rules and social norms depending on their absolute geolocation.
The entanglement of global and local effects makes recently popularized equivariant networks inapplicable, since they fail to capture global information.
To address this, in this work, we propose to disentangle local object interactions –which are equivariant to global roto-translations and depend on relative positions and orientations– from external global ﬁeld effects –which depend on absolute positions and orientations.
We theorize the presence of latent ﬁelds, which we aim to discover without directly observing them, but infer them instead from the dynamics alone.
We propose neural ﬁelds to learn the latent ﬁelds, and model the interactions with equivariant graph networks operating in local coordinate frames.
We combine the two components in a graph network that transforms ﬁeld effects in local frames and operates solely there.
Our experiments show that we can accurately discover the underlying ﬁelds in charged particles settings, trafﬁc scenes, and gravitational n-body problems, and effectively use them to learn the system and forecast future trajectories.
I NTRODUCTION
[Figure]
Systems of interacting objects are omnipresent in nature, with examples ranging from the subatomic to the astronomical scale –including colliding particles and n-body systems of celestial objects– as well as human-centric settings like trafﬁc scenes, governed by social dynamics.
The majority of these systems does not evolve in a vacuum, they instead evolve under the inﬂuences of underlying ﬁelds.
For example, electromagnetic ﬁelds may govern the dynamics of charged particles.
In trafﬁc scenes, the road network and trafﬁc rules govern the actions of trafﬁc scene participants.
N-body systems might swirl around supermassive black holes that create gravitational ﬁelds.
Earlier work on learning interacting systems proposed graph networks (Kipf et al., 2018;
Battaglia et al., 2016; Sanchez-Gonzalez et al., 2020), while state-of-the-art methods for interacting systems propose equivariant graph networks (Walters et al., 2021;
Satorras et al., 2021; Koﬁnas et al., 2021; Brandstetter et al., 2022) to model dynamics while respecting their underlying symmetries.
These networks exhibit increased robustness and performance, while maintaining parameter efﬁciency due to weight sharing.
They are, however, not compatible with underlying ﬁeld effects, since they can only capture local states, such as relative positions, while ﬁelds depend on absolute states (positions or orientations).
In other words, global ﬁelds violate the strict equivariance hypothesis .
We are, thus, in need of an augmented notion of equivariance that encapsulates both local and global effects.
In many real-world settings, strict SE(3) equivariance, –equivariance to the special Euclidean group of translations and rotations– does not hold.
A function f that predicts trajectories in interacting systems is equivariant to translations if f ( x ) + τ = f ( x + τ ) for a translation vector τ , and equivariant to rotations if Q f ( x ) =
f ( Qx ) for a rotation matrix Q .
That is, even if the symmetries exist in a particular setting, they only manifest themselves in local interactions, yet they are entangled with global effects that stem from absolute states.
For instance, pedestrians and vehicles in trafﬁc scenes operate within egocentric perspectives; individual local coordinate frames allow for differentiating objects based on relative states.
Objects, however, behave differently depending on their absolute geolocation, e.g. people in different countries or cities follow different trafﬁc rules and social norms, and exhibit different driving habits.
In such cases, global transformations are not even properly deﬁned; it would be illogical to perform a translation or a rotation within the global coordinate frame of the Earth, since this would coincide with a different location on the globe.
Even within a local region of space, though, certain effects may only depend on global object states.
N-body systems from physics, for example, exhibit roto-translational symmetries, since gravitational forces only depend on relative positions.
Dynamics, however, may be inﬂuenced by external gravitational ﬁelds, which are either unknown or not subject to transformations.
Thus, strict equivariance is violated, since equivariant object interactions are entangled with global ﬁeld effects.
In this work, we disentangle the interactions between objects –which are equivariant to global rototranslations and depend on relative positions and orientations– from external ﬁeld effects –which depend on absolute positions and orientations.
Furthermore, we propose neural ﬁelds to model the underlying ﬁeld effects.
Neural ﬁelds depend on absolute positions, and potentially orientations, and predict latent force ﬁelds .
We model discrete object interactions with equivariant local coordinate frame graph networks, and continuous ﬁeld effects with neural ﬁelds.
We make the following contributions.
First, we introduce the notion of entangled equivariance that intertwines global and local effects, and propose a novel architecture that disentangles equivariant local object interactions from global ﬁeld effects.
Second, we introduce neural ﬁelds to discover global latent ﬁelds in interacting dynamical systems, and infer them by observing the dynamics alone.
Third, we propose an approximately equivariant graph network that extends local coordinate frame graph networks by introducing an auxiliary origin node, resulting in a mixture of global and local coordinate frames.
Finally, we conduct experiments on a number of ﬁeld settings, and observe that explicitly modelling ﬁelds is mandatory for effective future forecasting, while their unsupervised discovery opens a window for model explainability.
We term our method Aether , inspired by the postulated medium that permeates all throughout space and allows for the propagation of light.
B ACKGROUND
In this section, we introduce background knowledge on interacting systems, local coordinate frame graph networks, and neural ﬁelds, which will serve as a foundation for our method.
2. I NTERACTING DYNAMICAL SYSTEMS
An interacting dynamical system comprises trajectories of N objects, recorded for T timesteps.
The snapshot of the i -th object at timestep t describes the state x ti =
[ p ti , u ti ] , i ∈ { 1 , . . .
, N } , t ∈ { 1 , . . .
, T } , where p denotes the position and u denotes the velocity, using [ · , · ] to denote vector concatenation along the feature dimension.
Interacting dynamical systems can be naturally formalized as spatio-temporal geometric graphs (Battaglia et al., 2016; Kipf et al., 2018; Graber & Schwing, 2020), G = {G t } Tt =1 , with graph snapshots G t =
( V t , E t ) at different time steps.
The set of graph nodes V t = { v t 1 , . .
. , v tN } describes the objects in the system; v ti corresponds to x ti .
The set of edges
E t ⊆ (cid:8)(cid:0) v tj , v ti (cid:1) | (cid:0) v tj , v ti (cid:1) ∈
V t × V t (cid:9) describes pair-wise object interactions; (cid:0) v tj , v ti (cid:1) corresponds to an interaction from node j to node i .
Finally, we denote the graph neighbors of node v i with N ( i ) .
2. L OCAL COORDINATE FRAME GRAPH NETWORKS
Local coordinate frame graph networks have been popularized in recent years (Koﬁnas et al., 2021;
Luo et al., 2022) as a method to achieve SE(3) equivariance, due to their low computational overhead and high performance.
Koﬁnas et al.
(2021) proposed LoCS and introduced local coordinate frames for all node-objects at all timesteps.
They deﬁne augmented node states v ti =
[ p ti , ω ti , u ti ] , where ω ti denotes the angular position of node i at timestep t .
Each local coordinate frame is translated to match the target object’s position and rotated to match its orientation.
Considering the representation of node j in the local coordinate frame of node i , they ﬁrst compute the relative positions r tj,i = p tj − p ti
and then they rotate the state using the matrix representation of the angular position Q ( ω ti ) :
[Equation]
where ˜R ( ω ti ) =
Q ( ω ti ) ⊕ Q ( ω ti ) ⊕ Q ( ω ti ) , and ⊕ denotes a direct sum.
LoCS then proposes a graph neural network (Scarselli et al., 2008; Li et al., 2016;
Gilmer et al., 2017) that uses local states:
[Equation]
where f v , f e , and g v are MLPs.
The output of this graph network comprises differences in positions and velocities from the previous time step, in the local frame of each object.
Since these outputs are invariant, LoCS performs an inverse transformation to convert them back to the global coordinate frame and achieve equivariance, x t +1 i =
[Equation]
ti
[Equation] [Equation]
t +1 i | i , where R ( ω ti ) =
Q ( ω ti ) ⊕ Q ( ω ti ) .
2. N EURAL FIELDS
Finally, we make a brief introduction to neural ﬁelds.
Neural ﬁelds, or coordinate-based MLPs, are a class of neural networks that parameterize ﬁelds using neural networks (Xie et al., 2022).
They take as input spatial or temporal coordinates and predict some quantity.
Neural ﬁelds can learn prior behaviors and generalize to new ﬁelds via conditioning on a latent variable z that encodes the properties of a speciﬁc ﬁeld.
Perez et al.
proposed Feature-wise Linear Modulation (FiLM), a conditioning mechanism that modulates a signal.
It comprises two sub-networks α, β that perform multiplicative and additive modulation to the input signal, and can be described by
FiLM ( h , z ) =
α ( z ) (cid:12) h + β ( z ) , where z is the conditioning latent variable, h is the signal to be modulated, and α, β are MLPs that scale the signal, and add a bias term, respectively.
M ETHOD
In this section, we present our method, termed Aether .
First, we describe the notion of entangled equivariance, and introduce our architecture that disentangles global ﬁeld effects from local object interactions.
Then, we continue with the description of the neural ﬁeld that infers latent ﬁelds by observing the dynamics alone.
Finally, we formulate approximately equivariant global-local coordinate frame graph networks.
We note that throughout this work, we focus on 2-dimensional settings, and ﬁelds that are unaffected by the observable objects and their interactions thereof.
3. A ETHER
Interacting dynamical systems rarely evolve in a vacuum, rather they evolve under the inﬂuence of external ﬁeld effects.
While object interactions depend on local information, the underlying ﬁelds depend on global states.
On the one hand, locality in object interactions stems from the fact that dynamics obey a number of symmetries.
By extension, object interactions are equivariant to a particular group of transformations.
On the other hand, ﬁeld effects are non-local; they depend on absolute object states.
Thus, strict equivariance is violated, since equivariant object interactions are entangled with global ﬁeld effects.
We refer to this phenomenon as entangled equivariance .
As an example, in ﬁg. 2 we observe a system of 2 objects that evolve under the inﬂuence of a gravitational ﬁeld.
The arrows positioned on the objects represent the forces exerted on them.
One constituent of the net force is caused by the object interactions, and is thus equivariant, while the other can be attributed to the gravitational pull.
However, we can only observe the net force exerted at each particle, i.e. the sum of equivariant particle forces and non-equivariant force ﬁeld effects.
Hence, in this system, we say that equivariance is entangled .
[Figure]
We now propose our architecture that disentangles local object interactions from global ﬁeld effects.
We model the object interactions with local coordinate frame graph networks (Koﬁnas et al., 2021), and ﬁeld effects with neural ﬁelds.
During training, and given a multitude of input systems, neural ﬁelds will, in principle, be able to isolate global from local effects, since only global effects are recurring phenomena.
We hypothesize that ﬁeld effects can be attributed to force ﬁelds, and therefore, our neural ﬁelds learn to discover latent force ﬁelds .
The pipeline of our method is shown in ﬁg. 3.
Our inputs comprise states x ti for trajectories of N objects for T timesteps.
Since neural ﬁelds model global ﬁeld effects, they depend on absolute states.
Thus, we feed positions p ti , and potentially orientations ω ti , of the trajectories as input to a neural ﬁeld that predicts latent forces f ti = f
( p ti , ω ti ) .
The predicted ﬁeld forces can be now considered part of the node states, and further, they can be treated similarly to other state variables like velocities; as vectors, forces are translation invariant and rotation equivariant.
Thus, moving onward, we can treat the problem setup as if we were once again back in the strict equivariance regime.
We append the predicted forces f ti for each node-object i and each timestep t to the node states, and transform them to corresponding local coordinate frames, similarly to eq.
Namely, the force exerted on node j , expressed in the local coordinate frame of node i is computed as:
[Equation]
We feed the new local node states to a local coordinate frame graph network as follows:
[Equation]
where R ( ω ti ) =
Q ( ω ti ) ⊕ Q ( ω ti ) .
The equations above are similar to eqs.
(2) and (3), with the addition of the highlighted parts that denote the predicted forces expressed at local coordinate frames.
In practice, we closely follow (Kipf et al., 2018; Graber & Schwing, 2020; Koﬁnas et al., 2021) and formulate our model as a variational autoencoder (Kingma & Welling, 2014; Rezende et al., 2014) with latent edge types.
The exact details are presented in appendix A.2.2.
3. F IELD DISCOVERY
In many settings, ﬁelds might not be directly observable for us to probe them at will and use them for supervision.
For example, astronomical observations of solar systems and galaxies might not include black holes, yet we can observe their effects.
Moreover, ﬁelds are often not even measurable or quantiﬁable, or they are not explicitly deﬁned.
For instance, social “ﬁelds” that guide trafﬁc, cannot be measured or deﬁned explicitly, but we can safely assume they exist.
Motivated by these observations, we design an architecture that performs unsupervised ﬁeld discovery , while solving the surrogate supervised task of trajectory forecasting.
In this work, we aim to discover two different types of ﬁelds, which we term “static” and “dynamic” ﬁelds.
Static ﬁelds refer to settings in which we have a single ﬁeld shared throughout the whole dataset.
On the other hand, dynamic ﬁelds refer to settings in which we have a different ﬁeld for each input system, and ﬁelds also differ between train, validation, and test sets.
We now describe neural ﬁelds, used in this work, to model the underlying ﬁeld effects.
Neural ﬁelds depend on absolute positions, and potentially orientations, and predict latent force ﬁelds.
When dealing with static ﬁelds, we use an unconditional neural ﬁeld, i.e. a neural ﬁeld that is a function only of the input positions-orientations, as the ﬁeld values are common across data samples.
In contrast, for dynamic ﬁelds, we use a conditional neural ﬁeld, i.e. a neural ﬁeld that also depends on a latent vector z ∈ R D z that represents the underlying ﬁeld.
The latent z will be inferred from the input trajectories and can be thought of as representing unusual non-equivariant dynamics.
We use z to explicitly condition the neural ﬁeld, and thus, its general form is f : R 2 × S 1 × R D z
→ R 2 .
Static ﬁelds We start with the description of unconditional neural ﬁelds used in static ﬁeld settings, since conditional neural ﬁelds share the same backbone.
First, we encode the input positions using
Gaussian random Fourier features (Gaussian RFF)
(Tancik et al., 2020).
They are deﬁned as:
[Equation]
where p ∈ R D in are the input coordinates, and
B ∈ R D c
× D in is a matrix with entries sampled from a Gaussian distribution, B kl ∼ N (0 , σ 2 ) .
The variance σ 2 can be chosen per task with a hyperparameter sweep.
We use a unit vector representation for the orientations θ , θ =
[cos θ, sin θ ] (cid:62) .
Following Koﬁnas et al.
(2021), we use the angles of the velocity vectors as a proxy for the orientations.
Then, we use a linear layer to encode the orientation vectors, δ ( θ ) =
W ω θ .
We ﬁnally concatenate the encoded positions and orientations in a single vector that is being fed as input to the neural ﬁeld.
After encoding the input coordinates, we use a 3-layer MLP with SiLU (Ramachandran et al., 2018) activations in-between, that outputs a latent force ﬁeld, f ( p , θ ) =
MLP([ γ ( p ) , δ ( θ )]) .
Dynamic ﬁelds The neural ﬁelds used to model the dynamic ﬁelds are conditioned on a latent vector representation z ∈ R D z that describes prior knowledge about the underlying ﬁeld, and are deﬁned as f ( p , θ | z ) .
In our case, the latent representation should “summarize” the input graph such that it isolates only global effects from the ﬁeld.
To that end, we employ a simple global spatio-temporal attention mechanism, similar to Li et al.
(2016), that aggregates the input system in a latent vector representation.
First, we deﬁne object embeddings o i =
GRU (cid:0) W g x 1: T i (cid:1) , where W g is a matrix used to linearly transform the inputs, and GRU is the Gated Recurrent Unit (Cho et al., 2014). We also deﬁne temporal embeddings t = PE( t ) , where PE are positional encodings (Vaswani et al., 2017).
The aggregation is then deﬁned as follows:
[Equation]
where f a : R D s → R , f b : R D s → R D z are 2-layer MLPs with SiLU activations Ramachandran et al.
(2018) in between.
After having obtained a latent vector representation z that summarizes the input system, we condition the neural ﬁeld using FiLM (Perez et al., 2018).
We include FiLM layers after the ﬁrst two linear layers of the neural ﬁeld.
The exact details are presented in appendix A.2.1.
3. A PPROXIMATE EQUIVARIANCE WITH GLOBAL - LOCAL COORDINATE FRAMES
Equivariant neural networks cannot capture non-local information, such as global ﬁeld effects.
In this work, we explicitly aim to discover these ﬁelds and disentangle them from local object interactions.
An alternative, or rather complementary approach, would be to directly combine global and local information, following the recently proposed notion of approximate equivariance (Wang et al., 2022).
Starting from LoCS (Koﬁnas et al., 2021), we can combine global information and still operate in local coordinate frames by deﬁning an auxiliary node-object corresponding to the global coordinate frame, i.e. an object positioned at the origin, and oriented to match the x-axis.
Similar to all objects in the system, the full state of the origin node
O comprises the concatenation of its position and velocity, x O =
[ p O , u O ] =
(0 , 0 , 1 , 0) (cid:62) .
We temporarily add this node in the geometric graph, and we further connect it with edges to all other objects.
Thus, the new vertex set is
V (cid:48)
=
V ∪ {O} , and the new edge set is E (cid:48)
=
E ∪ { e O ,i ∀ i ∈ V} .
The origin state can be expressed in local coordinate frames using similar transformations to eq.
The origin transformed in the local coordinate frame of the i -th object is computed as follows:
[Equation]
Since graph networks are permutation equivariant, we have to distinguish between this auxiliary origin node and the other nodes, otherwise, the origin would be treated as an actual object, and there is no way for the network to understand which the origin is.
Hence, we augment each object’s state with the origin node information expressed in local coordinate frames, extending eqs.
(2) and (3) to
[Equation]
This approach allows us to remove the origin node from the actual graph, and push the information in the node states.
We term this method G-LoCS ( G lobal-
Lo cal C oordinate Frame S ).
In practice, similar to Aether, we formulate our model as a variational autoencoder (Kingma & Welling, 2014;
Rezende et al., 2014) with latent edge types.
The full details are presented in appendix A.2.3.
Finally, in practice, we integrate G-LoCS in Aether, since it can enhance the performance of our method.
R ELATED WORK
Equivariant graph networks The seminal works of (Cohen & Welling, 2016; 2017; Worrall et al., 2017) introduced equivariant convolutional neural networks and demonstrated effectiveness, robustness, and increased parameter efﬁciency.
Recently, many works have proposed equivariant graph networks (Schütt et al., 2017; Thomas et al., 2018; Fuchs et al., 2020;
Walters et al., 2021;
Satorras et al., 2021; Koﬁnas et al., 2021; Brandstetter et al., 2022; Luo et al., 2022).
Walters et al.
(2021) propose rotationally equivariant continuous convolutions for trajectory prediction.
Satorras et al.
(2021) propose a computationally efﬁcient equivariant graph network that leverages invariant euclidean distances between node pairs.
Koﬁnas et al.
(2021) introduce roto-translated local coordinate frames for all objects in an interacting system and propose equivariant local coordinate frame graph networks.
Brandstetter et al.
(2022) generalize equivariant graph networks using steerable
MLPs (Thomas et al., 2018) and incorporate geometric and physical information in message passing.
Equivariant graph networks differ from our work since they cannot capture non-local information, while our work disentangles equivariant local interactions from global effects and captures them both.
Approximate equivariance
Recently, a number of works has proposed to shift away from strict equivariance, in what (Wang et al., 2022) termed as approximate equivariance .
Wang et al.
(2022) propose approximately equivariant networks for dynamical systems, by relaxing equivariance constraints in group convolutions and steerable convolutions.
van der Ouderaa et al.
(2022) propose to relax strict equivariance by interpolating between equivariant and non-equivariant operations, using non-stationary kernels that also depend on the absolute input group element.
Romero & Lohit (2021) propose Partial G-CNNs that learn layer-wise partial equivariances from data.
We note that even though these works and approximate equivariance share similarities with our work, our notion of disentangled equivariance is conceptually different.
That is because related work uses the term approximate equivariance to denote that equivariance is “broken” due to noise or imperfections, while our work disentangles the system dynamics that are actually equivariant, from the global ﬁeld effects that are not, and in fact, might be unaffected by such transformations.
Neural ﬁelds Neural ﬁelds have recently exploded in popularity in 3D computer vision, popularized by NeRF (Mildenhall et al., 2021).
Since MLPs are universal function approximators (Hornik et al., 1989), neural ﬁelds parameterized by MLPs can, in principle, encode continuous signals at arbitrary resolution.
However, neural networks can suffer from “spectral bias” (Rahaman et al., 2019; Basri et al., 2020), i.e. they are biased to ﬁt functions with low spatial frequency.
To address this issue, a number of solutions have been proposed.
Tancik et al.
(2020) leverage Neural Tangent Kernel (NTK) theory and propose Random Fourier Features (RFF), showing that they can overcome the spectral bias.
They also show that RFF are a generalization of positional encodings, popularized in recent years in natural language processing by Transformers (Vaswani et al., 2017).
Concurrently, Sitzmann et al.
(2020) proposed SIREN, neural networks with sinusoidal activation functions.
While neural ﬁelds have been used extensively in computer vision problems including 3D scene reconstruction (Park et al., 2019; Mescheder et al., 2019) and differentiable rendering (Sitzmann et al., 2019; Mildenhall et al., 2021), they have not seen wide usage in dynamical systems.
Notably, Raissi et al.
(2019) proposed Physics-Informed Neural Networks (PINNs), neural PDE solvers based on neural ﬁelds.
E XPERIMENTS
We evaluate our proposed method, Aether , on settings that include static as well as dynamic ﬁelds.
First, we explore charged particles (Kipf et al., 2018) that evolve under the effect of a static electrostatic ﬁeld.
Then, we evaluate our method on a subset of inD (Bock et al., 2020) that contains a single location, and thus a static ﬁeld as well.
Finally, we explore gravitational n-body problems (Brandstetter et al., 2022) with dynamic ﬁelds.
We compare our method against dNRI (Graber & Schwing, 2020) and LoCS (Koﬁnas et al., 2021).
DNRI (Graber & Schwing, 2020) is a graph network operating in global coordinates, and is, in principle, able to uncover both the global and the local dynamics.
It is formulated as a VAE (Kingma & Welling, 2014; Rezende et al., 2014) with latent edge types and explicitly infers a latent graph structure.
LoCS (Koﬁnas et al., 2021), on the other hand, operates in local coordinates, and is, thus, unable to uncover the global dynamics.
For all methods, we use their publicly available source code.
Our source code, data, and models will be shared upon publication.
Our architecture and experimental setup closely follow Graber & Schwing (2020); Koﬁnas et al.
Unless speciﬁed differently, our neural ﬁeld has a hidden size of 512.
In charged particles and in n-body problems, we only use positions as input to the neural ﬁeld, while in trafﬁc scenes we also use orientations.
The full implementation details are presented in appendix A.2.2.
In all settings, we report the mean squared error of positions and velocities over time, E ( t ) = 1 ND (cid:80) Nn =1 (cid:107) x
t n − ˆx t n (cid:107) 2 2 .
Here we demonstrate indicative visualizations, and provide more extensive qualitative results in appendix C.
5. C HARGED PARTICLES
First, we study the effect of static ﬁelds, i.e. a single ﬁeld across all train, validation, and test simulations.
We extend the charged particles dataset from Kipf et al.
by adding a number of immovable sources.
Overall, these sources act like regular particles, exerting forces on the observable particles, except we ignore any forces exerted to them, and ﬁx their positions and velocities to zero.
We use M = 20 “source” particles and N = 5 “observable” particles.
Sources particles can in principle have different charge magnitudes than the observable particles, but for simplicity we set all magnitudes to | q | = 1 .
All particles carry either positive or negative charges with equal probability.
Following
), we remove virtual borders that cause elastic collisions.
We generate a dataset of 50,000 simulations for training, 10,000 for validation and 10,000 for testing.
Following Kipf et al.
(2018), each simulation lasts for 49 timesteps.
During inference, we use the ﬁrst 29 steps as input and predict the remaining 20 steps.
dNRI
LoCS
G-LoCS
Aether (Ours)    17 20 Step 0 .0 .0 .0 .0 .1 .
M S E
Total Errors (a)    9 1112 Step 0 .0 .1 .1 .2 .2 .
M S E
Total Errors (b)    5 Step 0 .0 .1 .1 .2 .2 .
M S E
Total Errors (c)
We compare our method against dNRI and LoCS, as well as
G-LoCS, which should in principle be able to model both local and global dynamics effectively.
We plot MSE in ﬁg. 4a and
L 2 errors in ﬁg. and visualize the learned ﬁeld in ﬁg. 5.
We visualize sample predictions in ﬁg. 8a, and showcase more predicted trajectories in appendix C.1.
We observe that equivariant methods like LoCS perform poorly, while the approximately equivariant G-LoCS performs much better than equivariant and non-equivariant methods.
Aether outperforms all other methods, demonstrating that it can disentangle equivariance .
Furthermore, as shown in ﬁg. 5, and ﬁg. 12 in appendix C.1.1, Aether can effectively discover the underlying ﬁeld .
5. T RAFFIC SCENES
Next, we study the effectiveness of static ﬁeld discovery in trafﬁc scenes.
We use inD (Bock et al., 2020), a dataset with real-world trafﬁc scenes that comprises trajectories of pedestrians, vehicles, and cyclists, recorded at 4 different locations.
We create a subset that contains scenes from a single location.
Namely, we choose “Frankenburg, Aachen”, since it is the location with most interactions in the dataset.
The subset corresponds to 12 recordings; we use 8 for training, 2 for validation, andfor testing.
We follow a similar experimental setting with Graber & Schwing (2020); Koﬁnas et al.
We divide each scene into 18-step sequences.
We use the ﬁrst 6 time steps as input and predict the next 12 time steps.
We plot MSE in ﬁg. 4b and L 2 errors in ﬁg. 21, and visualize the learned ﬁeld in ﬁg. 6, and in ﬁg. 17 in appendix C.2.1.
Since the learned ﬁeld is a function of positions and orientations, we only visualize it for 4 discrete orientations, namely the group C 4 = (cid:8) 0 , π 2 , π, 3 π 2 (cid:9) .
We visualize predictions in ﬁg. 8b, and showcase more in appendix C.2.
Again, we see that Aether outperforms all other methods.
The discovered ﬁeld, while hard to interpret, shows high activations that coincide with road locations and directions, indicating that it can guide objects through the topology of the road network.
5. G RAVITATIONAL FIELD
[Figure]
We now study the task of ﬁeld discovery with dynamic ﬁelds, i.e. ﬁelds that are different across simulations.
We extend the gravity dataset by Brandstetter et al.
by adding gravitational sources.
We create a dataset of 5,000 simulations for training, 1,000 for validation and 1,000 for testing.
We use N = 5 particles and M =source.
We set the masses of particles to m p = 1 , while the source has a mass of m s = 10 .
We generate trajectories of 49 timesteps.
We use the ﬁrst 44 timesteps as input and predict the remainingsteps.
We plot MSE ﬁg. 4c and L 2 errors in ﬁg. 22, and visualize the learned ﬁeld in ﬁg. 7.
We visualize sample predictions in ﬁg. 8c, and showcase more in appendix C.3.
We observe that Aether outperforms other methods, and can accurately discover the correct “shape” of the force ﬁelds, as well as the location of the sources.
5. A BLATION EXPERIMENTS
[Figure]
Signiﬁcance of discovered ﬁeld In a simulated environment like the charged particles setting, we have access to the simulator, and by extension, to the groundtruth ﬁelds and the sources that generate them.
We can leverage the simulator to study the signiﬁcance of the discovered ﬁeld in the task of trajectory forecasting, and establish an upper bound to our performance.
To that end, we create two “oracle” models that have access to the groundtruth ﬁelds, a force oracle and a source oracle .
The force oracle is identical to Aether, but uses the groundtruth forces from the simulator at each point instead of predicting them with a neural ﬁeld.
The source oracle assumes knowledge of the “ﬁeld sources”.
Thus, there is no longer need for disentanglement, and the problem is strictly equivariant again.
We include the sources as virtual nodes in the graph, add include edges from the sources to the particles.
We modify LoCS to use virtual nodes, and predict future states for all the “observable” particles, but not for the “source” particles.
We describe this oracle in detail in appendix A.2.4.
We plot the MSE in ﬁg. 9 and the L 2 errors in ﬁg. 23 in appendix D.4.
We observe that our model’s performance closely follows the two oracle models, and is, in fact, on par, for roughly 10 steps.
This demonstrates that the discovered ﬁeld is almost as helpful as the groundtruth.
C ONCLUSION
In this work, we introduced Aether , a method that disentangles local object interactions from global ﬁeld effects in interacting dynamical systems.
We proposed neural ﬁelds to discover such ﬁeld effects, and infer them from the dynamics alone.
We combined neural ﬁelds with equivariant graph networks to learn the dynamical systems.
We demonstrate that our method can accurately discover the underlying ﬁelds in a range of settings with ﬁxed and dynamic ﬁelds, and effectively use them to forecast future trajectories.
Limitations Fields are not always passive, but they may react to the observable environment.