Original: T RAJECTORY P REDICTION USING E QUIVARIANT C ONTINUOUS C ONVOLUTION
Revised: T RAJECTORY P REDICTION USING E QUIVARIANT C ONTINUOUS C ONVOLUTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Trajectory prediction is a critical part of many AI applications, for example, the safe operation of autonomous vehicles.
Revised: Trajectory prediction is a critical part of many AI applications, for example, the safe operation of autonomous vehicles.

Original: However, current methods are prone to making inconsistent and physically unrealistic predictions.
Revised: However, current methods are prone to making inconsistent and physically unrealistic predictions.

Original: We leverage insights from ﬂuid dynamics to overcome this limitation by considering internal symmetry in trajectories.
Revised: We leverage insights from ﬂuid dynamics to overcome this limitation by considering internal symmetry in trajectories.

Original: We propose a novel model, E quivariant C ontinous CO nvolution ( ECCO ) for improved trajectory prediction.
Revised: We propose a novel model, E quivariant C ontinous CO nvolution ( ECCO ) for improved trajectory prediction.

Original: ECCO uses rotationally-equivariant continuous convolutions to embed the symmetries of the system.
Revised: ECCO uses rotationally-equivariant continuous convolutions to embed the symmetries of the system.

Original: On two realworld vehicle and pedestrian trajectory datasets, ECCO attains competitive accuracy with signiﬁcantly fewer parameters.
Revised: On two realworld vehicle and pedestrian trajectory datasets, ECCO attains competitive accuracy with signiﬁcantly fewer parameters.

Original: It is also more sample efﬁcient, generalizing automatically from few data points in any orientation.
Revised: It is also more sample efﬁcient, generalizing automatically from few data points in any orientation.

Original: Lastly, ECCO improves generalization with equivariance, resulting in more physically consistent predictions.
Revised: Lastly, ECCO improves generalization with equivariance, resulting in more physically consistent predictions.

Original: Our method provides a fresh perspective towards increasing trust and transparency in deep learning models.
Revised: Our method provides a fresh perspective towards increasing trust and transparency in deep learning models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Trajectory prediction is one of the core tasks in AI, from the movement of basketball players to ﬂuid particles to car trafﬁc (Sanchez-Gonzalez et al., 2020; Gao et al., 2020; Shah & Romijnders, 2016).
Revised: Trajectory prediction is one of the core tasks in AI, from the movement of basketball players to ﬂuid particles to car trafﬁc (Sanchez-Gonzalez et al., 2020; Gao et al., 2020; Shah & Romijnders, 2016).

Original: A common abstraction underlying these tasks is the movement of many interacting agents, analogous to a many-particle system.
Revised: A common abstraction underlying these tasks is the movement of many interacting agents, analogous to a many-particle system.

Original: Therefore, understanding the states of these particles, their dynamics, and hidden interactions is critical to accurate and robust trajectory forecasting.
Revised: Therefore, understanding the states of these particles, their dynamics, and hidden interactions is critical to accurate and robust trajectory forecasting.

Original: Even for purely physical systems such as particle physics, the complex interactions among large number of particles makes this a difﬁcult problem.
Revised: Even for purely physical systems such as particle physics, the complex interactions among large number of particles makes this a difﬁcult problem.

Original: For vehicle or pedestrian trajectories, this challenge is further compounded with latent factors such as human psychology.
Revised: For vehicle or pedestrian trajectories, this challenge is further compounded with latent factors such as human psychology.

Original: Given these difﬁculties, current approaches require large amounts of training data and many parameters in the model.
Revised: Given these difﬁculties, current approaches require large amounts of training data and many parameters in the model.

Original: State-of-the-art methods in this domain such as Gao et al.
Revised: State-of-the-art methods in this domain such as Gao et al.

Original: (2020) are based on graph neural networks.
Revised: (2020) are based on graph neural networks.

Original: They do not exploit the physical properties of system and often make predictions which are not selfconsistent or physically meaningful.
Revised: They do not exploit the physical properties of system and often make predictions which are not selfconsistent or physically meaningful.

Original: Furthermore, they predict a single agent trajectory at a time instead of multiple agents simultaneously.
Revised: Furthermore, they predict a single agent trajectory at a time instead of multiple agents simultaneously.

Original: Our model is built upon a key insight of many-particle systems pertaining to intricate internal symmetry.
Revised: Our model is built upon a key insight of many-particle systems pertaining to intricate internal symmetry.

Original: Consider a model which predicts the trajectory of cars on a road.
Revised: Consider a model which predicts the trajectory of cars on a road.

Original: To be successful, such a model must understand the physical behavior of vehicles together with human psychology.
Revised: To be successful, such a model must understand the physical behavior of vehicles together with human psychology.

Original: It should distinguish left from right turns, and give consistent outputs for intersections rotated with different orientation.
Revised: It should distinguish left from right turns, and give consistent outputs for intersections rotated with different orientation.

Original: As shown in Figure 1, a driver’s velocity rotates with the entire scene, whereas vehicle interactions are invariant to such a rotation.
Revised: As shown in Figure 1, a driver’s velocity rotates with the entire scene, whereas vehicle interactions are invariant to such a rotation.

Original: Likewise, psychological factors such as reaction speed or attention may be considered vectors with prescribed transformation properties.
Revised: Likewise, psychological factors such as reaction speed or attention may be considered vectors with prescribed transformation properties.

Original: In this paper, we propose an equivariant continuous convolutional model, ECCO , for trajectory forecasting.
Revised: In this paper, we propose an equivariant continuous convolutional model, ECCO , for trajectory forecasting.

Original: Continuous convolution generalizes discrete convolution and is adapted to data in many- particle systems with complex local interactions.
Revised: Continuous convolution generalizes discrete convolution and is adapted to data in many- particle systems with complex local interactions.

Original: Ummenhofer et al.
Revised: Ummenhofer et al.

Original: (2019) designed a model using continuous convolutions for particle-based ﬂuid simulations.
Revised: (2019) designed a model using continuous convolutions for particle-based ﬂuid simulations.

Original: Meanwhile, equivariance to group symmetries has proven to be a powerful tool to integrate physical intuition in physical science applications (Wang et al., 2020; Brown & Lunter, 2019; Kanwar et al., 2020).
Revised: Meanwhile, equivariance to group symmetries has proven to be a powerful tool to integrate physical intuition in physical science applications (Wang et al., 2020; Brown & Lunter, 2019; Kanwar et al., 2020).

Original: Here, we test the hypothesis that an equivariant model can also capture internal symmetry in non-physical human behavior.
Revised: Here, we test the hypothesis that an equivariant model can also capture internal symmetry in non-physical human behavior.

Original: Our model utilizes a novel weight sharing scheme, torus kernels, and is rotationally equivariant.
Revised: Our model utilizes a novel weight sharing scheme, torus kernels, and is rotationally equivariant.

Original: We evaluate our model on two real-world trajectory datasets: Argoverse autonomous vehicle dataset (Chang et al., 2019) and TrajNet++ pedestrian trajectory forecasting challenge (Kothari et al., 2020).
Revised: We evaluate our model on two real-world trajectory datasets: Argoverse autonomous vehicle dataset (Chang et al., 2019) and TrajNet++ pedestrian trajectory forecasting challenge (Kothari et al., 2020).

Original: We demonstrate on par or better prediction accuracy to baseline models with fewer parameters, better sample efﬁciency, and stronger generalization properties.
Revised: We demonstrate on par or better prediction accuracy to baseline models with fewer parameters, better sample efﬁciency, and stronger generalization properties.

Original: Lastly, we demonstrate theoretically and experimentally that our polar coordinate-indexed ﬁlters have lower equivariance discretization error due to being better adapted to the symmetry group.
Revised: Lastly, we demonstrate theoretically and experimentally that our polar coordinate-indexed ﬁlters have lower equivariance discretization error due to being better adapted to the symmetry group.

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows:

Original: • We propose E quivariant C ontinous CO nvolution ( ECCO ), a rotationally equivariant deep neural network that can capture internal symmetry in trajectories.
Revised: • We propose E quivariant C ontinous CO nvolution ( ECCO ), a rotationally equivariant deep neural network that can capture internal symmetry in trajectories.

Original: • We design ECCO using a novel weight sharing scheme based on orbit decomposition and polar coordinate-indexed ﬁlters.
Revised: • We design ECCO using a novel weight sharing scheme based on orbit decomposition and polar coordinate-indexed ﬁlters.

Original: We implement equivariance for both the standard and regular representation L 2 (SO(2)) .
Revised: We implement equivariance for both the standard and regular representation L 2 (SO(2)) .

Original: • On benchmark Argoverse and TrajNet++ datasets, ECCO demonstrates comparable accuracy while enjoying better generalization, fewer parameters, and better sample complexity.
Revised: • On benchmark Argoverse and TrajNet++ datasets, ECCO demonstrates comparable accuracy while enjoying better generalization, fewer parameters, and better sample complexity.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Trajectory Forecasting For vehicle trajectories, classic models in transportation include the CarFollowing model (Pipes, 1966) and Intelligent Driver model (Kesting et al., 2010).
Revised: Trajectory Forecasting For vehicle trajectories, classic models in transportation include the CarFollowing model (Pipes, 1966) and Intelligent Driver model (Kesting et al., 2010).

Original: Deep learning has also received considerable attention; for example, Liang et al.
Revised: Deep learning has also received considerable attention; for example, Liang et al.

Original: (2020) and Gao et al.
Revised: (2020) and Gao et al.

Original: (2020) use graph neural networks to predict vehicle trajectories.
Revised: (2020) use graph neural networks to predict vehicle trajectories.

Original: Djuric et al.
Revised: Djuric et al.

Original: (2018) use rasterizations of the scene with CNN.
Revised: (2018) use rasterizations of the scene with CNN.

Original: See the review paper by Veres & Moussa (2019) for deep learning in transportation.
Revised: See the review paper by Veres & Moussa (2019) for deep learning in transportation.

Original: For human trajectory modeling, Alahi et al.
Revised: For human trajectory modeling, Alahi et al.

Original: (2016) propose Social LSTM to learn these humanhuman interactions.
Revised: (2016) propose Social LSTM to learn these humanhuman interactions.

Original: TrajNet (Sadeghian et al., 2018) and TrajNet++ (Kothari et al., 2020) introduce benchmarking for human trajectory forecasting.
Revised: TrajNet (Sadeghian et al., 2018) and TrajNet++ (Kothari et al., 2020) introduce benchmarking for human trajectory forecasting.

Original: We refer readers to Rudenko et al.
Revised: We refer readers to Rudenko et al.

Original: (2020) for a comprehensive survey.
Revised: (2020) for a comprehensive survey.

Original: Nevertheless, many deep learning models are data-driven.
Revised: Nevertheless, many deep learning models are data-driven.

Original: They require large amounts of data, have many parameters, and can generate physically inconsistent predictions.
Revised: They require large amounts of data, have many parameters, and can generate physically inconsistent predictions.

Original: Continuous Convolution Continuous convolutions over point clouds (CtsConv) have been successfully applied to classiﬁcation and segmentation tasks (Wang et al., 2018;
Revised: Continuous Convolution Continuous convolutions over point clouds (CtsConv) have been successfully applied to classiﬁcation and segmentation tasks (Wang et al., 2018;

Original: Lei et al., 2019; Xu et al., 2018; Wu et al., 2019; Su et al., 2018; Li et al., 2018; Hermosilla et al., 2018; Atzmon et al., 2018; Hua et al., 2018).
Revised: Lei et al., 2019; Xu et al., 2018; Wu et al., 2019; Su et al., 2018; Li et al., 2018; Hermosilla et al., 2018; Atzmon et al., 2018; Hua et al., 2018).

Original: More recently, a few works have used continuous convolution for modeling trajectories or ﬂows.
Revised: More recently, a few works have used continuous convolution for modeling trajectories or ﬂows.

Original: For instance, Wang et al.
Revised: For instance, Wang et al.

Original: (2018) uses CtsConv for inferring ﬂow on
Revised: (2018) uses CtsConv for inferring ﬂow on

Original: LIDAR data.
Revised: LIDAR data.

Original: Schenck & Fox (2018) and Ummenhofer et al.
Revised: Schenck & Fox (2018) and Ummenhofer et al.

Original: (2019) model ﬂuid simulation using
Revised: (2019) model ﬂuid simulation using

Original: CtsConv .
Revised: CtsConv .

Original: Closely related to our work is Ummenhofer et al.
Revised: Closely related to our work is Ummenhofer et al.

Original: (2019), who design a continuous convolution network for particle-based ﬂuid simulations.
Revised: (2019), who design a continuous convolution network for particle-based ﬂuid simulations.

Original: However, they use a ball-to-sphere mapping which is not well-adapted for rotational equivariance and only encode 3 frames of input.
Revised: However, they use a ball-to-sphere mapping which is not well-adapted for rotational equivariance and only encode 3 frames of input.

Original: Graph neural networks (GNNs) are a related strategy which have been used for modeling particle system dynamics (Sanchez-Gonzalez et al., 2020).
Revised: Graph neural networks (GNNs) are a related strategy which have been used for modeling particle system dynamics (Sanchez-Gonzalez et al., 2020).

Original: GNNs are also permutation invariant, but they do not natively encode relative positions and local interaction as a CtsConv-based network does.
Revised: GNNs are also permutation invariant, but they do not natively encode relative positions and local interaction as a CtsConv-based network does.

Original: Equivariant and Invariant Deep Learning Developing neural nets that preserve symmetries has been a fundamental task in image recognition (Cohen et al., 2019b; Weiler & Cesa, 2019; Cohen & Welling, 2016a; Chidester et al., 2018; Lenc & Vedaldi, 2015; Kondor & Trivedi, 2018; Bao &
Revised: Equivariant and Invariant Deep Learning Developing neural nets that preserve symmetries has been a fundamental task in image recognition (Cohen et al., 2019b; Weiler & Cesa, 2019; Cohen & Welling, 2016a; Chidester et al., 2018; Lenc & Vedaldi, 2015; Kondor & Trivedi, 2018; Bao &

Original: Song, 2019; Worrall et al., 2017; Cohen & Welling, 2016b; Weiler et al., 2018; Dieleman et al., 2016; Maron et al., 2020).
Revised: Song, 2019; Worrall et al., 2017; Cohen & Welling, 2016b; Weiler et al., 2018; Dieleman et al., 2016; Maron et al., 2020).

Original: Equivariant networks have also been used to predict dynamics: for example, Wang et al.
Revised: Equivariant networks have also been used to predict dynamics: for example, Wang et al.

Original: (2020) predicts ﬂuid ﬂow using Galilean equivariance but only for gridded data.
Revised: (2020) predicts ﬂuid ﬂow using Galilean equivariance but only for gridded data.

Original: Fuchs et al.
Revised: Fuchs et al.

Original: (2020) use SE(3) -equivariant transformers to predict trajectories for a small number of particles as a regression task.
Revised: (2020) use SE(3) -equivariant transformers to predict trajectories for a small number of particles as a regression task.

Original: As in this paper, both Bekkers (2020) and Finzi et al.
Revised: As in this paper, both Bekkers (2020) and Finzi et al.

Original: (2020) address the challenge of parameterizing a kernel over continuous Lie groups.
Revised: (2020) address the challenge of parameterizing a kernel over continuous Lie groups.

Original: Finzi et al.
Revised: Finzi et al.

Original: (2020) apply their method to trajectory prediction on point clouds using a small number of points following strict physical laws.
Revised: (2020) apply their method to trajectory prediction on point clouds using a small number of points following strict physical laws.

Original: Worrall et al.
Revised: Worrall et al.

Original: (2017) also parameterizes convolutional kernels using polar coordinates, but maps these onto a rectilinear grid for application to image data.
Revised: (2017) also parameterizes convolutional kernels using polar coordinates, but maps these onto a rectilinear grid for application to image data.

Original: Weng et al.
Revised: Weng et al.

Original: (2018) address rotational equivariance by inferring a global canonicalization of the input.
Revised: (2018) address rotational equivariance by inferring a global canonicalization of the input.

Original: Similar to our work,
Revised: Similar to our work,

Original: Esteves et al.
Revised: Esteves et al.

Original: (2018) use functions evenly sampled on the circle, however, their features are only at a single point whereas we assign feature vectors to each point in a point cloud.
Revised: (2018) use functions evenly sampled on the circle, however, their features are only at a single point whereas we assign feature vectors to each point in a point cloud.

Original: Thomas et al.
Revised: Thomas et al.

Original: (2018) introduce Tensor Field Networks which are SO(3) -equivariant continuous convolutions.
Revised: (2018) introduce Tensor Field Networks which are SO(3) -equivariant continuous convolutions.

Original: Unlike our work, both Worrall et al.
Revised: Unlike our work, both Worrall et al.

Original: (2017) and Thomas et al.
Revised: (2017) and Thomas et al.

Original: (2018) deﬁne their kernels using harmonic functions.
Revised: (2018) deﬁne their kernels using harmonic functions.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We ﬁrst review the necessary background of continuous convolution and rotational equivariance.
Revised: We ﬁrst review the necessary background of continuous convolution and rotational equivariance.

Original: 3. C ONTINUOUS C ONVOLUTION
Revised: 3. C ONTINUOUS C ONVOLUTION

Original: Continuous convolution generalizes the discrete convolution to point clouds.
Revised: Continuous convolution generalizes the discrete convolution to point clouds.

Original: It provides an efﬁcient and spatially aware way to model the interactions of nearby particles.
Revised: It provides an efﬁcient and spatially aware way to model the interactions of nearby particles.

Original: Let f ( i ) ∈
Revised: Let f ( i ) ∈

Original: R
Revised: R

Original: c
Revised: c

Original: in denote the feature vector of particle i .
Revised: in denote the feature vector of particle i .

Original: Thus f is a vector ﬁeld which assigns to the points x ( i ) a vector in R c in .
Revised: Thus f is a vector ﬁeld which assigns to the points x ( i ) a vector in R c in .

Original: The kernel of the convolution K : R 2 → R c out × c in is a matrix ﬁeld: for each point x ∈ R 2 , K ( x ) is a c out × c in matrix.
Revised: The kernel of the convolution K : R 2 → R c out × c in is a matrix ﬁeld: for each point x ∈ R 2 , K ( x ) is a c out × c in matrix.

Original: Let a be a radial local attention map with a ( r ) = 0 for r > R .
Revised: Let a be a radial local attention map with a ( r ) = 0 for r > R .

Original: Deﬁne the output feature vector g ( i ) of particle i by
Revised: Deﬁne the output feature vector g ( i ) of particle i by

Original: [Equation]
Revised: [Equation]

Original: We omit the attention normalization term of Ummenhofer et al.
Revised: We omit the attention normalization term of Ummenhofer et al.

Original: CtsConv is naturally equivariant to permutation of labels and translation invariant.
Revised: CtsConv is naturally equivariant to permutation of labels and translation invariant.

Original: 3. R OTATIONAL E QUIVARIANCE
Revised: 3. R OTATIONAL E QUIVARIANCE

Original: Continuous convolution is not naturally rotationally equivariant.
Revised: Continuous convolution is not naturally rotationally equivariant.

Original: Fortunately, we can translate the technique of rotational equivariance on CNNs to continuous convolutions.
Revised: Fortunately, we can translate the technique of rotational equivariance on CNNs to continuous convolutions.

Original: We use the language of
Revised: We use the language of

Original: Lie groups and their representations.
Revised: Lie groups and their representations.

Original: For more background, see Hall (2015) and Knapp (2013).
Revised: For more background, see Hall (2015) and Knapp (2013).

Original: More precisely, denote the symmetry group of 2D rotations by SO(2) =
Revised: More precisely, denote the symmetry group of 2D rotations by SO(2) =

Original: { Rot θ : 0 ≤ θ < 2 π } .
Revised: { Rot θ : 0 ≤ θ < 2 π } .

Original: It has a topological structure with underlying manifold the circle S 1 ∼ = { x ∈ R 2 : (cid:107) x
Revised: It has a topological structure with underlying manifold the circle S 1 ∼ = { x ∈ R 2 : (cid:107) x

Original: (cid:107) = 1 } .
Revised: (cid:107) = 1 } .

Original: The group SO(2) can act on a vector space R c by specifying a representation map ρ : SO(2) →
Revised: The group SO(2) can act on a vector space R c by specifying a representation map ρ : SO(2) →

Original: GL( R c ) which assigns to each element of SO(2) an element of the set of invertible c × c matrices GL( R c ) .
Revised: GL( R c ) which assigns to each element of SO(2) an element of the set of invertible c × c matrices GL( R c ) .

Original: For example, the standard representation ρ 1 on R 2 is by 2 × 2 rotation matrices.
Revised: For example, the standard representation ρ 1 on R 2 is by 2 × 2 rotation matrices.

Original: The regular representation ρ reg on L 2 (SO(2)) =
Revised: The regular representation ρ reg on L 2 (SO(2)) =

Original: { ϕ : SO(2)
Revised: { ϕ : SO(2)

Original: → R : | ϕ | 2 is integrable } is ρ reg (Rot φ )( ϕ ) =
Revised: → R : | ϕ | 2 is integrable } is ρ reg (Rot φ )( ϕ ) =

Original: ϕ ◦
Revised: ϕ ◦

Original: Rot − φ .
Revised: Rot − φ .

Original: Given input f with representation ρ in and output with representation ρ out , a map F is
Revised: Given input f with representation ρ in and output with representation ρ out , a map F is

Original: SO(2) -equivariant
Revised: SO(2) -equivariant

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: Discrete CNNs are equivariant to a group G if the input, output, and hidden layers carry a G -action and the linear layers and activation functions are all equivariant (Kondor & Trivedi, 2018).
Revised: Discrete CNNs are equivariant to a group G if the input, output, and hidden layers carry a G -action and the linear layers and activation functions are all equivariant (Kondor & Trivedi, 2018).

Original: One method for constructing equivariant discrete CNNs is steerable CNN (Cohen & Welling, 2016b).
Revised: One method for constructing equivariant discrete CNNs is steerable CNN (Cohen & Welling, 2016b).

Original: Cohen et al.
Revised: Cohen et al.

Original: (2019a) derive a general constraint for when a convolutional kernel K : R b
Revised: (2019a) derive a general constraint for when a convolutional kernel K : R b

Original: → R c out × c in is G -equivariant.
Revised: → R c out × c in is G -equivariant.

Original: Assume G acts on R b and that R c out and R c in are G -representations ρ out and ρ in respectively, then K is G -equivariant if for all g ∈ G, x ∈ R 2 ,
Revised: Assume G acts on R b and that R c out and R c in are G -representations ρ out and ρ in respectively, then K is G -equivariant if for all g ∈ G, x ∈ R 2 ,

Original: [Equation]
Revised: [Equation]

Original: ρ in ( g − 1 ) .
Revised: ρ in ( g − 1 ) .

Original: [Equation]
Revised: [Equation]

Original: For the group SO (2) , Weiler & Cesa (2019) solve this constraint using circular harmonic functions to give a basis of discrete equivariant kernels.
Revised: For the group SO (2) , Weiler & Cesa (2019) solve this constraint using circular harmonic functions to give a basis of discrete equivariant kernels.

Original: In contrast, our method described in then next section will use orbits and stabilizers to create continuous convolution kernels.
Revised: In contrast, our method described in then next section will use orbits and stabilizers to create continuous convolution kernels.

Original:  ECCO: T RAJECTORY P REDICTION USING R OTATIONALLY E QUIVARIANT C ONTINUOUS C ONVOLUTION
Revised:  ECCO: T RAJECTORY P REDICTION USING R OTATIONALLY E QUIVARIANT C ONTINUOUS C ONVOLUTION

Original: In trajectory prediction, given historical position and velocity data of n particles over t in timesteps, we want to predict their positions over the next t out timesteps.
Revised: In trajectory prediction, given historical position and velocity data of n particles over t in timesteps, we want to predict their positions over the next t out timesteps.

Original: Denote the ground truth dynamics as ξ , which maps ξ ( x t − t in : t , v t − t in : t ) =
Revised: Denote the ground truth dynamics as ξ , which maps ξ ( x t − t in : t , v t − t in : t ) =

Original: x t : t + t out .
Revised: x t : t + t out .

Original: Motivated by the observation in Figure 1, we wish to learn a model f that approximates the dynamics while preserving the internal symmetry, speciﬁcally rotational equivariance.
Revised: Motivated by the observation in Figure 1, we wish to learn a model f that approximates the dynamics while preserving the internal symmetry, speciﬁcally rotational equivariance.

Original: We introduce ECCO , a model for trajectory prediction based on rotationally equivariant continuous convolution.
Revised: We introduce ECCO , a model for trajectory prediction based on rotationally equivariant continuous convolution.

Original: We implement rotationally equivariant continuous convolutions using a weight sharing scheme based on orbit decomposition.
Revised: We implement rotationally equivariant continuous convolutions using a weight sharing scheme based on orbit decomposition.

Original: We also describe equivariant per-particle linear layers which are a special case of continuous convolution with radius R = 

Revised: We also describe equivariant per-particle linear layers which are a special case of continuous convolution with radius R =

Original: analogous to 1x1 convolutions in
Revised: analogous to 1x1 convolutions in

Original: CNNs.
Revised: CNNs.

Original: Such layers are useful for passing information between layers from each particle to itself.
Revised: Such layers are useful for passing information between layers from each particle to itself.

Original: [Figure]
Revised: [Figure]

Original: The high-level architecture of ECCO is illustrated in Figure 2.
Revised: The high-level architecture of ECCO is illustrated in Figure 2.

Original: It is important to remember that the input, output, and hidden layers are all vector ﬁelds over the particles.
Revised: It is important to remember that the input, output, and hidden layers are all vector ﬁelds over the particles.

Original: For one dataset, there is also environmental information available in the form of road lane markers.
Revised: For one dataset, there is also environmental information available in the form of road lane markers.

Original: Denote marker positions by x map and direction vectors by v map .
Revised: Denote marker positions by x map and direction vectors by v map .

Original: This data is thus also a particle ﬁeld, but static.
Revised: This data is thus also a particle ﬁeld, but static.

Original: To design an equivariant network, one must choose the group representation.
Revised: To design an equivariant network, one must choose the group representation.

Original: This choice plays an important role in shaping the learned hidden states.
Revised: This choice plays an important role in shaping the learned hidden states.

Original: We focus on two representations of SO(2) : ρand ρ reg .
Revised: We focus on two representations of SO(2) : ρand ρ reg .

Original: The representation ρ 1 is that of our input features, and ρ reg is for the hidden layers.
Revised: The representation ρ 1 is that of our input features, and ρ reg is for the hidden layers.

Original: For ρ 1 , we constrain the kernel in Equation 1.
Revised: For ρ 1 , we constrain the kernel in Equation 1.

Original: For ρ reg , we further introduce a new operator, convolution with torus kernels.
Revised: For ρ reg , we further introduce a new operator, convolution with torus kernels.

Original: In order to make continuous convolution rotationally equivariant, we translate the general constraint for discrete CNNs developed in Weiler & Cesa (2019) to continuous convolution.
Revised: In order to make continuous convolution rotationally equivariant, we translate the general constraint for discrete CNNs developed in Weiler & Cesa (2019) to continuous convolution.

Original: Let R c out and R c in be SO(2) -representations ρ out and ρ in respectively, then we require
Revised: Let R c out and R c in be SO(2) -representations ρ out and ρ in respectively, then we require

Original: [Equation]
Revised: [Equation]

Original: Imposing such a constraint for continuous convolution requires us to develop an efﬁcient weight sharing scheme for the kernels K ( θ, r ) , which solve Equation 3.
Revised: Imposing such a constraint for continuous convolution requires us to develop an efﬁcient weight sharing scheme for the kernels K ( θ, r ) , which solve Equation 3.

Original: 4. W EIGHT S HARING BY O RBITS AND S TABILIZERS .
Revised: 4. W EIGHT S HARING BY O RBITS AND S TABILIZERS .

Original: Given a point x ∈ R 2 and a group G , the set
Revised: Given a point x ∈ R 2 and a group G , the set

Original: O x = { g x : g ∈ G } is the orbit of the point x .
Revised: O x = { g x : g ∈ G } is the orbit of the point x .

Original: The set of orbits gives a partition of R 2 into the origin and circles of radius r > 0 .
Revised: The set of orbits gives a partition of R 2 into the origin and circles of radius r > 0 .

Original: The set of group elements G x =
Revised: The set of group elements G x =

Original: { g : g x = x } ﬁxing x is called the stabilizer of the point x .
Revised: { g : g x = x } ﬁxing x is called the stabilizer of the point x .

Original: We use the orbits and stabilizers to constrain the weights of K .
Revised: We use the orbits and stabilizers to constrain the weights of K .

Original: Simply put, we share weights across orbits and constrain weights according to stabilizers, as shown in Figure 3-Left.
Revised: Simply put, we share weights across orbits and constrain weights according to stabilizers, as shown in Figure 3-Left.

Original: The ray D = { (0 , r ) : r ≥ 0 } is a fundamental domain for the action of G = SO(2) on base space R 2 .
Revised: The ray D = { (0 , r ) : r ≥ 0 } is a fundamental domain for the action of G = SO(2) on base space R 2 .

Original: That is, D contains exactly one point from each orbit.
Revised: That is, D contains exactly one point from each orbit.

Original: We ﬁrst deﬁne K (0 , r ) for each
Revised: We ﬁrst deﬁne K (0 , r ) for each

Original: (0 , r ) ∈ D .
Revised: (0 , r ) ∈ D .

Original: Then we obtain K ( θ, r ) from K (0 , r ) by setting φ = 0 in Equation 3 as such
Revised: Then we obtain K ( θ, r ) from K (0 , r ) by setting φ = 0 in Equation 3 as such

Original: [Equation]
Revised: [Equation]

Original: For r > 0 , the group acts freely on (0 , r ) , i.e. the stabilizer contains only the identity.
Revised: For r > 0 , the group acts freely on (0 , r ) , i.e. the stabilizer contains only the identity.

Original: This means that Equation 3 imposes no additional constraints on K (0 , r ) .
Revised: This means that Equation 3 imposes no additional constraints on K (0 , r ) .

Original: Thus K (0 , r ) ∈ R c out × c in is a matrix of freely learnable weights.
Revised: Thus K (0 , r ) ∈ R c out × c in is a matrix of freely learnable weights.

Original: For r = 0 , however, the orbit O (0 , 0) is only one point.
Revised: For r = 0 , however, the orbit O (0 , 0) is only one point.

Original: The stabilizer of (0 , 0) is all of G , which requires
Revised: The stabilizer of (0 , 0) is all of G , which requires

Original: [Equation]
Revised: [Equation]

Original: Thus K (0 , 0) is an equivariant per-particle linear map ρ in → ρ out .
Revised: Thus K (0 , 0) is an equivariant per-particle linear map ρ in → ρ out .

Original: We can analytically solve Eqn.
Revised: We can analytically solve Eqn.

Original: (5) for K (0 , 0) using representation theory.
Revised: (5) for K (0 , 0) using representation theory.

Original: Table 1 shows the unique solutions for different combinations of ρ 1 and
Revised: Table 1 shows the unique solutions for different combinations of ρ 1 and

Original: ρ reg .
Revised: ρ reg .

Original: For details see subsection A.2.
Revised: For details see subsection A.2.

Original: linear maps for K (0 , 0) .
Revised: linear maps for K (0 , 0) .

Original: Trainable weights are c ∈ R and κ : S 1 → R , where S 1 is the manifold underlying SO(2) .
Revised: Trainable weights are c ∈ R and κ : S 1 → R , where S 1 is the manifold underlying SO(2) .

Original: [Equation]
Revised: [Equation]

Original: Note that 2D and 3D rotation equivariant continuous convolutions are implemented in Worrall et al.
Revised: Note that 2D and 3D rotation equivariant continuous convolutions are implemented in Worrall et al.

Original: (2017) and Thomas et al.
Revised: (2017) and Thomas et al.

Original: (2018) respectively.
Revised: (2018) respectively.

Original: They both use harmonic functions which require expensive evaluation of analytic functions at each point.
Revised: They both use harmonic functions which require expensive evaluation of analytic functions at each point.

Original: Instead, we provide a simpler solution.
Revised: Instead, we provide a simpler solution.

Original: We require only knowledge of the orbits, stabilizers, and input/output representations.
Revised: We require only knowledge of the orbits, stabilizers, and input/output representations.

Original: Additionally, we bypass Clebsch-Gordon decomposition used in
Revised: Additionally, we bypass Clebsch-Gordon decomposition used in

Original: Thomas et al.
Revised: Thomas et al.

Original: (2018) by mapping directly between the representations in our network.
Revised: (2018) by mapping directly between the representations in our network.

Original: Next, we describe an efﬁcient implementation of equivariant continuous convolution.
Revised: Next, we describe an efﬁcient implementation of equivariant continuous convolution.

Original: 4. P OLAR C OORDINATE K ERNELS
Revised: 4. P OLAR C OORDINATE K ERNELS

Original: Rotational equivariance informs our kernel discretization and implementation.
Revised: Rotational equivariance informs our kernel discretization and implementation.

Original: We store the kernel
Revised: We store the kernel

Original: K of continuous convolution as a 4-dimensional tensor by discretizing the domain.
Revised: K of continuous convolution as a 4-dimensional tensor by discretizing the domain.

Original: Speciﬁcally, we discretize R 2 using polar coordinates with k θ angular slices and k r radial steps.
Revised: Speciﬁcally, we discretize R 2 using polar coordinates with k θ angular slices and k r radial steps.

Original: We then evaluate
Revised: We then evaluate

Original: K at any ( θ, r ) using interpolation.
Revised: K at any ( θ, r ) using interpolation.

Original: This method accelerates computation since we do not need to use Equation 4 to repeatedly compute K ( θ, r ) from K (0 , r ) .
Revised: This method accelerates computation since we do not need to use Equation 4 to repeatedly compute K ( θ, r ) from K (0 , r ) .

Original: The special case of K (0 , 0) results in a polar grid with a “bullseye” at the center (see Figure 3-Left).
Revised: The special case of K (0 , 0) results in a polar grid with a “bullseye” at the center (see Figure 3-Left).

Original: We discretize angles ﬁnely and radii more coarsely.
Revised: We discretize angles ﬁnely and radii more coarsely.

Original: This choice is inspired by real-world observation that drivers tend to be more sensitive to the angle of an incoming car than its exact distance, Our equivariant kernels are computationally efﬁcient and have very few parameters.
Revised: This choice is inspired by real-world observation that drivers tend to be more sensitive to the angle of an incoming car than its exact distance, Our equivariant kernels are computationally efﬁcient and have very few parameters.

Original: Moreover, we will discuss later in Section 4.5 that despite discretization, the use of polar coordinates allows for very low equivariance error.
Revised: Moreover, we will discuss later in Section 4.5 that despite discretization, the use of polar coordinates allows for very low equivariance error.

Original: 4. H IDDEN L AYERS AS R EGULAR R EPRESENTATIONS
Revised: 4. H IDDEN L AYERS AS R EGULAR R EPRESENTATIONS

Original: The representation ρ reg has shown better performance than ρ 1 for ﬁnite groups (Cohen et al., 2019a;
Revised: The representation ρ reg has shown better performance than ρ 1 for ﬁnite groups (Cohen et al., 2019a;

Original: Weiler & Cesa, 2019).
Revised: Weiler & Cesa, 2019).

Original: But the naive ρ reg representation { ϕ : G → R } for an inﬁnite group G is too large to work with.
Revised: But the naive ρ reg representation { ϕ : G → R } for an inﬁnite group G is too large to work with.

Original: We choose the space of square-integrable functions L 2 ( G ) .
Revised: We choose the space of square-integrable functions L 2 ( G ) .

Original: It contains all irreducible representations of G and is compatible with pointwise non-linearities.
Revised: It contains all irreducible representations of G and is compatible with pointwise non-linearities.

Original: Discretization.
Revised: Discretization.

Original: However, L 2 (SO(2)) is still inﬁnite-dimensional.
Revised: However, L 2 (SO(2)) is still inﬁnite-dimensional.

Original: We resolve this by discretizing the manifold S 1 underlying SO(2) into k reg even intervals.
Revised: We resolve this by discretizing the manifold S 1 underlying SO(2) into k reg even intervals.

Original: We represent functions f ∈ L 2 (SO(2)) by the vector of values [ f (Rot 2 πi/k reg )] 0 ≤ i<k reg .
Revised: We represent functions f ∈ L 2 (SO(2)) by the vector of values [ f (Rot 2 πi/k reg )] 0 ≤ i<k reg .

Original: We then evaluate f (Rot θ ) using interpolation.
Revised: We then evaluate f (Rot θ ) using interpolation.

Original: We separate the number of angular slices k θ and the size of the kernel k reg .
Revised: We separate the number of angular slices k θ and the size of the kernel k reg .

Original: If we tie them together and set k θ = k reg , this is equivalent to implementing cyclic group C k reg symmetry with the regular representation.
Revised: If we tie them together and set k θ = k reg , this is equivalent to implementing cyclic group C k reg symmetry with the regular representation.

Original: Then increasing k θ would also increases k reg , which incurs more parameters.
Revised: Then increasing k θ would also increases k reg , which incurs more parameters.

Original: Convolution with Torus Kernel.
Revised: Convolution with Torus Kernel.

Original: In addition to constraining the kernel K of Equation 1 as in ρ 1 , ρ reg poses an additional challenge as it is a function on a circle.
Revised: In addition to constraining the kernel K of Equation 1 as in ρ 1 , ρ reg poses an additional challenge as it is a function on a circle.

Original: We introduce a new operator from functions on the circle to functions on the circle called a torus kernel.
Revised: We introduce a new operator from functions on the circle to functions on the circle called a torus kernel.

Original: First, we replace vectors in R c with elements of L 2 (SO(2)) .
Revised: First, we replace vectors in R c with elements of L 2 (SO(2)) .

Original: The input feature f becomes a ρ regﬁeld, that is, for each x ∈ R 2 , f ( x ) is a function S 1 → R .
Revised: The input feature f becomes a ρ regﬁeld, that is, for each x ∈ R 2 , f ( x ) is a function S 1 → R .

Original: For the kernel K , we replace the matrix ﬁeld with a map K : R 2 → ρ reg ⊗ ρ reg .
Revised: For the kernel K , we replace the matrix ﬁeld with a map K : R 2 → ρ reg ⊗ ρ reg .

Original: Instead of a matrix, K ( x ) is a map S 1 × S 1 → R .
Revised: Instead of a matrix, K ( x ) is a map S 1 × S 1 → R .

Original: Topologically, S 1 × S 1 is a torus and hence we call K ( x ) a torus kernel.
Revised: Topologically, S 1 × S 1 is a torus and hence we call K ( x ) a torus kernel.

Original: The matrix multiplication
Revised: The matrix multiplication

Original: K ( x ) · f ( x ) in Equation 1 must be replaced by the integral transform
Revised: K ( x ) · f ( x ) in Equation 1 must be replaced by the integral transform

Original: [Equation]
Revised: [Equation]

Original: which is a linear transformation L 2 (SO(2))
Revised: which is a linear transformation L 2 (SO(2))

Original: → L 2 (SO(2)) .
Revised: → L 2 (SO(2)) .

Original: See the illustration in Figure 3-Right.
Revised: See the illustration in Figure 3-Right.

Original: We compute Equation 3 for ρ reg → ρ reg as K (Rot θ ( x ))
Revised: We compute Equation 3 for ρ reg → ρ reg as K (Rot θ ( x ))

Original: ( φ 2 , φ 1 )
Revised: ( φ 2 , φ 1 )

Original: =
Revised: =

Original: K ( x )( φ 2 − θ, φ 1 − θ ) .
Revised: K ( x )( φ 2 − θ, φ 1 − θ ) .

Original: We can use the same weight sharing scheme as in Section 4.2.
Revised: We can use the same weight sharing scheme as in Section 4.2.

Original: 4. A NALYSIS : E QUIVARIANCE E RROR
Revised: 4. A NALYSIS : E QUIVARIANCE E RROR

Original: The practical value of equivariant neural networks has been demonstrated in a variety of domains.
Revised: The practical value of equivariant neural networks has been demonstrated in a variety of domains.

Original: However, theoretical analysis (Kondor & Trivedi, 2018; Cohen et al., 2019a; Maron et al., 2020) of continuous Lie group symmetries is usually performed assuming continuous functions and using the integral representation of the convolution operator.
Revised: However, theoretical analysis (Kondor & Trivedi, 2018; Cohen et al., 2019a; Maron et al., 2020) of continuous Lie group symmetries is usually performed assuming continuous functions and using the integral representation of the convolution operator.

Original: In practice, discretization can lead the model f to be not exactly equivariant but have some equivariance error EE
Revised: In practice, discretization can lead the model f to be not exactly equivariant but have some equivariance error EE

Original: = (cid:107) f
Revised: = (cid:107) f

Original: ( T ( x )) − T (cid:48)
Revised: ( T ( x )) − T (cid:48)

Original: ( f ( x )) (cid:107) with respect to group transformations T and T (cid:48) of input and output respectively (Wang et al., 2020, A6).
Revised: ( f ( x )) (cid:107) with respect to group transformations T and T (cid:48) of input and output respectively (Wang et al., 2020, A6).

Original: Rectangular grids are well-suited to translations, but poorly-suited to rotations.
Revised: Rectangular grids are well-suited to translations, but poorly-suited to rotations.

Original: The resulting equivariance error can be so large to practically undermine the advantages of a theoretically equivariant network.
Revised: The resulting equivariance error can be so large to practically undermine the advantages of a theoretically equivariant network.

Original: The polar-coordinate indexed circular ﬁlters we propose are designed speciﬁcally to adapt well to the rotational symmetry we implement.
Revised: The polar-coordinate indexed circular ﬁlters we propose are designed speciﬁcally to adapt well to the rotational symmetry we implement.

Original: In Figure 4 we demonstrate experimentally that expected EE is inversely proportional to k θ .
Revised: In Figure 4 we demonstrate experimentally that expected EE is inversely proportional to k θ .

Original: Choosing, e.g., k θ ≥ 16 gives very low EE and does not increase the number of parameters.
Revised: Choosing, e.g., k θ ≥ 16 gives very low EE and does not increase the number of parameters.

Original: We also prove for ρ 1 features that the equivariance error has low expectation.
Revised: We also prove for ρ 1 features that the equivariance error has low expectation.

Original: See Appendix A.5 for the precise statement and proof.
Revised: See Appendix A.5 for the precise statement and proof.

Original: Proposition.
Revised: Proposition.

Original: Let α = 2 π/k θ , and ¯ θ be θ rounded to nearest value in Z α , and ˆ θ = | θ − ¯ θ | .
Revised: Let α = 2 π/k θ , and ¯ θ be θ rounded to nearest value in Z α , and ˆ θ = | θ − ¯ θ | .

Original: Let
Revised: Let

Original: F = CtsConv K,R and T = ρ 1 (Rot θ ) .
Revised: F = CtsConv K,R and T = ρ 1 (Rot θ ) .

Original: For some constant C , the expected EE is bounded
Revised: For some constant C , the expected EE is bounded

Original: E K, f , x [ T ( F ( f , x ))
Revised: E K, f , x [ T ( F ( f , x ))

Original: − F ( T ( f ) , T ( x ))] ≤ | sin(ˆ θ ) | C ≤ 2 πC/k θ .
Revised: − F ( T ( f ) , T ( x ))] ≤ | sin(ˆ θ ) | C ≤ 2 πC/k θ .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present experiments in two different domains, trafﬁc and pedestrian trajectory prediction, where interactions among agents are frequent and the inﬂuence is obvious.
Revised: In this section, we present experiments in two different domains, trafﬁc and pedestrian trajectory prediction, where interactions among agents are frequent and the inﬂuence is obvious.

Original: We ﬁrst introduce the datasets and the metrics.
Revised: We ﬁrst introduce the datasets and the metrics.

Original: Secondly, we compare different feature encoders and hidden feature representation types.
Revised: Secondly, we compare different feature encoders and hidden feature representation types.

Original: Lastly, we compare our model with baselines.
Revised: Lastly, we compare our model with baselines.

Original: 5. E XPERIMENTAL S ET U P
Revised: 5. E XPERIMENTAL S ET U P

Original: Dataset We discuss the performances of our models on (1) Argoverse autonomous vehicle motion forecasting (Chang et al., 2019), a recently released vehicle trajectory prediction benchmark, and (2) TrajNet++ pedestrian trajectory forecasting challenge (Kothari et al., 2020).
Revised: Dataset We discuss the performances of our models on (1) Argoverse autonomous vehicle motion forecasting (Chang et al., 2019), a recently released vehicle trajectory prediction benchmark, and (2) TrajNet++ pedestrian trajectory forecasting challenge (Kothari et al., 2020).

Original: For
Revised: For

Original: Argoverse, the task is to predict three-second trajectories based on all vehicles history in the past 2 seconds.
Revised: Argoverse, the task is to predict three-second trajectories based on all vehicles history in the past 2 seconds.

Original: We split 32K samples from the validation set as our test set.
Revised: We split 32K samples from the validation set as our test set.

Original: Baselines We compare against several state-of-the-art baselines used in Argoverse and TrajNet++.
Revised: Baselines We compare against several state-of-the-art baselines used in Argoverse and TrajNet++.

Original: We use three original baselines from (Chang et al., 2019):
Revised: We use three original baselines from (Chang et al., 2019):

Original: Constant velocity, Nearest Neighbour, and Long Short Term Memory (LSTM).
Revised: Constant velocity, Nearest Neighbour, and Long Short Term Memory (LSTM).

Original: We also compare with a non-equivariant continuous convolutional model, CtsConv (Ummenhofer et al., 2019) and a hierarchical GNN model VectorNet (Gao et al., 2020).
Revised: We also compare with a non-equivariant continuous convolutional model, CtsConv (Ummenhofer et al., 2019) and a hierarchical GNN model VectorNet (Gao et al., 2020).

Original: Note that VectorNet only predicts a single agent at a time, which is not directly comparable with ours.
Revised: Note that VectorNet only predicts a single agent at a time, which is not directly comparable with ours.

Original: We include VectorNet as a reference nevertheless.
Revised: We include VectorNet as a reference nevertheless.

Original: Evaluation Metrics We use domain standard metrics to evaluate the trajectory prediction performance, including (1) Average Displacement Error (ADE): the average L2 displacement error for the whole 30 timestamps between prediction and ground truth, (2) Displacement Error at t seconds (DE@ts): the L2 displacement error at a given timestep t .
Revised: Evaluation Metrics We use domain standard metrics to evaluate the trajectory prediction performance, including (1) Average Displacement Error (ADE): the average L2 displacement error for the whole 30 timestamps between prediction and ground truth, (2) Displacement Error at t seconds (DE@ts): the L2 displacement error at a given timestep t .

Original: DE@ts for the last timestamp is also called Final Displacement Error (FDE).
Revised: DE@ts for the last timestamp is also called Final Displacement Error (FDE).

Original: For Argoverse, we report ADE and DE@ts for t ∈ { 1 , 2 , 3 } .
Revised: For Argoverse, we report ADE and DE@ts for t ∈ { 1 , 2 , 3 } .

Original: For TrajNet++, we report ADE and FDE.
Revised: For TrajNet++, we report ADE and FDE.

Original: 5. P REDICTION P ERFORMANCE C OMPARISON
Revised: 5. P REDICTION P ERFORMANCE C OMPARISON

Original: We evaluate the performance of different models from multiple aspects: forecasting accuracy, parameter efﬁciency and the physical consistency of the predictions.
Revised: We evaluate the performance of different models from multiple aspects: forecasting accuracy, parameter efﬁciency and the physical consistency of the predictions.

Original: The goal is to provide a comprehensive view of various characteristics of the model to guide practical deployment.
Revised: The goal is to provide a comprehensive view of various characteristics of the model to guide practical deployment.

Original: See Appendix
Revised: See Appendix

Original: A.8 for an additional ablative study.
Revised: A.8 for an additional ablative study.

Original: Forecasting Accuracy We compare the trajectory prediction accuracy across different models on
Revised: Forecasting Accuracy We compare the trajectory prediction accuracy across different models on

Original: Argoverse and TrajNet++.
Revised: Argoverse and TrajNet++.

Original: Table 2 displays the prediction ADE and FDE comparision.
Revised: Table 2 displays the prediction ADE and FDE comparision.

Original: We can see that ECCO with the regular representation ρ reg achieves on par or better forecasting accuracy on both datasets.
Revised: We can see that ECCO with the regular representation ρ reg achieves on par or better forecasting accuracy on both datasets.

Original: Comparing ECCO and a non-equivariant counterpart of our model CtsConv, we observe a signiﬁcant 14.8% improvement in forecasting accuracy.
Revised: Comparing ECCO and a non-equivariant counterpart of our model CtsConv, we observe a signiﬁcant 14.8% improvement in forecasting accuracy.

Revised: And we also observe a 9% improvement when comparing ECCO with the non-equivariant counterpart trained on random-rotation-augmented dataset.

Original: These results demonstrate the beneﬁts of incorporating equivariance principles into deep learning models.
Revised: These results demonstrate the beneﬁts of incorporating equivariance principles into deep learning models.

Revised: [Table]

Original: Parameter Efﬁciency
Revised: Parameter Efﬁciency

Original: Another important feature in deploying deep learning models to embedded systems such as autonomous vehicles is parameter efﬁciency.
Revised: Another important feature in deploying deep learning models to embedded systems such as autonomous vehicles is parameter efﬁciency.

Original: The number of parameters has strong implications in memory usage, battery consumption and compute speed.

Original: We report the number of

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Figure] parameters in each of the models in Table 2.
Revised: We report the number of parameters in each of the models in Table 2.

Original: Compare with LSTM, our forecasting performance is better.
Revised: Compare with LSTM, our forecasting performance is better.

Original: CtsConv and VectorNet have competitive forecasting performance, but uses more parameters than ECCO .
Revised: CtsConv and VectorNet have competitive forecasting performance, but uses more parameters than ECCO .

Original: By encoding inductive bias of equivariance into CtsConv, we drastically reduce the number of the parameters needed in our model.
Revised: By encoding inductive bias of equivariance into CtsConv, we drastically reduce the number of the parameters needed in our model.

Original: For VectorNet, Gao et al.
Revised: For VectorNet, Gao et al.

Original: (2020) only provided the number of parameters for their encoder; a fair decoder size can be estimated based on MLP using 59 polygraphs with each 64 dimensions as input, predicting 30 timestamps, that is 113K
Revised: (2020) only provided the number of parameters for their encoder; a fair decoder size can be estimated based on MLP using 59 polygraphs with each 64 dimensions as input, predicting 30 timestamps, that is 113K.

Revised: Runtime and Memory Efﬁciency We compare runtime and memory usage with VectorNet.

Revised: Firstly, we compare ﬂoating point operations (FLOPs) versus those reported in Gao et al.

Original: [Figure]
Revised: They report n × 0 . 041 GFLOPs for the encoder part of their model alone, where n is the number of predicted vehicles. We tested ECCO on a scene with 30 vehicles and approximately 180 lane marker nodes, which is similar to the test conditions used to compute FLOPs in Gao et al. Our full model used 1.03 GFLOPs versus 1.23 GFLOPs for VectorNet’s encoder.

Revised: Moreover, we compare the runtimes and memory usage using a version of VectorNet we implement since VectorNet is not open source.

Revised: For runtimes on the same test machine, ECCO runs 684ms versus 1103ms for VectorNet.

Revised: The disadvantage of VectorNet is needing to reprocess the scene for each agent, whereas ECCO predicts all agents simultaneously.

Revised: For memory usage in the same test ECCO uses 296 MB and VectorNet uses 171 MB.

Revised: [Figure]

Original: Sample Efﬁciency
Revised: Sample Efﬁciency

Original: A major beneﬁt of incorporating symmetry is to incorporate inductive bias and improve the sample efﬁciency of learning.
Revised: A major beneﬁt of incorporating symmetry is to incorporate inductive bias and improve the sample efﬁciency of learning.

Original: For each sample an equivariant model is trained on, it learns as though it were trained on all transformations of that sample by the symmetry group (Wang et al., 2020, Prop 3).
Revised: For each sample an equivariant model is trained on, it learns as though it were trained on all transformations of that sample by the symmetry group (Wang et al., 2020, Prop 3).

Original: Thus
Revised: Thus

Original: ECCO requires fewer samples to learn from.
Revised: ECCO requires fewer samples to learn from.

Original: In
Revised: In

Original: Figure 5, we plot a comparison of validation
Revised: Figure 5, we plot a comparison of validation

Original: FDE over number of training samples and show the equivariant models converge faster.
Revised: FDE over number of training samples and show the equivariant models converge faster.

Original: Physical Consistency We also visualize the predictions from ECCO and non-equivariant
Revised: Physical Consistency We also visualize the predictions from ECCO and non-equivariant

Original: CtsConv, as shown in Figure 6.
Revised: CtsConv, as shown in Figure 6.

Original: Top row visualizes the predictions on the original data.
Revised: Top row visualizes the predictions on the original data.

Original: In the bottom row, we rotate the whole scene by 160 ◦ and make predictions on rotated data.
Revised: In the bottom row, we rotate the whole scene by 160 ◦ and make predictions on rotated data.

Original: This mimics the distributional shift in the real world.
Revised: This mimics the distributional shift in the real world.

Original: Note that CtsConv predicts inconsistently: a right turn in the top row but a left turn after the scene has been rotated.
Revised: Note that CtsConv predicts inconsistently: a right turn in the top row but a left turn after the scene has been rotated.

Original: We see similar results for TrajNet++ (see Figure 8 in Appendix A.9).
Revised: We see similar results for TrajNet++ (see Figure 8 in Appendix A.9).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Many AI applications require more than low average error, they require consistent and physically realistic predictions.
Revised: Many AI applications require more than low average error, they require consistent and physically realistic predictions.

Original: We work towards this goal for trajectory prediction.
Revised: We work towards this goal for trajectory prediction.

Original: Our model ECCO achieves improved consistency by embedding the symmetries of the problem as inductive biases.
Revised: Our model ECCO achieves improved consistency by embedding the symmetries of the problem as inductive biases.

Original: On two real-world vehicle and pedestrians trajectory datasets, ECCO attains competitive accuracy with signiﬁcantly fewer parameters.
Revised: On two real-world vehicle and pedestrians trajectory datasets, ECCO attains competitive accuracy with signiﬁcantly fewer parameters.

Original: It is also more sample efﬁcient; generalizing automatically from few data points in any orientation.
Revised: It is also more sample efﬁcient; generalizing automatically from few data points in any orientation.

Original: Lastly, equivariance gives ECCO improved generalization performance.
Revised: Lastly, equivariance gives ECCO improved generalization performance.

Original: Our method provides a fresh perspective towards increasing trust in deep learning models through guaranteed properties.
Revised: Our method provides a fresh perspective towards increasing trust in deep learning models through guaranteed properties.

Original: Future directions include applying equivariance to problem of predicting a distribution of many possible trajectories
Revised: Future directions include applying equivariance to problem of predicting a distribution of many possible trajectories, or developing a faster version of ECCO which does not require autoregressive computation

Original: A A PPENDIX A. C ONTINUOUS CONVOLUTION INVOLVING ρ reg
Revised: A A PPENDIX A. C ONTINUOUS CONVOLUTION INVOLVING ρ reg

Original: This section is a more detailed version of Section 4.4.
Revised: This section is a more detailed version of Section 4.4.

Original: Deﬁne the input f to be ρ reg -ﬁeld, that is, a distribution over R 2 valued in ρ reg .
Revised: Deﬁne the input f to be ρ reg -ﬁeld, that is, a distribution over R 2 valued in ρ reg .

Original: Deﬁne K : R 2 → ρ reg ⊗ ρ reg .
Revised: Deﬁne K : R 2 → ρ reg ⊗ ρ reg .

Original: After identifying SO(2) with its underlying manifold S 1 , we can identify K ( x ) as a map S 1 × S 1 → R and f ( x ) : S 1 → R .
Revised: After identifying SO(2) with its underlying manifold S 1 , we can identify K ( x ) as a map S 1 × S 1 → R and f ( x ) : S 1 → R .

Original: Deﬁne the integral transform
Revised: Deﬁne the integral transform

Original: [Equation]
Revised: [Equation]

Original: For y ∈ R 2 , deﬁne the convolution g = K (cid:63) f by
Revised: For y ∈ R 2 , deﬁne the convolution g = K (cid:63) f by

Original: [Equation]
Revised: [Equation]

Original: The (cid:125) -operation parameterizes linear maps ρ reg → ρ reg and is thus analogous to matrix multiplication.
Revised: The (cid:125) -operation parameterizes linear maps ρ reg → ρ reg and is thus analogous to matrix multiplication.

Original: If we chose to restrict our choice of κ to κ ( φ 2 , φ 1 )
Revised: If we chose to restrict our choice of κ to κ ( φ 2 , φ 1 )

Original: =
Revised: =

Original: ˜ κ
Revised: ˜ κ

Original: ( φ 2 − φ 1 ) for some function ˜ κ : S 

Revised: ( φ 2 − φ 1 ) for some function ˜ κ : S

Original: → R then this becomes the circular convolution operation.
Revised: → R then this becomes the circular convolution operation.

Original: The SO (2) -action on ρ reg by Rot θ ( f )( φ ) = f
Revised: The SO (2) -action on ρ reg by Rot θ ( f )( φ ) = f

Original: ( φ − θ ) induces an action on κ : S 1 × S 

Revised: ( φ − θ ) induces an action on κ : S 1 × S

Original: → R by
Revised: → R by

Original: [Equation]
Revised: [Equation]

Original: This, in turn, gives an action on the torus-ﬁeld K by
Revised: This, in turn, gives an action on the torus-ﬁeld K by

Original: [Equation]
Revised: [Equation]

Original: Thus Equation 3, the convolutional kernel constraint, implies that K is equivariant if and only if
Revised: Thus Equation 3, the convolutional kernel constraint, implies that K is equivariant if and only if

Original: [Equation]
Revised: [Equation]

Original: θ, φ 1 − θ ) .
Revised: θ, φ 1 − θ ) .

Original: We use this to deﬁne a weight sharing scheme as described in Section 3.2.
Revised: We use this to deﬁne a weight sharing scheme as described in Section 3.2.

Original: The cases of continuous convolution ρ 1 → ρ reg and ρ reg → ρ 1 may be derived similarly.
Revised: The cases of continuous convolution ρ 1 → ρ reg and ρ reg → ρ 1 may be derived similarly.

Original: A. E QUIVARIANT P ER -P ARTICLE L INEAR L AYERS
Revised: A. E QUIVARIANT P ER -P ARTICLE L INEAR L AYERS

Original: Since this operation is pointwise, unlike positive radius continuous convolution, we cannot map between different irreducible representations of SO(2) .
Revised: Since this operation is pointwise, unlike positive radius continuous convolution, we cannot map between different irreducible representations of SO(2) .

Original: Consider as input a ρ in -ﬁeld I and output a ρ out -ﬁeld O where ρ in and ρ out are ﬁnite-dimensional representations of SO(2) .
Revised: Consider as input a ρ in -ﬁeld I and output a ρ out -ﬁeld O where ρ in and ρ out are ﬁnite-dimensional representations of SO(2) .

Original: We deﬁne O ( i ) =
Revised: We deﬁne O ( i ) =

Original: WI ( i ) using the same W , an equivariant linear map, for each particle 1 ≤ i ≤ N .
Revised: WI ( i ) using the same W , an equivariant linear map, for each particle 1 ≤ i ≤ N .

Original: Denote the decomposition of ρ in and ρ out into irreducible representations of SO (2) as ρ in ∼ =
Revised: Denote the decomposition of ρ in and ρ out into irreducible representations of SO (2) as ρ in ∼ =

Original: ρ
Revised: ρ

Original: i 1 1 ⊕ . . .
Revised: i 1 1 ⊕ . . .

Original: ⊕ ρ i n n and ρ out ∼ = ρ j 1 1 ⊕ . . .
Revised: ⊕ ρ i n n and ρ out ∼ = ρ j 1 1 ⊕ . . .

Original: ⊕ ρ j n n respectively.
Revised: ⊕ ρ j n n respectively.

Original: By Schur’s lemma, the equivariant linear map W : ρ in → ρ out is deﬁned by a block diagonal matrix with blocks { W k } nk =1 where W k is an i k × j k matrix.
Revised: By Schur’s lemma, the equivariant linear map W : ρ in → ρ out is deﬁned by a block diagonal matrix with blocks { W k } nk =1 where W k is an i k × j k matrix.

Original: That is, maps between different irreducible representations are zero and each map
Revised: That is, maps between different irreducible representations are zero and each map

Original: ρ k → ρ k is given by a single scalar.
Revised: ρ k → ρ k is given by a single scalar.

Original: Per-particle linear mapping ρ 1 → ρ reg and ρ 1 → ρ reg .
Revised: Per-particle linear mapping ρ 1 → ρ reg and ρ 1 → ρ reg .

Original: Since the input and output features are ρ 1 -ﬁelds, but the hidden features may be represented by ρ reg , we need mappings between ρ 1 and ρ reg .
Revised: Since the input and output features are ρ 1 -ﬁelds, but the hidden features may be represented by ρ reg , we need mappings between ρ 1 and ρ reg .

Original: In all cases we pair continuous convolutions with dense per-particle mappings, this we must describe per-particle mappings between ρ 1 and ρ reg .
Revised: In all cases we pair continuous convolutions with dense per-particle mappings, this we must describe per-particle mappings between ρ 1 and ρ reg .

Original: By the Peter-Weyl theorem, L 2 (SO(2)) ∼ = (cid:76) ∞ i =

Revised: By the Peter-Weyl theorem, L 2 (SO(2)) ∼ = (cid:76) ∞ i =

Original: ρ i .
Revised: ρ i .

Original: In the case of SO(2) , this decomposition is also called the Fourier decomposition or decomposition into circular harmonics.
Revised: In the case of SO(2) , this decomposition is also called the Fourier decomposition or decomposition into circular harmonics.

Original: Most importantly, there is one copy of ρ 1 inside of L 2 (SO(2)) .
Revised: Most importantly, there is one copy of ρ 1 inside of L 2 (SO(2)) .

Original: Hence, up to scalar, there is a unique linear map i 1 : ρ 1 → L 2 (SO(2)) given by ( a, b ) (cid:55)→ a cos( θ ) + b sin( θ ) .
Revised: Hence, up to scalar, there is a unique linear map i 1 : ρ 1 → L 2 (SO(2)) given by ( a, b ) (cid:55)→ a cos( θ ) + b sin( θ ) .

Original: The reverse mapping pr 1 : L 2 (SO(2))
Revised: The reverse mapping pr 1 : L 2 (SO(2))

Original: → ρ 1 is projection onto the ρ 1 summand and is given by the
Revised: → ρ 1 is projection onto the ρ 1 summand and is given by the

Original: Fourier transform pr i ( f ) =
Revised: Fourier transform pr i ( f ) =

Original: ( (cid:82) S 1 f ( θ ) cos( θ ) dθ, (cid:82) S 1 f ( θ ) sin( θ ) dθ ) .
Revised: ( (cid:82) S 1 f ( θ ) cos( θ ) dθ, (cid:82) S 1 f ( θ ) sin( θ ) dθ ) .

Original: Per-particle linear mapping ρ reg → ρ reg .
Revised: Per-particle linear mapping ρ reg → ρ reg .

Original: Though ρ reg is not ﬁnite-dimensional, the fact that it decomposes into a direct sum of irreducible representations means that we may take ρ in
Revised: Though ρ reg is not ﬁnite-dimensional, the fact that it decomposes into a direct sum of irreducible representations means that we may take ρ in

Original: = ρ out = ρ reg above.
Revised: = ρ out = ρ reg above.

Original: Practically, however, it is easier to realize the linear equivariant map ρ i reg → ρ j reg as a convolution over S 1 ,
Revised: Practically, however, it is easier to realize the linear equivariant map ρ i reg → ρ j reg as a convolution over S 1 ,

Original: [Equation]
Revised: [Equation]

Original: where κ ( θ ) is an i × j matrix of trainable weights, independent for each θ .
Revised: where κ ( θ ) is an i × j matrix of trainable weights, independent for each θ .

Original: A. E NCODING I NDIVIDUAL P ARTICLE P AST B EHAVIOR
Revised: A. E NCODING I NDIVIDUAL P ARTICLE P AST B EHAVIOR

Original: We can encode these individual attributes using a per vehicle LSTM (Hochreiter & Schmidhuber, 1997).
Revised: We can encode these individual attributes using a per vehicle LSTM (Hochreiter & Schmidhuber, 1997).

Original: Let X ( i ) t denote the position of car i at time t .
Revised: Let X ( i ) t denote the position of car i at time t .

Original: Denote a fully connected LSTM cell by h t , c t = LSTM ( X ( i ) t , h t − 1 , c t − 1 ) .
Revised: Denote a fully connected LSTM cell by h t , c t = LSTM ( X ( i ) t , h t − 1 , c t − 1 ) .

Original: Deﬁne h 

Revised: Deﬁne h

Original: = c 0 = 0 .
Revised: = c 0 = 0 .

Original: We then use the concatenation of the hidden states [ h (1) t in . . .
Revised: We then use the concatenation of the hidden states [ h (1) t in . . .

Original: h ( n ) t in ] of all particles as Z ∈ R N ⊗ R k as the encoded per-vehicle latent features.
Revised: h ( n ) t in ] of all particles as Z ∈ R N ⊗ R k as the encoded per-vehicle latent features.

Original: A. E NCODING P AST I NTERACTIONS
Revised: A. E NCODING P AST I NTERACTIONS

Original: In addition, we also encode past interactions of particles by introducing a continuous convolution
Revised: In addition, we also encode past interactions of particles by introducing a continuous convolution

Original: LSTM .
Revised: LSTM .

Original: Similar to convLSTM we replace the fully connected layers of the original LSTM above with another operation Xingjian et al.
Revised: Similar to convLSTM we replace the fully connected layers of the original LSTM above with another operation Xingjian et al.

Original: While convLSTM is well-suited for capturing spatially local interactions over time, it requires gridded information.
Revised: While convLSTM is well-suited for capturing spatially local interactions over time, it requires gridded information.

Original: Since the particle system we consider are distributed in continuous space, we replace the standard convolution with rotation-equivariant continuous convolutions.
Revised: Since the particle system we consider are distributed in continuous space, we replace the standard convolution with rotation-equivariant continuous convolutions.

Original: We can now deﬁne H t , C t =
Revised: We can now deﬁne H t , C t =

Original: CtsConvLSTM ( X t , H t − 1 , C t − 1 ) which is an LSTM cell using equivariant continuous convolutions throughout.
Revised: CtsConvLSTM ( X t , H t − 1 , C t − 1 ) which is an LSTM cell using equivariant continuous convolutions throughout.

Original: Note that in this case X t , H t − 1 , C t − 1 are all particle feature ﬁelds, that is, functions { 1 , . . . , n } → R k .
Revised: Note that in this case X t , H t − 1 , C t − 1 are all particle feature ﬁelds, that is, functions { 1 , . . . , n } → R k .

Original: Deﬁne CtsConvLSTM by
Revised: Deﬁne CtsConvLSTM by

Original: [Equation]
Revised: [Equation]

Original: where (cid:63) cts denotes CtsConv .
Revised: where (cid:63) cts denotes CtsConv .

Original: We then can use H t in as input feature for the prediction network.
Revised: We then can use H t in as input feature for the prediction network.

Original: A. E QUIVARIANCE E RROR
Revised: A. E QUIVARIANCE E RROR

Original: We prove the proposition in Section 4.5.
Revised: We prove the proposition in Section 4.5.

Original: Proposition.
Revised: Proposition.

Original: Let α = 2 π/k θ .
Revised: Let α = 2 π/k θ .

Original: Let ¯ θ be θ rounded to nearest value in Z α .
Revised: Let ¯ θ be θ rounded to nearest value in Z α .

Original: Set ˆ θ = | θ − ¯ θ | .
Revised: Set ˆ θ = | θ − ¯ θ | .

Original: Assume n particles samples uniformly in a ball of radius R with features f ∈ ρ c 1 .
Revised: Assume n particles samples uniformly in a ball of radius R with features f ∈ ρ c 1 .

Original: Let f and K have entries sampled uniformly in [ − a, a ] .
Revised: Let f and K have entries sampled uniformly in [ − a, a ] .

Original: Let the bullseye have radius 0 < R e < R .
Revised: Let the bullseye have radius 0 < R e < R .

Original: Let F = CtsConv K,R and T θ = ρ 1 (Rot θ ) .
Revised: Let F = CtsConv K,R and T θ = ρ 1 (Rot θ ) .

Original: Then the expected EE is bounded
Revised: Then the expected EE is bounded

Original: [Equation]
Revised: [Equation]

Original: where C = 4 cna 2 (1 − R 2 e /R 2 ) .
Revised: where C = 4 cna 2 (1 − R 2 e /R 2 ) .

Original: Proof.
Revised: Proof.

Original: We may compute for a single particle x = ( ψ, r ) and multiply our result by n by linearity.
Revised: We may compute for a single particle x = ( ψ, r ) and multiply our result by n by linearity.

Original: We separate two cases: x in bullseye with probability R 2 e /R 2 and x in angular slice with probability 1 − R 2 e /R 2 .
Revised: We separate two cases: x in bullseye with probability R 2 e /R 2 and x in angular slice with probability 1 − R 2 e /R 2 .

Original: If x is in the bullseye, then there is no equivariance error since K ( x ) is a scalar matrix.
Revised: If x is in the bullseye, then there is no equivariance error since K ( x ) is a scalar matrix.

Original: Assume x is an angular sector.
Revised: Assume x is an angular sector.

Original: For nearest interpolation, the equivariance error is then
Revised: For nearest interpolation, the equivariance error is then

Original: [Equation]
Revised: [Equation]

Original: Since ρ 1 ( θ ) is length preserving, this is
Revised: Since ρ 1 ( θ ) is length preserving, this is

Original: [Equation]
Revised: [Equation]

Original: where β
Revised: where β

Original: = ±
Revised: = ±

Original: ˆ θ .
Revised: ˆ θ .

Original: We consider only a single factor of ρ 1 in f .
Revised: We consider only a single factor of ρ 1 in f .

Original: The result will then be multiplied by c .
Revised: The result will then be multiplied by c .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: We can factor out an a from K ( x ) and an a from f and assume k ij , f i samples from
Revised: We can factor out an a from K ( x ) and an a from f and assume k ij , f i samples from

Original: Uniform([ − 1 , 1]) .
Revised: Uniform([ − 1 , 1]) .

Original: One may then directly compute that Equation 6 equals
Revised: One may then directly compute that Equation 6 equals

Original: [Equation]
Revised: [Equation]

Original: This is bounded above by 4 | sin( β ) | = 4 | sin(ˆ θ ) | .
Revised: This is bounded above by 4 | sin( β ) | = 4 | sin(ˆ θ ) | .

Original: Collecting the above factors, this proves the bound
Revised: Collecting the above factors, this proves the bound

Original: C | sin( β ) | .
Revised: C | sin( β ) | .

Original: The further bound follows by the ﬁrst order bound,
Revised: The further bound follows by the ﬁrst order bound,

Original: [Equation]
Revised: [Equation]

Original: The relationship EE ≈ 2 πC/k θ is visible in Figure 4.
Revised: The relationship EE ≈ 2 πC/k θ is visible in Figure 4.

Original: We can also see clearly the signiﬁcance of the term | sin(ˆ θ ) | by plotting equivariance error against θ as in Figure 7.
Revised: We can also see clearly the signiﬁcance of the term | sin(ˆ θ ) | by plotting equivariance error against θ as in Figure 7.

Original: [Figure]
Revised: [Figure]

Original: Argoverse dataset includes 324K samples, which are split into 206K training data, 39K validation and 78K test set.
Revised: Argoverse dataset includes 324K samples, which are split into 206K training data, 39K validation and 78K test set.

Original: All the samples are real data extracted from Miami and Seattle, and the dataset provides HD maps of lanes in each city.
Revised: All the samples are real data extracted from Miami and Seattle, and the dataset provides HD maps of lanes in each city.

Original: Every sample contains data for 5 seconds long, and is sampled in 10Hz frequency.
Revised: Every sample contains data for 5 seconds long, and is sampled in 10Hz frequency.

Original: TrajNet++ Real dataset contains 200K samples.
Revised: TrajNet++ Real dataset contains 200K samples.

Original: All the tracking in this dataset is captured in both indoor and outdoor locations, for example, university, hotel, Zara, and train stations.
Revised: All the tracking in this dataset is captured in both indoor and outdoor locations, for example, university, hotel, Zara, and train stations.

Original: Every sample in this dataset contains 21 timestamps, and the goal is to predict the 2D spatial positions for each pedestrain in the future 12 timestamps.
Revised: Every sample in this dataset contains 21 timestamps, and the goal is to predict the 2D spatial positions for each pedestrain in the future 12 timestamps.

Original: A. I MPLEMENTATION D ETAILS
Revised: A. I MPLEMENTATION D ETAILS

Original: Argoverse dataset is not fully observed, so we only use cars with complete observation as our input.
Revised: Argoverse dataset is not fully observed, so we only use cars with complete observation as our input.

Original: Since every sample doesn’t include the same number of cars, we only choose those scenes with less than or equal to 60 cars and insert dummy cars into them to achieve consistent car numbers.
Revised: Since every sample doesn’t include the same number of cars, we only choose those scenes with less than or equal to 60 cars and insert dummy cars into them to achieve consistent car numbers.

Original: TrajNet++ Real dataset is also not fully observed.
Revised: TrajNet++ Real dataset is also not fully observed.

Original: And here we keep our pedestrain number consistent to 160.
Revised: And here we keep our pedestrain number consistent to 160.

Original: Moreover, for each car, we use the average velocity in the past 0.1 second as an approximate to the current instant velocity, i.e. v t
Revised: Moreover, for each car, we use the average velocity in the past 0.1 second as an approximate to the current instant velocity, i.e. v t

Original: =
Revised: =

Original: ( p t − p t − 1 ) / 2 .
Revised: ( p t − p t − 1 ) / 2 .

Original: As for map information, we only include center lanes with lane directions as features.
Revised: As for map information, we only include center lanes with lane directions as features.

Original: Also, we introduce dummy lane node into each scene to make lane numbers consistently equal to 650.
Revised: Also, we introduce dummy lane node into each scene to make lane numbers consistently equal to 650.

Original: In TrajNet++ task, no map information is included.
Revised: In TrajNet++ task, no map information is included.

Original: And since pedestrians don’t have a speedometers to tell them exactly how fast they are moving as drivers, instead they depends more on the relative velocities and relative positions to other pedestrians, we tried different combination of features in ablative study besides only using history velocities.
Revised: And since pedestrians don’t have a speedometers to tell them exactly how fast they are moving as drivers, instead they depends more on the relative velocities and relative positions to other pedestrians, we tried different combination of features in ablative study besides only using history velocities.

Original: Our models are all trained by Adam optimizer with base learning rate 0.001, and the gamma rate for linear rate scheduler is set to be 0.95.
Revised: Our models are all trained by Adam optimizer with base learning rate 0.001, and the gamma rate for linear rate scheduler is set to be 0.95.

Original: All our models without map information are trained for 15K iterations with batch size 16 and learning rate is updated every 300 iterations; for models with map information, we train them for 30K iterations with batch size 16 and learning rate is updated every 600 iterations.
Revised: All our models without map information are trained for 15K iterations with batch size 16 and learning rate is updated every 300 iterations; for models with map information, we train them for 30K iterations with batch size 16 and learning rate is updated every 600 iterations.

Original: For CtsConv, we set the layer sizes to be 32, 64, 64, 64, and kernel size 4 × 4 × 4 ; for ρ 1 - ECCO , the layer sizes are 16, 32, 32, 32, k θ is 16, k r is 3; for ρ reg - ECCO , we choose layer size 8, 16, 8, 8, k θ 16, k r 3, and regular feature dimension is set to be 8.
Revised: For CtsConv, we set the layer sizes to be 32, 64, 64, 64, and kernel size 4 × 4 × 4 ; for ρ 1 - ECCO , the layer sizes are 16, 32, 32, 32, k θ is 16, k r is 3; for ρ reg - ECCO , we choose layer size 8, 16, 8, 8, k θ 16, k r 3, and regular feature dimension is set to be 8.

Original: For Argoverse task, we set the CtsConv radius to be 40, and for TrajNet++ task we set it to be 6.
Revised: For Argoverse task, we set the CtsConv radius to be 40, and for TrajNet++ task we set it to be 6.

Original: A. A BLATIVE S TUDY
Revised: A. A BLATIVE S TUDY

Original: We perform ablative study for ECCO to further diagnose different encoders, usage of HD maps and other model design choices.
Revised: We perform ablative study for ECCO to further diagnose different encoders, usage of HD maps and other model design choices.

Original: Choice of encoders Unlike ﬂuid simulations (Ummenhofer et al., 2019) where the dynamics are
Revised: Choice of encoders Unlike ﬂuid simulations (Ummenhofer et al., 2019) where the dynamics are

Original: Markovian, human behavior exhibit long-term dependency.
Revised: Markovian, human behavior exhibit long-term dependency.

Original: We experiment with three different encoders refered to as Enc to model such long-term dependency: (1) concatenating the velocities from the past m frames as input feature
Revised: We experiment with three different encoders refered to as Enc to model such long-term dependency: (1) concatenating the velocities from the past m frames as input feature

Original: [Table]
Revised: [Table]

Original: (2) passing the past velocities of each particle to the same LSTM to encode individual behavior of each particle
Revised: (2) passing the past velocities of each particle to the same LSTM to encode individual behavior of each particle

Original: [Table]
Revised: [Table]

Original: and (3) implementing continuous convolution LSTM to encode past particle interactions.
Revised: and (3) implementing continuous convolution LSTM to encode past particle interactions.

Original: Our continuous convolution LSTM is similar to convLSTM (Xingjian et al., 2015) but uses continuous convolutions instead of discrete gridded convolutions.
Revised: Our continuous convolution LSTM is similar to convLSTM (Xingjian et al., 2015) but uses continuous convolutions instead of discrete gridded convolutions.

Original: We use different encoders to time-aggregate features and compare their performances (Table 3)
Revised: We use different encoders to time-aggregate features and compare their performances (Table 3)

Original: [Table]
Revised: [Table]

Original: Use of HD Maps In Table 4, we compare performance with and without map input features.
Revised: Use of HD Maps In Table 4, we compare performance with and without map input features.

Original: [Table]
Revised: [Table]

Original: Choice of features for pedestrian Unlike vehicles, people do not have a velocity meter to tell him how fast they actually walk.
Revised: Choice of features for pedestrian Unlike vehicles, people do not have a velocity meter to tell him how fast they actually walk.

Original: We realize that people actually tend to adjust their velocities based on others’ relative velocity and relative position.
Revised: We realize that people actually tend to adjust their velocities based on others’ relative velocity and relative position.

Original: We experiment different combination of features (Table 5), ﬁnding using relative velocities and relative positions as feature has the best performance.
Revised: We experiment different combination of features (Table 5), ﬁnding using relative velocities and relative positions as feature has the best performance.

Original: [Table]
Revised: [Table]

Original: A. Q UALITATIVE R ESULTS FOR T RAJ N ET ++
Revised: A. Q UALITATIVE R ESULTS FOR T RAJ N ET ++

Original: Figure 8 show qualitative results for TrajNet++.
Revised: Figure 8 show qualitative results for TrajNet++.

Original: Note that the non-equivariant baseline (2nd column) depends highly on the global orientation whereas the ground truth and equivariant models do not.
Revised: Note that the non-equivariant baseline (2nd column) depends highly on the global orientation whereas the ground truth and equivariant models do not.