Original: I NTERACTIVE S EQUENTIAL G ENERATIVE M ODELS
Revised: I NTERACTIVE S EQUENTIAL G ENERATIVE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Understanding spatiotemporal relationships among several agents is of considerable relevance for many domains.
Revised: Understanding spatiotemporal relationships among several agents is of considerable relevance for many domains.

Original: Team sports represent a particularly interesting real-world proving ground since modeling interacting athletes requires capturing highly dynamic and complex agent-agent dependencies in addition to temporal components.
Revised: Team sports represent a particularly interesting real-world proving ground since modeling interacting athletes requires capturing highly dynamic and complex agent-agent dependencies in addition to temporal components.

Original: However, existing generative methods in this ﬁeld either entangle all latent factors into a single variable and are thus constrained in practical applicability, or they focus on uncovering interaction structures, which restricts their generative ability.
Revised: However, existing generative methods in this ﬁeld either entangle all latent factors into a single variable and are thus constrained in practical applicability, or they focus on uncovering interaction structures, which restricts their generative ability.

Original: To address this gap, we propose a framework for multiagent trajectories that augments  sequential generative models with  latent social structures.
Revised: To address this gap, we propose a framework for multiagent trajectories that augments graph-structured sequential generative models with explicit latent social dependencies.

Original: First, we derive a novel objective via approximate inference using a disentangled latent space that accurately describes the data generating process in such sequential systems.
Revised: First, we derive a novel objective within the variational autoencoder family using a disentangled latent space that aims to encapsulate inherent data traits.

Original: Based on the proposed training criterion, we then present a model architecture that uniﬁes insights from neural interaction inference and graph-structured variational recurrent neural networks for generating collective movements while allocating latent information.
Revised: Based on the proposed training criterion, we then present a model architecture that uniﬁes insights from neural interaction inference and graph-structured variational recurrent neural networks for generating collective movements while allocating latent information.

Original: We validate our model on data from professional soccer and basketball.
Revised: We validate our model on data from professional soccer and basketball.

Original: Our framework not only improves upon existing state-of-the-art approaches in forecasting trajectories, but also infers semantically meaningful representations that can be used in downstream tasks.
Revised: Our framework not only improves upon existing state-of-the-art approaches in forecasting trajectories, but also infers semantically meaningful representations that can be used in downstream tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The study of agent behavior governed by temporal and spatial dependencies is of great importance in many different ﬁelds, such as autonomous driving (Brown et al., 2020; Rasouli & Tsotsos, 2019), robot navigation (Rudenko et al., 2020), or sports analytics (Tuyls et al., 2021).
Revised: The study of agent behavior governed by temporal and spatial dependencies is of great importance in many different ﬁelds, such as autonomous driving (Brown et al., 2020; Rasouli & Tsotsos, 2019), robot navigation (Rudenko et al., 2020), or sports analytics (Tuyls et al., 2021).

Original: In particular, accurate detection of implicit causal social structures offers several advantages by removing confounding factors for trajectory forecasting tasks and providing practitioners with interpretable dynamics that can in turn be integrated into downstream decision-making processes or applications.
Revised: In particular, accurate detection of implicit causal social structures offers several advantages by removing confounding factors for trajectory forecasting tasks and providing practitioners with interpretable dynamics that can in turn be integrated into downstream decision-making processes or applications.

Original: Modeling the dynamics of multiplayer sports games (Omidshaﬁei et al., 2022; Le et al., 2017;
Revised: Modeling the dynamics of multiplayer sports games (Omidshaﬁei et al., 2022; Le et al., 2017;

Original: Yue et al., 2014; Liu et al., 2020) is particularly challenging since accurate trajectory generation in this environment requires capturing highly dynamic and complex underlying modular structures (Makansi et al., 2022).
Revised: Yue et al., 2014; Liu et al., 2020) is particularly challenging since accurate trajectory generation in this environment requires capturing highly dynamic and complex underlying modular structures (Makansi et al., 2022).

Original: For example, the roles prescribed in a team formation are a poor indicator of the actual behavior observed in a given situation.
Revised: For example, the roles prescribed in a team formation are a poor indicator of the actual behavior observed in a given situation.

Original: Moreover, most of the interacting elements inject noise into the forecasting process because they are either irrelevant (e.g., goal keepers) or their inﬂuential nature changes as the situation evolves.
Revised: Moreover, most of the interacting elements inject noise into the forecasting process because they are either irrelevant (e.g., goal keepers) or their inﬂuential nature changes as the situation evolves.

Original: However, existing methods for modeling sports data rely on graph encoding strategies (Kipf & Welling, 2016; Vaswani et al., 2017) that aggregate social information into only single variables that need to capture all latent stochasticity (Zhan et al., 2018; Yeh et al., 2019; Sun et al., 2019; Omidshaﬁei et al., 2022).
Revised: However, existing methods for modeling sports data rely on graph encoding strategies (Kipf & Welling, 2016; Vaswani et al., 2017) that aggregate social information into only single variables that need to capture all latent stochasticity (Zhan et al., 2019; Yeh et al., 2019; Sun et al., 2019; Omidshaﬁei et al., 2022).

Original: In recent years, a considerable amount of methods have been proposed that aim to infer interactive components in general multiagent systems via discrete latent variables.
Revised: In recent years, a considerable amount of methods have been proposed that aim to infer interactive components in general multiagent systems via discrete latent variables.

Original: These methods are usually formulated as some form of variational autoencoder (Kingma & Welling, 2013;
Revised: These methods are usually formulated as some form of variational autoencoder (Kingma & Welling, 2013;

Original: Sohn et al., 2015) that learns latent edge categories of an assumed underlying graph structure (Kipf et al., 2018;
Revised: Sohn et al., 2015) that learns latent edge categories of an assumed underlying graph structure (Kipf et al., 2018;

Original: Graber & Schwing, 2020; L¨owe et al., 2022).
Revised: Graber & Schwing, 2020; L¨owe et al., 2022).

Original: However, being the only causal factors speciﬁed, the proposed frameworks neglect other potential latent characteristics not originating in mere interactive categories but equally affecting multimodal agent behavior, such as agent intents.
Revised: However, being the only causal factors speciﬁed, the proposed frameworks neglect other potential latent characteristics not originating in mere interactive categories but equally affecting multimodal agent behavior, which limits their generative capacity.

Original: To address previous shortcomings, we propose a novel framework for modeling multiagent trajectory data that enhances existing graph-structured latent variable models by explicitly encoding social structures in sports games.
Revised: To address previous shortcomings, we propose a novel framework for modeling multiagent trajectory data that enhances existing graph-structured latent variable models by explicitly encoding social structures in sports games.

Original: Since the contemplated spatiotemporal systems are caused by dynamic dependencies among heterogeneous agents, we deﬁne this component as a causal graph comprising categorical agent roles and pairwise interactions.
Revised: Since the contemplated spatiotemporal systems are caused by dynamic dependencies among heterogeneous agents, we deﬁne this component as a causal graph comprising categorical agent roles and pairwise interactions.

Original: Based on the speciﬁed generative setting, we then introduce an objective function within the variational autoencoder family and instantiate a concrete architecture for computing the derived training components.
Revised: Based on the speciﬁed generative setting, we then introduce an objective function within the variational autoencoder family and instantiate a concrete architecture for computing the derived training components.

Original: Empirically, our model exceeds existing state-of-the-art methods in forecasting trajectories on data from professional soccer and basketball.
Revised: Empirically, our model exceeds existing state-of-the-art methods in forecasting trajectories on data from professional soccer and basketball.

Original: In addition, we report on extensive quantitative and qualitative analyses wrt.
Revised: In addition, we report on extensive quantitative and qualitative analyses wrt.

Original: the learned latent variables that show informative properties in generative tasks and downstream applications.
Revised: the learned latent variables that show informative properties in generative tasks and downstream applications.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Given data D = { x  ≤ T } Ni =1 consisting of N sequences x ≤ T = { x 1 , ..., x T } , our goal is to estimate the underying data distribution via maximizing the likelihood of the collected evidence, i.e,.
Revised: Given data D = { x ( i ) ≤ T } Ni =1 consisting of N sequences x ≤ T = [ x 1 , ..., x T ] , our goal is to estimate the underying data distribution via maximizing the likelihood of the collected evidence, i.e,.

Original: max p θ ( x ≤ T ) .
Revised: max p θ ( x ≤ T ) .

Original: In practice, p θ ( x ≤ T ) is often highly multimodal, which complicates direct deployment of MLE.
Revised: In practice, p θ ( x ≤ T ) is often highly multimodal, which complicates direct deployment of MLE.

Original: A frequently used modeling paradigm for stochasticity in complex multimodal distributions is introducing latent variables and optimizing the variational lower bound on the maginal log-likelihood (Kingma & Welling, 2013; Rezende et al., 2014; Sohn et al., 2015).
Revised: A frequently used modeling paradigm for stochasticity in complex multimodal distributions is introducing latent variables and optimizing the variational lower bound on the maginal log-likelihood (Kingma & Welling, 2013; Rezende et al., 2014; Sohn et al., 2015).

Original: Existing conditional variational models for generating highly-structured sequential data x ≤ T usually associate a latent variable z 1 , ..., z T with each timestep of the segment to describe the generative process (Bayer & Osendorfer, 2014;
Revised: Existing conditional variational models for generating highly-structured sequential data x ≤ T usually associate a latent variable z 1 , ..., z T with each timestep of the segment to describe the generative process (Bayer & Osendorfer, 2014;

Original: Goyal et al., 2017; Fraccaro et al., 2016).
Revised: Goyal et al., 2017; Fraccaro et al., 2016).

Original: The variational RNN ( VRNN , (Chung et al., 2015)) is one renowned instantiation in this domain that, assuming speciﬁc dependency structures in the generative and inference parts, arrives at the following lower bound on log p θ ( x ≤ T ) :
Revised: The variational RNN ( VRNN , (Chung et al., 2015)) is one renowned instantiation in this domain that, assuming speciﬁc dependency structures in the generative and inference parts, arrives at the following lower bound on log p θ ( x ≤ T ) :

Original: [Equation]
Revised: [Equation]

Original: where the joint distribution p θ ( x <t , z <t) is captured via a recurrent neural network h t = f RNN ( x t , z t , h t − 1 ) .
Revised: where information x <t , z <t is captured via a recurrent neural network h t = f RNN ( x t , z t , h t − 1 ) .

Original: Given the temporal and multimodal notion of human movement, sequential generative models constitute a good starting point for designing a framework tailored to multiagent trajectories.
Revised: Given the temporal and multimodal notion of human movement, sequential generative models constitute a good starting point for designing a framework tailored to multiagent trajectories.

Original: However, such approaches only account for the temporal aspect of the problem, but neglect potential social dependencies at each timestep.
Revised: However, such approaches only account for the temporal aspect of the problem, but neglect potential social dependencies at each timestep.

Original: Adding Graph Structure As a remedy, sequential data can be augmented by a social dimension x ≤ T =
Revised: Adding Graph Structure As a remedy, sequential data can be augmented by a social dimension x ≤ T =

Original: { x ( a ) ≤ T , ∀ a ∈ A} , where x ( a ) t ∈ R d denotes a d -dimensional feature representation of agent a ∈ A at time t (e.g., its 2D position).
Revised: { x ( a ) ≤ T , ∀ a ∈ A} , where x ( a ) t ∈ R d denotes a d -dimensional feature representation of agent a ∈ A at time t (e.g., its 2D position).

Original: Permutation invariant models are a prerequisite for processing sequential sets with potentially divergent cardinality, so a direct adoption of Eq.
Revised: Permutation invariant models are a prerequisite for processing sequential sets with potentially divergent cardinality, so a direct adoption of Eq.

Original: 1 in multiagent settings would implicitly impose the assumption of social independence across agent trajectories.
Revised: 1 in multiagent settings would implicitly impose the assumption of social independence across agent trajectories.

Original: This assumption is trivially inappropriate for interactive systems; thus, related work proposes sensitive solutions - usually in the form of graph encoding strategies - to capture agentagent interactions.
Revised: This assumption is trivially inappropriate for interactive systems; thus, related work proposes sensitive solutions - usually in the form of graph encoding strategies - to capture agentagent interactions.

Original: Yeh et al.
Revised: Yeh et al.

Original: (2019) introduce the graph VRNN ( GVRNN ), which operates within the VRNN framework with graph neural networks (GNNs, Battaglia et al.
Revised: (2019) introduce the graph VRNN ( GVRNN ), which operates within the VRNN framework with graph neural networks (GNNs, Battaglia et al.

Original: representing agents and their interactions as nodes and edges, respectively.
Revised: representing agents and their interactions as nodes and edges, respectively.

Original: More formally, the architecture for computing the components in
Revised: More formally, the architecture for computing the components in

Original: Eq.
Revised: Eq.

Original: 1 amounts to the following structure:
Revised: 1 amounts to the following structure:

Original: [Equation]
Revised: [Equation]

Original: where h t is the set of recurrent agent states h ( a ) t
Revised: where h t is the set of recurrent agent states h ( a ) t

Original: [Equation]
Revised: [Equation]

Original: We emphasize that, although factorized, the latent space is not marginally independent across agents since each z ( a ) t is conditioned on information of all other entities via the (assumed) fully-connected graph.
Revised: We emphasize that, although factorized, the latent space is not marginally independent across agents since each z ( a ) t is conditioned on information of all other entities via the (assumed) fully-connected graph.

Original: However, in many spatiotemporal patterns, most of the observed elements are irrelevant or even distracting, and the speciﬁc composition of relevant factors can change rapidly.
Revised: However, in many spatiotemporal patterns, most of the observed elements are irrelevant or even distracting, and the speciﬁc composition of relevant factors can change rapidly.

Original: Thus, a better strategy is to explicitly detect semantic classes that describe the underlying structural component before aggregating social information intothe entangled variables.
Revised: Thus, a better strategy is to explicitly detect semantic classes that describe the underlying structural component before aggregating social information into entangled variables.

Original:  M ETHOD
Revised:  M ETHOD

Original: In the following we derive a novel training criterion to model the joint distribution of agent trajectories p θ ( x ≤ T ) using latent variables.
Revised: In the following, we propose a novel line of thought on how to model the joint distribution of interactive sequential data p θ ( x ≤ T ) using variational methods.

Original: To address shortcomings of existing methods, our formalization uses disentangled factors of variation that explicitly include a discrete latent subspace describing an assumed social structure inherent in multiagent systems.
Revised: To address shortcomings of existing methods, our formalization deﬁnes disentangled factors of variation that explicitly include a discrete latent subspace describing causal factors that arise from social perspectives.

Original: Having conceptualized our modeling goal, wethen instantiate a concrete architecture  that computes the derived training components. The architecture draws oninsights from neural interaction learning to allocate all latent information into the intended variables.
Revised: Having conceptualized our modeling goal, we instantiate a concrete architecture termed DIA (Detecting Important Agents) that computes the inferred training components via imposing an inductive bias on the latent space.

Original: Appendix B.2 details how the model is trained in an end-to-end fashion.
Revised: Appendix B.2 details how the model is trained in an end-to-end fashion.

Original: 3. D ERIVING THE T RAINING O BJECTIVE
Revised: 3. D ERIVING THE T RAINING O BJECTIVE

Original: Latent Variables
Revised: Latent Variables

Original: A good starting point for conceptualizing appropriate modeling solutions is to concretize the assumptions wrt.

Original: the causal factors underlying our collected data x ≤ T .

Original: In this context, we argued previously that spatiotemporal multiagent data differ from their 1D sequential counterpart solely in its unknown form of relational patterns.

Original: For the multiagent systems contemplated in this work, this interaction form is highly dynamic and varies both in the type of interaction and in the speciﬁc role of the interacting participants.
Revised: The interactive systems contemplated in this work are caused by implicit and dynamic structural dependencies that vary both in the type of interaction as well as in the speciﬁc role of the interacting participants.

Original: For example, in a turnover sequence in soccer, both the role performed (e.g., attacker to defender) and the inﬂuential nature (e.g., on the ball agent from ball carrying to pressure exerting) of each player change within a few moments.
Revised: For example, in a turnover sequence in soccer, both the role performed (e.g., attacker to defender) and the inﬂuential nature (e.g., on the ball agent from ball carrying to pressure exerting) of each player change within a few moments.

Original: To reﬂect such structuresin the model, we augment sequential VAEs by  introducing latent causal graphs G t 
Revised: We propose to explicitly account for such structures by via introducing latent causal graphs G t =

Original: = {K t , E t } , where K t represents the set of discrete agent roles at t (e.g., ball, attacker or defender) {K ( a ) t , a ∈ A} 1 , and E t encodes their (discrete) pairwise strategies in the form of interaction types {E ( a,j ) t , ( a
Revised: {K t , E t } , where K t represents the set of discrete agent roles at t (e.g., ball, attacker or defender) {K ( a ) t , a ∈ A} 1 , and E t encodes their (discrete) pairwise strategies in the form of interaction types {E ( a,j ) t , ( a

Original: , j ) ∈
Revised: , j ) ∈

Original: A × A} (e.g., ball handler, intended pass receiver, etc.).
Revised: A × A} (e.g., ball handler, intended pass receiver, etc.).

Original: To enable the model toselectively exclude potential social confounders, we externally deﬁne an interaction type in E t as class “no interaction” while the remaining categories are learned from the data.
Revised: To enable the model to exclude potential social confounders, we externally deﬁne an interaction type in E t as class “no interaction” while the remaining categories are learned from the data.

Original: Stochastic agent behavior is affected not only by the structural state captured in G t , but also by agent intents, in turn contingent on the inferred communication mechanisms and agent roles.
Revised: However, the structural state captured in G t is not sufﬁciently informative to precisely account for factors that arise from both individual and social perspectives.

Original: Intuitively, each realization z ( a ) t ∈ z t thus encodes latent factors of variation in the trajectories not represented by the inferred causal graphs, yet may vary as a function of G t .Furthermore, it is worth mentioning that via attaching latent variables to individual agents, their respective representative capacity is governed by the cardinality of the agent set. Hence, the complexity of information inherent in larger multi-agent systems can be accounted for in a natural way.
Revised: Therefore, to improve the generative capacity of our model, we adopt latent concepts z t from existing methods described in Sectionthat account for all remaining sources of uncertainty not represented by the inferred causal graphs, yet may vary as a function of G t .

Original: Generative Process We formally express the generative process by incorporating the above description of causal factors into the deﬁnition of our modeling goal, i.e., the marginal likelihood p θ ( x ≤ T ) :
Revised: Generative Process We formally express the generative process by incorporating the above description of causal factors into the deﬁnition of our modeling goal, i.e., the marginal likelihood p θ ( x ≤ T ) :

Original: [Equation]
Revised: [Equation]

Original: where the model likelihood p θ 1 is parameterized using a decoder , and p θ 2 and p θ 3 are the prior distributions over the speciﬁed latent factors.
Revised: where the model likelihood p θ 1 is parameterized using a decoder , and p θ 2 and p θ 3 are the prior distributions over the speciﬁed latent factors.

Original: Inference Model Given the generative model, learning representations of data can be seen as learning a variational approximation of the posterior using an encoder that constructs the distribution of latent values for a given observation x ≤ T .
Revised: Inference Model Given the generative model, learning representations of data can be seen as learning a variational approximation of the posterior using an encoder that constructs the distribution of latent values for a given observation x ≤ T .

Original: We propose the following factorization:
Revised: We propose the following factorization:

Original: [Equation]
Revised: [Equation]

Original: Note that, as opposed to the VRNN objective, we do impose restrictions on this approximation when deﬁning its dependency structure by including also future inputs x t
Revised: Note that, as opposed to the VRNN objective, we do impose restrictions on this approximation when deﬁning its dependency structure by including also future inputs x t

Original: +1: T in the conditions of q φ 1 and q φ 2 .
Revised: +1: T in the conditions of q φ 1 and q φ 2 .

Original: This has been shown to be empirically beneﬁcial in sequential settings (Goyal et al., 2017;
Revised: This has been shown to be empirically beneﬁcial in sequential settings (Goyal et al., 2017;

Original: Fraccaro et al., 2016).
Revised: Fraccaro et al., 2016).

Original: Objective Function We construct an objective function L DIA ( x ≤ T ; φ, θ ) for training the introduced concepts p θ and q φ , by deﬁning a lower-bound on log-likelihood log p θ ( x ≤ T ) :E q φ ( z ≤ T , G ≤ T | x ≤ T ) (cid:34)
Revised: Objective Function We construct an objective function L DIA ( x ≤ T ; φ, θ ) for training the introduced concepts p θ and q φ , by deﬁning a lower-bound on log-likelihood log p θ ( x ≤ T ) :

Original: [Equation]
Revised: [Equation]

Original: (cid:88) t = log p θ 1 ( x t | x <t , z ≤ t , G ≤ t ) p θ 2 ( z t | x <t , z <t , G ≤ t ) p θ 3 ( G t | x <t , z <t , G <t )
Revised: | x <t , z <t , G <t )

Original: q φ 1 ( G t | x ≤ T , z <t , G <t )
Revised: q φ 1 ( G t | x ≤ T , z <t , G <t )

Original: q φ 2 ( z t | x ≤ T , z <t , G ≤ t ) (cid:35) , (5) with q φ 1 ( G t | x ≤ T , z <t , G <t ) =
Revised: q φ 2 ( z t | x ≤ T , z <t , G ≤ t ) (cid:35) , (5) with q φ 1 ( G t | x ≤ T , z <t , G <t ) =

Original: q φ 1 ( K t | x ≤ T , z <t , G <t )
Revised: q φ 1 ( K t | x ≤ T , z <t , G <t )

Original: q φ 1 ( E t | x ≤ T , z <t , G <t ) and p θ 3 ( G t | x <t , z <t , G <t ) =
Revised: q φ 1 ( E t | x ≤ T , z <t , G <t ) and p θ 3 ( G t | x <t , z <t , G <t ) =

Original: p θ 3 ( K t | x <t , z <t , G <t ) p θ 3 ( E t | x <T , z <t , G <t ) .
Revised: p θ 3 ( K t | x <t , z <t , G <t ) p θ 3 ( E t | x <T , z <t , G <t ) .

Original: See Appendix A for a derivation of Eq.
Revised: See Appendix A for a derivation of Eq.

Original: 5 and an elaboration of the relationship to Eq.
Revised: 5 and an elaboration of the relationship to Eq.

Original: 3. A RCHITECTURAL C OMPONENTS
Revised: 3. A RCHITECTURAL C OMPONENTS

Original: Given past realizations x <t , z <t , G <t , the introduced dependency structure in Eq.
Revised: Given past realizations x <t , z <t , G <t , the introduced dependency structure in Eq.

Original: 5 requires to ﬁrst infer the structural component G t before subsequently generating variables z t and x t .
Revised: 5 requires to ﬁrst infer the structural component G t before subsequently generating variables z t and x t .

Original: Hence, we divide the training procedure at each timestep into two distinct phases, with the ﬁrst phase addressing the calculation of components over G t ( p θ 3 and q φ 1 ), while the subsequent second phase infers components over z t and x t ( p θ 1 , p θ 2 and q φ 2 ).
Revised: Hence, we divide the training procedure at each timestep into two distinct phases, with the ﬁrst phase addressing the calculation of components over G t ( p θ 3 and q φ 1 ), while the subsequent second phase infers components over z t and x t ( p θ 1 , p θ 2 and q φ 2 ).

Original: The overall architecture is illustrated in Figure 2.
Revised: The overall architecture is illustrated in Figure 2.

Original: We describe the computations in more detail below.
Revised: We describe the computations in more detail below.

Original: Interaction Graph Learning First, we uncover the (dynamic) latent graph space by constructing the encoder (and prior) modules estimating G t , ∀ t ∈ { 1 , ..., T } .
Revised: Interaction Graph Learning First, we uncover the (dynamic) latent graph space by constructing the encoder (and prior) modules estimating G t , ∀ t ∈ { 1 , ..., T } .

Original: Since we deﬁned latent concepts
Revised: Since we deﬁne latent concepts

Original: G t =
Revised: G t =

Original: {K t , E t } in discrete terms (cf.
Revised: {K t , E t } in discrete terms (cf.

Original: Section 3.1), it is naturalis to model the encoder (and prior) as a multivariate distribution of independent Categoricals:
Revised: Section 3.1), it is natural to model the encoder (and prior) as a multivariate distribution of independent Categoricals:

Original: [Equation]
Revised: [Equation]

Original: where function f enc is an encoder neural net.
Revised: where function f enc is an encoder neural net.

Original: Since the interaction space contains the deﬁned “no interaction” label, the derived probability values over E t in a Bernoulli context can be interpreted as importance values that encode the level of inﬂuence of interacting neighbors on target agents.
Revised: Since the interaction space contains the deﬁned “no interaction” label, the derived probability values over E t in a Bernoulli context can be interpreted as importance values that encode the level of inﬂuence of interacting neighbors on target agents.

Original: While elements in K t are anchored to the elements of the agent set, interactive patterns E t , however, are deﬁned non-symmetrically over all agent pairs.
Revised: While elements in K t are anchored to the elements of the agent set, interactive patterns E t , however, are deﬁned non-symmetrically over all agent pairs.

Original: Thus, we impose a function class on f enc that enables learning single-agent as well as pairwise representations based on which we can classify over both node and relation variables.
Revised: Thus, we impose a function class on f enc that enables learning single-agent as well as pairwise representations based on which we can classify over both node and relation variables.

Original: Inspired by Kipf et al.
Revised: Inspired by Kipf et al.

Original: (2018), we employ an adapted variant of GNNs similar to interaction networks (Gilmer et al., 2017) that operates by learning alternating node and edge representations through an iterative process where the learned edge embeddings are used as messages for neighborhood aggregation 2 .
Revised: (2018), we employ an adapted variant of GNNs similar to interaction networks (Gilmer et al., 2017) that operates by learning alternating node and edge representations through an iterative process where the learned edge embeddings are used as messages for neighborhood aggregation 2 .

Original: We model the input to f enc ( x ≤ T , z <t , G <t ) via multiple GRU cells (Chung et al., 2014) deﬁned over quantities related to the individual agents.
Revised: We model the input to f enc ( x ≤ T , z <t , G <t ) via multiple GRU cells (Chung et al., 2014) deﬁned over quantities related to the individual agents.

Original: More formally, f enc is given by
Revised: More formally, f enc is given by

Original: [Equation]
Revised: [Equation]

Original: (8) where [ · , · ] denotes concatenation, −→ h t − 1 is a set of forward RNN states f RNN ( x ( a ) t , z ( a ) t , −→ h ( a ) t − 1 ) , ←− h t is a set of backward RNN states f RNN ( x ( a ) t , ←− h ( a ) t +1 ) , h intt − 1 is a set of hidden states deﬁned along the edges f RNN ( E ( a,j ) t − 1 , h ( a,j ) ,int t −) of a fully-connected graph structure, and v intenc , e intenc are the node and edge embeddings of the last/output GNN layer, respectively (cf.
Revised: where [ · , · ] denotes concatenation, −→ h t − 1 is a set of forward RNN states f RNN ( x ( a ) t , z ( a ) t , −→ h ( a ) t − 1 ) , ←− h t is a set of backward RNN states f RNN ( x ( a ) t , ←− h ( a ) t +1 ) , h intt − 1 is a set of hidden states deﬁned along the edges f RNN ( E ( a,j ) t − 1 , h ( a,j ) ,int t −) of a fully-connected graph structure, and v intenc , e intenc are the node and edge embeddings of the last/output GNN layer, respectively (cf.

Original: Figure 1, left).
Revised: Figure 1, left).

Original: The concatenated agent sets [ −→ h t − 1 , ←− h t ] refer to the initial node representations of the network and the hidden interaction set h intt − 1 is concatenated with the edge embeddings of the ﬁrst GNN layer (cf.
Revised: The concatenated agent sets [ −→ h t − 1 , ←− h t ] refer to the initial node representations of the network and the hidden interaction set h intt − 1 is concatenated with the edge embeddings of the ﬁrst GNN layer (cf.

Original: Figure 2, left).
Revised: Figure 2, left).

Original: The usage of multiple message passing rounds ( ≥ 2 ) in conjunction with a fullyconnected adjacency matrix allows the model to accomodate structural information from the full input when estimating agent and interaction types.
Revised: The usage of multiple message passing rounds ( ≥ 2 ) in conjunction with a fullyconnected adjacency matrix allows the model to accomodate structural information from the full input when estimating agent and interaction types.

Original: Finally, we model the distributions comprising Eq.
Revised: Finally, we model the distributions comprising Eq.

Original: (6) and (7) as q φ 1 ( G t | x ≤ T , z <t , G <t ) =
Revised: (6) and (7) as q φ 1 ( G t | x ≤ T , z <t , G <t ) =

Original: softmax ( Linear ( f intenc ( x ≤ T , z <t , G <t ; φ 1 ))) ; see
Revised: softmax ( Linear ( f intenc ( x ≤ T , z <t , G <t ; φ 1 ))) ; see

Original: Figure 1.
Revised: Figure 1.

Original: The prior distribution p θ 3 ( G t | x <t , z <t , G <t ) is derived analogously to the interaction encoder, omitting the summarized future ←− h t as GNN input.
Revised: The prior distribution p θ 3 ( G t | x <t , z <t , G <t ) is derived analogously to the interaction encoder, omitting the summarized future ←− h t as GNN input

Original: Adapted GVRNN Once G t ∼ q φ 1 is generated, the model needs to reason over the remaining components of Eq.
Revised: Adapted GVRNN Once G t ∼ q φ 1 is generated, the model needs to reason over the remaining components of Eq.

Original: (5), while ensuring fully-unsupervised information allocation in latent space wrt.

Original: the disentanglment {G t , z t } .

Original: To achieve this, we observe that the functionality of G t is grounded in uncovering latent social dependencies at every timestep t within a multiagent segment.
Revised: To achieve this, we observe that the functionality of G t is grounded in uncovering latent social dependencies at every timestep t within a multiagent segment.

Original: Thus, the generated G t implicitly characterizes the computational graph at speciﬁc junctures and, since predictions are carried out autoregressively, encompasses all information required for computing agent states x t and z t .
Revised: Thus, the generated G t implicitly characterizes the computational graph at speciﬁc junctures and, since predictions are carried out autoregressively, encompasses all information required for computing agent states x t and z t .

Original: Motivated by the previous considerations, we assume G t is Markov and thus discard G <t from the conditions of objects p θ 1 , p θ 2 and q φ 2 .The
Revised: Motivated by the previous considerations, we assume G t is Markov and thus discard G <t from the conditions of objects p θ 1 , p θ 2 and q φ 2 .The

Original: remaining computations of Eq.
Revised: remaining computations of Eq.

Original: (5) thereby comply with the structure of G t -conditional VRNN components (cf.
Revised: (5) thereby comply with the structure of G t -conditional VRNN components (cf.

Original: Eq.
Revised: Eq.

Original: In general, we can use any graph encoding strategy performing forecasting on the inferred latent subspace G t to model the distributions over x t and z t .
Revised: In general, we can use any graph encoding strategy performing forecasting on the inferred latent subspace G t to model the distributions over x t and z t .

Original: An instantiation with interaction networks constitutes a reasonable design choice in the present context, as there is already empirical evidence of their effectiveness for modeling sports tracking data (Yeh et al., 2019; Dick et al., 2022).
Revised: An instantiation with interaction networks constitutes a reasonable design choice in the present context, as there is already empirical evidence of their effectiveness for modeling sports tracking data (Yeh et al., 2019; Dick et al., 2022).

Revised: For ease of exposition, we assume an interactive system consisting of two agents and deﬁne computations for graph i ← j , where i ∈ A and j ∈ A as follows.

Original: To accomodate G t , we parameterize the node and edge functions inherent in each graph network dependent on the predicted values in K t  and E t  .
Revised: To accomodate G t , we parameterize the node and edge functions inherent in each graph network dependent on the predicted classes E ( i,j ) t ∈ { 1 , ..., K } and K ( i ) t ∈ { 1 , ..., M } .

Original: Consequently, the amount of function approximators the framework can choose from is deﬁned by the externally speciﬁed dimensionality in the respective variables.
Revised: More speciﬁcally, we pick corresponding parameters for the edge MLP f e from the set { ψ 1 , ..., ψ K } with ψ 1 = ∅ denoting the manually deﬁned “no interaction” label and parameters for the node MLP f v from the set { γ 1 , ..., γ M } . The message passing operations for target agent i amount to

Original:  Intuitively, this inductive bias encourages the model to learn semantically meaningful concepts describing distinct movement and interaction patterns andexplicitly governs attention towardsthe important pieces of data when predicting collective movements.
Revised: [Equation] Intuitively, this inductive bias encourages the model to learn semantically meaningful concepts describing distinct movement and interaction patterns and governs attention towards important pieces of data when predicting collective movements.

Original: See Figure 1 for a detailed visual depiction of the logic and Appendix B.1 for an description of the message passing operations underlying the individual modules.
Revised: See Figure 1 for a detailed visual depiction of the logic.

Original: Figure 5b gives an overview of the computational dependencies of the proposed framework.
Revised: An overv´ıew over the computational dependencies is given in Figure 5b.

Original:  E XPERIMENTS4. S ETUP
Revised:  E XPERIMENTS

Original: Data The soccer data contains trajectories of soccer players andthe ball extracted from 16 professional soccer matches sampled at 25 frames per second. We assemble a dataset consisting of game excerpts such that the center frames of the sequences at timestep t = T// 2 correspond to an on-ball event from the set Y =
Revised: In this section, we empirically validate our proposed model on two challenging real-world datasets: basketball and soccer data 3 .

Original: { pass, other ball action, shot, none } . We choose a sequence length of 2 s and downsample the data to 10 Hz , i.e., T = 20 . This extraction process yields a total of roughlymultiagent segments divided into 70% training, 15% validation, and 15% test data. We center and normalize the trajectories onto the range [ − 1 , 1] andtransform them so that the team in possession of the ball always plays from left to right.
Revised: The dataset details, model architectures, and hyperparameters of both our model and the baseline approaches are discussed in Appendix C.1.

Original: The STATS SportVU basketball data consists of tracking data recorded from the 2016 NBA regular season. Every game sequence has two-dimensional positions of 10 players and the ball sampled atframes per second. The data is split into 60% training, 20% validation, and 20% test sets. All data is translated so that the origin of the underlying coordinate system is mapped onto the top-left corner.
Revised: 4. S ETUP

Original: Metrics We adopt the commonly used Average Displacement Error (ADE) and Final Displacement Error (FDE) metrics to assess the generative performance of future predictions.
Revised: Metrics We adopt the commonly used Average Displacement Error (ADE) and Final Displacement Error (FDE) metrics to assess the generative performance of future predictions.

Original: ADE refers to the l 2 error between the predicted locations and the ground truth averaged over the entire trajectory, while FDE is the l 2 error for the last predicted point (Alahi et al., 2014).
Revised: ADE refers to the l 2 error between the predicted locations and the ground truth averaged over the entire trajectory, while FDE is the l 2 error for the last predicted point (Alahi et al., 2014).

Original: Following prior works, we report the minimum over 20 generated samples.
Revised: Following prior works, we report the minimum over 20 generated samples.

Original: Baselines We compare our method3 with a variety of SOTA baselines for modeling  sports tracking data: Weak-Sup (Zhan et al., 2018), GVRNN (Yeh et al., 2019),dNRI (Graber & Schwing, 2020),
Revised: Baselines We compare our method with a variety of SOTA baselines for modeling (i) sports tracking data: Weak-Sup (Zhan et al., 2019), GVRNN (Yeh et al., 2019),

Original:  DAG-Net (Monti et al., 2021),  and GRIN (Li et al., 2021b).
Revised: dNRI (Graber & Schwing, 2020), DAG-Net (Monti et al., 2021), GRIN (Li et al., 2021b); and (ii) for modeling urban dynamics: Joint- β -cVAE (Bhattacharyya et al., 2021) and GRIN (Li et al., 2021b).

Revised: Generally, we observe that methods proposed for are formalized as some form of conditional variational autoencoder that aggregates latent information across time.

Original: To enable direct comparison against this diverse set of methods, we benchmark in two experimental conﬁgurations.
Revised: To enable direct comparison against this diverse set of methods, we benchmark in two experimental conﬁgurations.

Original: Firstly, DAG-Net and Weak-Sup are inherently advantageous in generating agent trajectories by incorporating future locations in form of heuristically generated labels at prediction time.
Revised: Firstly, DAG-Net and Weak-Sup are inherently advantageous in generating agent trajectories by incorporating future locations in form of heuristically generated labels at prediction time.

Original: This renders a direct comparison to fully unsupervised generative methods (like ours) biased.
Revised: This renders a direct comparison to fully unsupervised generative methods (like ours) biased.

Original: To minimize this bias, we report quantitative results on a rather long prediction horizon of T obs = 10 , T pred = 40 for both methods.
Revised: To minimize this bias, we report quantitative results on a rather long prediction horizon of T obs = 10 , T pred =for both methods.

Original: Other baselines such as GRIN and dNRI cannot be trivially tested on these long-term predictions, so we generate results on conﬁguration T obs = 40 , T pred = 10 .
Revised: Other baselines such as GRIN and dNRI cannot be trivially tested on these longterm predictions, so we generate results on conﬁguration T obs = 40 , T pred = 10 .

Original: Finally, GVRNN and our framework (DIA) can be applied to the full range of prediction scenarios since both methods learn spatiotemporal representations autoregressively. [Table] [Table]
Revised: Finally, GVRNN, Joint- β -cVAE and our framework (DIA) can be applied to the full range of prediction scenarios.

Original: 4. Q UANTITATIVE E VALUATION
Revised: 4. Q UANTITATIVE E VALUATION

Original: Baseline Comparisons In the ﬁrst set of experiments, we benchmark our model against recent generative methods for the task of trajectory forecasting.
Revised: Baseline Comparisons In the ﬁrst set of experiments, we benchmark our model against recent generative methods for the task of trajectory forecasting.

Original: Our empirical ﬁndings are summarized in
Revised: Our empirical ﬁndings are summarized in

Original: Table 1 and Table 2 (left).
Revised: Table 1 and Table 2 (left).

Original: DIA emerges as the best generative tool across all tested tasks improving the best (unsupervised) competitor performance by at least 17.3% minADE and 25.5% minFDE for basketball.
Revised: DIA emerges as the best generative tool across all tested tasks improving the best (unsupervised) competitor performance by at least 17.3% minADE and 25.5% minFDE for basketball.

Original: The empirical results suggest that our proposed modeling strategy effectively both eliminates interfering social signals and discriminates between different agent and interaction types so that future agent movements can be captured more accurately.

Original: In addition, since GRIN represents the only tested method that learns latent variables at sequence level, its comparatively dismal results highlight the hypothesized heterogeneous and dynamic causal nature of multiagent systems in the sports domain. Remarkably, despite its inherent advantages, DIA also outperforms recent supervised generative methods (Weak-Sup and DAG-Net) by at least 8.3% minADE and 16.8% minFDE. 
Revised: Remarkably, despite their inherent advantages, DIA also outperforms recent supervised generative methods (Weak-Sup and DAG-Net) by at least 8.3% minADE and 16.8% minFDE. Furthermore, we observe that the methods that learn global latent variables (GRIN and Joint- β -cVAE) perform signiﬁcantly worse than the remaining models. Though it may be premature to arrive at a

Revised: [Table]

Original:  Investigating Latent Structure To quantify the beneﬁt of augmenting the latent space with an explicit causal graph G t , we compare performance metrics of the GVRNN and an adapted DIA version at varying observation and prediction lengths.
Revised: [Table] deﬁnite conclusion, however the comparatively dismal performance of such baselines in our results tend to suggest limited transferability of such models for sports data. We hypothesize that this is due to lower inherent social interaction patterns in urban environments in relation to the heterogeneous and dynamic causal nature of multiagent systems in the sports domain. Investigating Latent Structure To quantify the beneﬁt of augmenting the latent space with an explicit causal graph G t , we compare performance metrics of the GVRNN and an adapted DIA version at varying observation and prediction lengths.

Original: For this, we modify the DIA architecture such that the second part of the training procedure is identical to the GVRNN computations.
Revised: For this, we modify the DIA architecture such that the second part of the training procedure is identical to the GVRNN computations.

Original: In this way, the resulting performance differences capture only inﬂuences originating from the proposed graph mechanism.
Revised: In this way, the resulting performance differences capture only inﬂuences originating from the proposed graph mechanism.

Original: The left part of ﬁgure 3 visualizes the resulting numbers.
Revised: The left part of ﬁgure 3 visualizes the resulting numbers.

Original: As can be seen, our DIA version yields substantial performance gains, with the performance difference increasing with task complexity, i.e., prediction horizon.
Revised: As can be seen, our DIA version yields substantial performance gains, with the performance difference increasing with task complexity, i.e., prediction horizon.

Revised: We note that an increase in GVRNN complexity is accompanied by a further increase in performance difference, see Appendix C.3.

Original: The results  highlight that learning agent relationships via information aggregation strategiesentangled in single latent variables is insufﬁcient to capture decisive social signals in non-trivial multiagent systems.
Revised: The results thus highlight that modeling relations via attention-based aggregation strategies is insufﬁcient to capture decisive social signals in non-trivial multiagent systems.

Original: The right part of Figure 3 shows predictive results for different agent subgroups when varying the representational capacity of the latent interaction graph
Revised: The right part of Figure 3 shows predictive results for different agent subgroups when varying the representational capacity of the latent interaction graph

Original: E t ∈ G t .
Revised: E t ∈ G t .

Original: Since separate function approximators are parameterized for each selected interaction class, enlarging the interaction space dimensionality causes overﬁtting issues, despite higher model expressivity.
Revised: Since separate function approximators are parameterized for each selected interaction class, enlarging the interaction space dimensionality causes overﬁtting issues, despite higher model expressivity.

Original: Thus, the best results are realized by the model conﬁguration with the lowest dimensionality, which still sufﬁciently encapsulates the underlying system dynamics.
Revised: Thus, the best results are realized by the model conﬁguration with the lowest dimensionality, which still sufﬁciently encapsulates the underlying system dynamics.

Original: Accordingly, this experimental setup can shed light on whether the proposed interaction mechanism E t behaves according to our theoretical considerations.
Revised: Accordingly, this experimental setup can shed light on whether the proposed interaction mechanism E t behaves according to our theoretical considerations.

Original: For example, in modeling defensive players, we would expect reactive patterns based on the behavior of attacking players, resulting in little intra-group dependencies.
Revised: For example, in modeling defensive players, we would expect reactive patterns based on the behavior of attacking players, resulting in little intra-group dependencies.

Original: Indeed, the results in Figure 3 indicate that our model inherits the expected social behaviors, as the best parameter conﬁguration is in line with the structural complexity of the different player subgroups.
Revised: Indeed, the results in Figure 3 indicate that our model inherits the expected social behaviors, as the best parameter conﬁguration is in line with the structural complexity of the different player subgroups.

Original: Appendix C.1 provides an analogous discussion for agent types K t ∈ G t .
Revised: Appendix C.4 provides an analogous discussion for agent types K t ∈ G t .

Original: Auxiliary Task
Revised: Auxiliary Task

Original: Finally, we examine the extent to which learned latent categories G t serve meaningful semantic contributions beyond trajectory generation tasks.
Revised: Finally, we examine the extent to which learned latent categories G t serve meaningful semantic contributions beyond trajectory generation tasks.

Original: To this end, we report quantitative metrics on a classiﬁcation task using soccer data.
Revised: To this end, we report quantitative metrics on a classiﬁcation task using soccer data.

Original: In the ﬁrst step, we assemble a dataset in which the social context at each time step is deﬁned by the estimated agent-ball inﬂuences of a DIA training run.
Revised: In the ﬁrst step, we assemble a dataset in which the social context at each time step is deﬁned by the estimated agent-ball inﬂuences of a DIA training run.

Original: That is, agent locations annotated as “no interaction” when considering inﬂuences on the ball node are discarded.
Revised: That is, agent locations annotated as “no interaction” when considering inﬂuences on the ball node are discarded.

Original: We then train a spatiotemporal classiﬁcation network, termed Interaction-RNN , for event detection over Y on the updated data (more details given in Appendix B.3).
Revised: We then train a spatiotemporal classiﬁcation network, termed Interaction-RNN , for event detection over Y on the updated data (more details given in Appendix B.3).

Original: We compare our method quantitatively against a diagonal (i.e., using ball trajectories only) and a fully-connected (i.e., using all agents for neighborhood aggregation) version of Interaction-RNN in Table 2 (right).
Revised: We compare our method quantitatively against a diagonal (i.e., using ball trajectories only) and a fully-connected (i.e., using all agents for neighborhood aggregation) version of Interaction-RNN in Table 2 (right).

Original: The model conﬁguration trained on the data previously extracted by DIA outperforms both baseline conﬁgurations.
Revised: The model conﬁguration trained on the data previously extracted by DIA outperforms both baseline conﬁgurations.

Original: The results indicate potential advantages of injecting interaction structures induced by DIA in downstream applications, as they facilitate detecting key information for learning spatiotemporal classiﬁcation tasks.
Revised: The results indicate potential advantages of injecting interaction structures induced by DIA in downstream applications, as they facilitate detecting key information for learning spatiotemporal classiﬁcation tasks.

Original: 4. Q UALITATIVE E VALUATION
Revised: 4. Q UALITATIVE E VALUATION

Original: To provide more insights in the learned sequential graphs G ≤ T of DIA, we visually depict a multiagent segment from the test set and estimated probability values within two interaction categories (and one “no interaction” category) in Figure 4.
Revised: To provide more insights in the learned sequential graphs G ≤ T of DIA, we visually depict a multiagent segment from the test set and estimated probability values within two interaction categories (and one “no interaction” category) in Figure 4.

Original: The upper right image shows the inﬂuencing factors of the red player on the ball.
Revised: The upper right image shows the inﬂuencing factors of the red player on the ball.

Original: This interaction structure is mainly characterized by frequent alterna- tions between the two interaction types with scarce instants of low aggregated inﬂuence.
Revised: This interaction structure is mainly characterized by frequent alternations between the two interaction types with scarce instants of low aggregated inﬂuence.

Original: Since the interaction values are determined to accurately reﬂect the input data, the ﬂuctuating pattern observed illustrates highly dynamic social structures in our tested setting.
Revised: Since the interaction values are determined to accurately reﬂect the input data, the ﬂuctuating pattern observed illustrates highly dynamic social structures in our tested setting.

Original: The ﬁgure below encodes the inﬂuential structure of the ball as the sending agent on the movements of the red player.
Revised: The ﬁgure below encodes the inﬂuential structure of the ball as the sending agent on the movements of the red player.

Original: Compared to the previous ﬁgure, probability values are consistent with one interaction type being active over most of the observed time period.
Revised: Compared to the previous ﬁgure, probability values are consistent with one interaction type being active over most of the observed time period.

Original: The second interaction type becomes the dominant inﬂuential force from the moment the ball is passed to the spatially closest neighbor (green player), and also remains active when the ball reaches the target agent (red player).
Revised: The second interaction type becomes the dominant inﬂuential force from the moment the ball is passed to the spatially closest neighbor (green player), and also remains active when the ball reaches the target agent (red player).

Original: Thus, the ﬁrst interaction type encodes more holistic inﬂuences, while the second type focuses on more immediate social motion effects.
Revised: Thus, the ﬁrst interaction type encodes more holistic inﬂuences, while the second type focuses on more immediate social motion effects.

Original: Notably, there are no periods of time in which either of the two interaction modes is inactive.
Revised: Notably, there are no periods of time in which either of the two in- teraction modes is inactive.

Original: This distinguishes the ball as the central element in basketball and thus affects the movement of all offensive players at any time.
Revised: This distinguishes the ball as the central element in basketball and thus affects the movement of all offensive players at any time.

Original: The qualitative results provide further evidence for the ability of our model to learn a semantically meaningful discrete latent subspace.
Revised: The qualitative results provide further evidence for the ability of our model to learn a semantically meaningful discrete latent subspace.

Original: More qualitative results are given in Appendix C.3.
Revised: More qualitative results are given in Appendix C.6.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Recent methods for trajectory prediction tasks mostly propose some form of latent variable model, generally formulated as a conditional VAE (Sohn et al., 2015), to capture the stochasticity inherent in future trajectories.
Revised: A conceptual line of work adresses human trajectory forecasting as a deterministic regression problem by minimizing the negative log-likelihood and assuming bi-variate Gaussian output distributions Mohamed et al.

Original: At a high level, the concrete approaches differ in their choices of methodological tools for modeling temporal dependencies (e.g. LSTMs (Hochreiter & Schmidhuber, 1997) or GRUs (Chung et al., 2014)) and social relations (e.g., GATs (Veli ˇ ckovi ´
Revised: ; Rudenko et al. While assuming unimodal Gaussians may be sufﬁcient for pedestrian datasets with rather linear trajectories Makansi et al. (2022), they fall short for multimodal trajectory distributions like in team sports Brefeld et al.

Original: c et al., 2017) or Transformers (Vaswani et al., 2017)), as well as  their latent space characterizations (discrete or continuous).
Revised: ; so we do not consider these models in this work. Instead, recent methods mostly propose some form of latent variable model, generally formulated as a conditional VAE (Sohn et al., 2015), to capture the stochasticity inherent in future trajectories. At a higher level, the concrete approaches can be broadly categorized by how they model temporal and social dependencies, as well as by their latent space characterization. We operate within this model class and provide an overview over such methods below.

Original: For example, Yeh et al.
Revised: For example, Yeh et al.

Original: ; Sun et al.
Revised: ; Sun et al.

Revised: ; Zhan et al.

Original: ; Monti et al.
Revised: ; Monti et al.

Original: (2021) extend the VRNN to handle agent-agent relations via estimating components of Eq.
Revised: (2021) extend the VRNN to handle agent-agent relations via estimating components of Eq.

Original: 1 using graph neural networks.
Revised: 1 using graph neural networks.

Original: Casas et al.
Revised: Casas et al.

Original: (2020b)
Revised: (2020b)

Original: ; Salzmann et al.
Revised: ; Salzmann et al.

Original: (2020) choose a similar task formulation as well as interaction encoding strategy Casas et al. (2020a), but generate future trajectories non-autoregressively by ﬂattening the time axis into a single dimension.
Revised: (2020); Bhattacharyya et al. (2021) choose a similar task formulation as well as interaction encoding strategy (Casas et al., 2020a; Anderson et al., 2018), but generate future trajectories non-autoregressively by ﬂattening the time axis into a single dimension.

Original: Other  similar approaches (e.g., Girgis et al.
Revised: Other conceputally similar approaches (e.g., Girgis et al.

Original: ; Yuan et al.
Revised: ; Yuan et al.

Original: propose transformer-based architectures to encode spatial and temporal relations.
Revised: propose transformer-based architectures to encode spatial and temporal relations.

Original: There is a plethora of work that aims to improve trajectory forecasting by extending latent information with speciﬁc types of agent long-term goals extracted from the trajectory data (Mangalam et al., 2020; 2021;
Revised: There is a plethora of work that aims to improve trajectory forecasting by extending latent information with speciﬁc types of agent long-term goals extracted from the trajectory data (Mangalam et al., 2020; 2021;

Original: Zhao et al., 2021; Monti et al., 2021; Zhan et al., 2018).
Revised: Zhao et al., 2021; Monti et al., 2021; Zhan et al., 2019; Fan et al., 2021; Choi et al., 2021; Girase et al., 2021).

Revised: However, all these approaches model interactions only implicitly by aggregating messages along the social dimension into spatiotemporal representations.

Revised: Interestingly, Rudolph et al.

Revised: (2020) conclude that graph conditional variational methods for predicting multiagent trajectories are often too powerful and simpliﬁed methods (e.g., MDNs) should be preferred.

Original: Alternatively, our method can be seen as an novel contribution to the line of research that aims to explicitly infer agent interactions while executing a trajectory prediction task.
Revised: Alternatively, our method can be seen as an novel contribution to the line of research that aims to explicitly infer agent interactions while executing a trajectory prediction task.

Original: This task formulation was originally proposed by Kipf et al.
Revised: This task formulation was originally proposed by Kipf et al.

Original: (2018), who introduce the NRI framework, a variational autoencoder, where the discrete latent code represents edge predictions in a causal graph.
Revised: (2018), who introduce the NRI framework, a variational autoencoder, where the discrete latent code represents edge predictions in a causal graph.

Original: Since the original formulation is limited to learning a single graph for the entire multiagent sequence, several works propose dynamic extensions where the interaction graph can adapt to the conditions per time step (Graber & Schwing, 2020; Li et al., 2020;
Revised: Since the original formulation is limited to learning a single graph for the entire multiagent sequence, several works propose dynamic extensions where the interaction graph can adapt to the conditions per time step (Graber & Schwing, 2020; Li et al., 2020;

Original: Gong et al., 2021; Li et al., 2021a).
Revised: Gong et al., 2021; Li et al., 2021a).

Original: Recently, L ¨ owe et al.
Revised: Recently, L ¨ owe et al.

Original: (2022) propose to infer causal relations across data points with different underlying interaction graphs but shared dynamics.
Revised: (2022) propose to infer causal relations across data points with different underlying interaction graphs but shared dynamics.

Original: Other work (Li et al., 2021b) builds upon the NRI framework but leverage a continuous latent space that aims to separate interactive factors from agent intentions.
Revised: Other work (Li et al., 2021b) builds upon the NRI framework but leverage a continuous latent space that aims to separate interactive factors from agent intentions.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we presented novel framework for modeling the joint distribution of agent trajectories using latent variables.
Revised: In this paper, we presented novel framework for modeling the joint distribution of agent trajectories using latent variables.

Original: To alleviate shortcomings of previous works, we described the data generation process using disentangled factors of variation that explicitly include a discrete latent subspace reﬂecting social structures in sports games.
Revised: To alleviate shortcomings of previous works, we described the data generation process using disentangled factors of variation that explicitly include a discrete latent subspace reﬂecting social structures in sports games.

Original: We demonstrated that the emerging architecture performs better in predicting trajectories compared to existing strategies and learns informative latent variables.
Revised: We demonstrated that the emerging architecture performs better in predicting trajectories compared to existing strategies and learns informative latent variables.

Revised: An interesting future line of research is modeling the underlying dynamics at a group level to identify unique team strategies (Raman et al., 2021).

Original: A UTHOR C ONTRIBUTIONS
Revised: A UTHOR C ONTRIBUTIONS

Original: If you’d like to, you may include a section for author contributions as is done in many journals.
Revised: If you’d like to, you may include a section for author contributions as is done in many journals.

Original: This is optional and at the discretion of the authors.
Revised: This is optional and at the discretion of the authors.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: Use unnumbered third level headings for the acknowledgments.
Revised: Use unnumbered third level headings for the acknowledgments.

Original: All acknowledgments, including those to funding agencies, go at the end of the paper.
Revised: All acknowledgments, including those to funding agencies, go at the end of the paper.

Original: A A LTERNATIVE P ERSPECTIVE ON THE O BJECTIVE F UNCTION
Revised: A A LTERNATIVE P ERSPECTIVE ON THE O BJECTIVE F UNCTION

Original: [Figure]
Revised: [Figure]

Original: Alternatively, Eq. 5 can be written as
Revised: Alternatively, Eq. 5 can be written as

Original: [Equation]
Revised: [Equation]

Original: where G ≤ T
Revised: where G ≤ T

Original: [Equation]
Revised: [Equation]

Original: z ≤ T ∼ q φ ( z ≤ T , G ≤ T | x ≤ T ) .
Revised: z ≤ T ∼ q φ ( z ≤ T , G ≤ T | x ≤ T ) .

Original: See also Figure 5.
Revised: See also Figure 5.

Original: This can be shown as follows:
Revised: This can be shown as follows:

Original: [Equation]
Revised: [Equation]

Original: x ≤ t , z <t , G ≤ t ) (cid:107) p θ ( G t | x <t , z <t , G <t )] (cid:35) .
Revised: x ≤ t , z <t , G ≤ t ) (cid:107) p θ ( G t | x <t , z <t , G <t )] (cid:35) .

Original: B M ODEL D ETAILS B. A DAPTED GVRNN M ODULE
Revised: B M ODEL D ETAILS B. G RAPH N EURAL N ETWORKS

Original: For ease of exposition, we assume an interactive system consisting of two agents and deﬁne computations for graph i ← j , where i ∈ A and j ∈ A .
Revised: The core operations of the GNN modules comprising the DIA can be formally expressed as follows: [Equation]

Original: Given the predicted class E ( i,j ) t ∈ { 1 , ..., K } , we pick associated parameters for the edge MLP f e from the set { ψ 1 , ..., ψ K } with ψ 1 = ∅ denoting the manually deﬁned “no interaction” label. Given the predicted class K ( i ) t ∈ { 1 , ..., M } , we pick associated parameters for the node MLP f v from the set { γ 1 , ..., γ K } . The message passing operations for target agent i amount to
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Intuitively, embedding e ( i,j ) encodes the inﬂuence of agent j on target agent i in the form ofan vectorial representation. Under the assumption of ceteris paribus, the inferred feature representation varies wrt.
Revised: (12) where v i is the input feature representation of agent/node i , o i is the respective node embedding after 1 GNN layer, N ( i ) denotes the set of agents/nodes that interact with target agent i , e ( i,j ) is the edge embedding of the ﬁrst GNN layer, and f e and f v are the MLPs described above.

Original:  theunderlying interaction type that instantiates parameter conﬁguration ψ k . Finally, our module deﬁnition realizes role-dependent behavioral patterns onhow agent i acts upon the inferred neighborhood inﬂuence
Revised: For the GNNs of the interaction encoding phase, we deﬁne 2 rounds of message passing and utilize additional linear layers operating on the node and edge embeddings of the second GNN layer to output K t and

Original: e ( i,j ) .
Revised: E t , respectively.

Original: These patterns are reﬂected in set { γ 1 , ..., γ M } .
Revised: For the adapted GVRNN modules, we use 1 layer as described in the main text with additional MLPs computing the mean and variance vectors of the Gaussians.

Original: B.  T RAINING & T ESTINGD ETAILS
Revised: B. M ODEL T RAINING & T ESTING

Original: Instead of absolute positions, our model predicts movements ∆ˆ x t at each timestep.
Revised: Instead of absolute positions, our model predicts movements ∆ˆ x t at each timestep.

Original: Consequently, we estimate agent locations via ˆ x t
Revised: Consequently, we estimate agent locations via ˆ x t

Original: = x t − 1 + ∆ˆ x t .
Revised: = x t − 1 +∆ˆ x t .

Original: For simplicity, x t refers to both relative and absolute positions.
Revised: For simplicity, x t refers to both relative and absolute positions.

Original: For training, the model predicts the next step for the full sequence using samples fromthe encoder and ground-truth information x ≤ T .
Revised: For training, the model uses the entire T = T obs + T pred timesteps from ground-truth sequences x ≤ T .

Original: To enable gradient ﬂow through stochastic operations, we use the reparameterization and gumbel-softmax trick (Maddison et al., 2017; Jang et al., 2017) for  the continuous z t and  discrete G t latent subspace, respectively. 
Revised: To enable gradient ﬂow through stochastic operations, we use reparametrization and gumbel-softmax trick (Maddison et al., 2017; Jang et al., 2017) for sampling from the encoders over the continuous z t and the discrete G t latent subspace, respectively. .

Original: At test time, given a mutliagent sequence { x 1 , ..., x T obs } , we get ˆ x T obs + i , i ∈
Revised: At test time, we divide the trajectories into an observation and prediction period, where the model only observes the ﬁrst portion of the ground-truth trajectory { x 1 , ..., x T obs } and predicts the remaining T pred timesteps autoregressively: ˆ x T obs + i , i ∈

Original: Z using samples from the prior distributions and take the movement predictions as input.
Revised: Z . Latent variables are sampled from the prior distributions. B. C LASSIFICATION B ASELINE

Original: All models are implemented using PyTorch (Paszke et al., 2019).
Revised: The proposed classiﬁcation framework for the spatiotemporal multi-agent regime ( Interaction-RNN ) is essentially geared towards the functionality of an RNN for capturing temporal dependencies of player trajectories. Additionally, to encode interactive agent patterns, we update the inferred RNN states at each time step using an attention-based graph neural network architecture (Veli ˇ ckovi ´ c et al., 2017). Thus, the updated feature vectors jointly encode social and temporal information. A ﬁnal softmax layer is then used for classiﬁcation and the model is optimized by minimizing the crossentropy loss.

Revised: Because the labels denote ball-centric events, we use the output of the ball node for loss computation and evaluation.

Revised: C E XPERIMENTS D ETAILS C. D ATA

Revised: The soccer data contains trajectories of soccer players and the ball extracted from 16 professional soccer matches sampled at 25 frames per second.

Revised: We assemble a dataset consisting of game excerpts such that the center frames of the sequences at timestep t = T// 2 correspond to an on-ball event from the set Y =

Revised: { pass, other ball action, shot, none } .

Revised: We choose a sequence length of 2 s and downsample the data to 10 Hz , i.e., T = 20 .

Revised: This extraction process yields a total of roughlymultiagent segments divided into 70% training, 15% validation, and 15% test data.

Revised: We center and normalize the trajectories onto the range [ − 1 , 1] and transform them so that the team in possession of the ball always plays from left to right.

Revised: The STATS SportVU basketball data consists of tracking data recorded from the 2016 NBA regular season.

Revised: Every game sequence has two-dimensional positions of 10 players and the ball sampled atframes per second.

Revised: The data is split into 60% training, 20% validation, and 20% test sets.

Revised: All data is translated so that the origin of the underlying coordinate system is mapped onto the top-left corner.

Revised: C. I MPLEMENTATION D ETAILS

Revised: DIA All model variants are implemented using PyTorch (Paszke et al., 2019).

Original: Training is carried out using Adam (Kingma & Ba, 2014) with default parameters and learning rate of 0 . 001 and teacher forcing.
Revised: Training is carried out using Adam (Kingma & Ba, 2014) with default parameters and learning rate of 0 . 001 and teacher forcing.

Original: We select the best performing model using minADE on the validation set.
Revised: We select the best performing model using minADE on the validation set.

Original: The message passing operations within our graph networks f e and f v are 2 -layer MLPs with batch normalization (Ioffe & Szegedy, 2015), dropout (Srivastava et al., 2014), and ELU activations (Clevert et al., 2015).
Revised: The message passing operations within our graph networks f e and f v are 2 -layer MLPs with batch normalization (Ioffe & Szegedy, 2015), dropout (Srivastava et al., 2014), and ELU activations (Clevert et al., 2015).

Original: In addition, we leverage 2 -layer MLPs with LeakyReLU
Revised: In addition, we leverage 2 -layer MLPs with LeakyReLU

Original: (Xu et al., 2015) activations as feature extractors operating on z t , x t , and G t .
Revised: (Xu et al., 2015) activations as feature extractors operating on z t , x t , and G t .

Original: All fully-connected layers are initialized using Xavier initialization (Glorot & Bengio, 2010).
Revised: All fully-connected layers are initialized using Xavier initialization (Glorot & Bengio, 2010).

Original: For recurrence, we use 2 -layer GRU networks (Chung et al., 2014).
Revised: For recurrence, we use 2 -layer GRU networks (Chung et al., 2014).

Original: The core operations of the GNN modules comprising the DIA can be formally expressed as follows: [Equation] where v i is the input feature representation of agent/node i ,
Revised: Baselines We used the code from the ofﬁcial repositories for baselines (Zhan et al., 2019; Graber & Schwing, 2020;

Original: o i is the respective node embedding after 1 GNN layer, N ( i ) denotes the set of agents/nodes that interact with target agent i , e ( i,j ) is the edge embedding of the ﬁrst GNN layer, and f e and f v are the MLPs described above. For the GNNs of the interaction encoding phase, we deﬁne 2 rounds of message passing and utilize additional linear layers operating on the node and edge embeddings of the second GNN layer to output K t and
Revised: Monti et al., 2021; Li et al., 2021b; Bhattacharyya et al., 2021) and chose the best perfoming hyperparameters from their experiments. For (Yeh et al., 2019), we re-implemented the model by faithfully following the descriptions in their paper. The overall architecture of GVRNN was initially designed such that it is comparable in parameter number to DIA.

Original: E t , respectively. For the adapted GVRNN modules, we use 1 layer as described in Section B.1 with additional MLPs computing the mean and variance vectors of the Gaussians.
Revised: However, we found that reducing model expressivity led to improved results. If not explicitly stated, the reported numbers correspond to a GVRNN version with approx 84% parameters compared to DIA.

Original: The proposed classiﬁcation framework for the spatiotemporal multi-agent regime ( Interaction-RNN ) is essentially geared towards the functionality of an RNN for capturing temporal dependencies of player trajectories. Additionally, to encode interactive agent patterns, we update the inferred RNN states at each time step using an attention-based graph neural network architecture (Veliˇckovi´c et al., 2017). Thus, the updated feature vectors jointly encode social and temporal information.
Revised: [Figure]

Original: A ﬁnal softmax layer is then used for classiﬁcation and the model is optimized by minimizing the crossentropy loss. Because the labels denote ball-centric events, we use the output of the ball node for loss computation and evaluation. C A DDITIONAL R ESULTS C.
Revised: minADE minFDE # Parameters

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Table]

Revised: To demonstrate the importance of our proposed structure

Original: [Table]
Revised: [Table]

Revised: we perform two sets of additional experiments.

Revised: First, we directly compare the importance of structure versus larger latent capacity by making adjustments to the GVRNN baseline and our proposed model.

Revised: To achieve this, we (i) enlarge the encoder/prior and decoder by increasing latent, hidden and rnn dimensions of the GVRNN modules and (ii) remove the proposed graph mechanism from our (adjusted) model so that it essentially resembles a deeper GVRNN.

Revised: Secondly, we compare generative capabilities of DIA and GVRNN at different data percentages of 5%, 10%, 20%, and 50% of the training data.

Revised: The training data were randomly selected and the same data is used to train the different models.

Revised: Hence, the purpose here is to investigate whether the integrated structure translates into data efﬁciency by, for example, governing attention to important pieces of data.

Revised: Table 3 shows results to investigate the effect of model size on generative capacities.

Revised: Clearly, larger decoders/encoder/priors yield degenerate results compared to the initial (best) GVRNN conﬁguration.

Revised: Furthermore, the DIA comparisons highlight the signiﬁcant importance of enforcing a graph structure since the only difference between both model conﬁgurations is our interaction learning strategy.

Revised: Figure 6 shows the data learning efﬁciency experiments results with mean and error.

Revised: Impressively, our model still signiﬁcantly outperforms the full GVRNN baseline on both error metrics using only 5% (!) of the data.

Revised: Our model discovers the underlying patterns much faster than

Revised: GVRNN, whose min ADE numbers improve signiﬁcantly between 5% and 100% data partitions.

Revised: Although the results may not be conclusive, they do provide further evidence that our realized improvements do not originate from increased model complexity, but rather because G t accurately captures the latent ground-truth factors that arise from social perspectives.

Revised: [Table]

Revised: Table 4 shows parameter sizes and sampling speeds of SOTA methods expressed as averaging over mini-batches for modeling offensive players in basketball.

Revised: While our model is slower than GVRNN, we emphasize that DIA can reduce runtime by at least 10x compared to GVRNN while maintaining superior results (cf.

Original:  To provide further evidence for potential semantic interpretations of the discrete latent space, we examine the effect of capacity change in agent types K t ∈ G t on the generative performance.
Revised: Figure 6). C. A GENT T YPES [Table] To provide further evidence for potential semantic interpretations of the discrete latent space, we examine the effect of capacity change in agent types K t ∈ G t on the generative performance.

Original: Table 3 summarizes the results for modeling the dynamics of all players on basketball data.
Revised: Table 5 summarizes the results for modeling the dynamics of all players on basketball data.

Original: Intuitively, we would expect 3 fundamentally different agent types causing the observed movements since the data consists of attacking and defensive players, and the ball.
Revised: Intuitively, we would expect 3 fundamentally different agent types causing the observed movements since the data consists of attacking and defensive players, and the ball.

Original: Similar to Section 4.2, the empirical results conﬁrm this intuition.
Revised: Similar to Section 4.2, the empirical results conﬁrm this intuition.

Original: [Figure]
Revised: [Figure]

Original: An appropriate local solution in our learning problem accurately describes the data while collecting implicit information into latent space {G t , z t } .
Revised: An appropriate local solution in our learning problem accurately describes the data while collecting implicit information into latent space {G t , z t } .

Original: As part of the main paper, we qualitatively and quantitatively investigated the discovered relational structure G t , ﬁnding semantically signiﬁcant patterns, and demonstrated an accurate approximation of the underlying multimodal data distributions through baseline comparisons.
Revised: As part of the main paper, we qualitatively and quantitatively investigated the discovered relational structure G t , ﬁnding semantically signiﬁcant patterns, and demonstrated an accurate approximation of the underlying multimodal data distributions through baseline comparisons.

Original: Thus, this section is devoted to quantify the information captured in the continuous subspace z t .
Revised: Thus, this section is devoted to quantify the information captured in the continuous subspace z t .

Original: A potential metric to measure the degree of dependency between x and z i.e., to decide whether the continuous latent variable encodes useful information, is by monitoring the KL divergence between the variational posterior and prior.
Revised: A potential metric to measure the degree of dependency between x and z i.e., to decide whether the continuous latent variable encodes useful information, is by monitoring the KL divergence between the variational posterior and prior.

Original: When naively training VAE architectures that consist of a decoder exhibiting sufﬁciently powerful function approximators with autoregressive dependencies (e.g., RNNs), existing sequential latent variable models frequently report a phenomenon named posterior collapse (Chung et al., 2015), where the model tends to converge to regions of the loss surface that contain bad local minima (or saddle points) at KL = 0 .
Revised: When naively training VAE architectures that consist of a decoder exhibiting sufﬁciently powerful function approximators with autoregressive dependencies (e.g., RNNs), existing sequential latent variable models frequently report a phenomenon named posterior collapse (Chung et al., 2015), where the model tends to converge to regions of the loss surface that contain bad local minima (or saddle points) at KL = 0 .

Original: Since the latent information the decoder receives is then essentially equivalent to Gaussian noise, the model remits to a standard unconditional RNN, i.e., it learns to generate the input solely based on the autoregressive properties independent of the latent information z .
Revised: Since the latent information the decoder receives is then essentially equivalent to Gaussian noise, the model remits to a standard unconditional RNN, i.e., it learns to generate the input solely based on the autoregressive properties independent of the latent information z .

Original: Figure 6 displays (average) values for losses L DIA and KL [ q φ 1 ( z t | x ≤ T , z <t , G ≤ t ) (cid:107) p θ 2 ( z t | x <t , z <t , G ≤ t )] during a training run on soccer data.
Revised: Figure 7 displays (average) values for losses L DIA and KL [ q φ 1 ( z t | x ≤ T , z <t , G ≤ t ) (cid:107) p θ 2 ( z t | x <t , z <t , G ≤ t )] during a training run on basketball data.

Original: Here we observe a gradual increase in KL loss values that is accompanied by a steady decrease in training loss.
Revised: Here we observe a gradual increase in KL loss values that is accompanied by a steady decrease in training loss.

Original: Thus, the results indicate an effective exploitation of the continuous subspace z t to generate future agent movements without the use of common optimization strategies such as cost annealing (Bowman et al., 2016;
Revised: Thus, the results indicate an effective exploitation of the continuous subspace z t to generate future agent movements without the necessity to rely on common optimization strategies such as cost annealing (Bowman et al., 2016;

Original: Sønderby et al., 2016).
Revised: Sønderby et al., 2016).

Original: C. Q UALITATIVE E VALUATION
Revised: C. Q UALITATIVE E VALUATION

Original: Figure 7 shows another example of agent-ball interactions for the multi-agent segment from section 4.3.
Revised: Figure 8 shows another example of agent-ball interactions for the multi-agent segment from section 4.3.

Original: When considering inﬂuences on the ball (upper image), we again see a high frequency with respect to the dominant interaction type.
Revised: When considering inﬂuences on the ball (upper image), we again see a high frequency with respect to the dominant interaction type.

Original: When reversing the inﬂuence direction, we again observe more constant patterns, where the second interaction type becomes dominant as soon as the ball arrives at the spatially closest agent.
Revised: When reversing the inﬂuence direction, we again observe more constant patterns, where the second interaction type becomes dominant as soon as the ball arrives at the spatially closest agent.

Original: Figure 8 depicts an exemplary timeline for player-player interactions in a Bernoulli setting.
Revised: Figure 9 depicts an exemplary timeline for player-player interactions in a Bernoulli setting.

Original: Again, our model learns an interpretable semantic structure, which we describe below.
Revised: Again, our model learns an interpretable semantic structure, which we describe below.

Original: The sharp drop in probability values at around t = 10 (right plot) is accompanied with the direction change of the blue player (indicated by symbol 1 ) in the left 
Revised: The sharp drop in probability values at around t = 10 (right plot) is accompanied with the direction change of the blue player (indicated by symbol 1 ) in the left plot.

Revised: The probability spike between t = 20 and t = 30 is caused by the change of direction of the orange player (symbol 2 ) at the right corner on the pitch.

Original: The spike at around t = 40 is caused by the blue player turning towards the orange player (symbol 3 ); the last increase in probability occurs when the blue player accelerates towards the orange player (symbol 4 ).
Revised: The spike at around t = 40 is caused by the blue player turning towards the orange player (symbol 3 ); the last increase in probability occurs when the blue player accelerates towards the orange player (symbol 4 ).