Original: Semi-Supervised Generative Models for Multi-Agent Trajectories
Revised: Semi-Supervised Generative Models for Multi-Agent Trajectories

Original: Abstract
Revised: Abstract

Original: Analyzing the spatiotemporal behavior of multiple agents is of great interest to many communities.
Revised: Analyzing the spatiotemporal behavior of multiple agents is of great interest to many communities.

Original: Existing probabilistic models in this realm are formalized either in an unsupervised framework, where the latent space is described by discrete or continuous variables, or in a supervised framework, where weakly preserved labels add explicit information to continuous latent representations.
Revised: Existing probabilistic models in this realm are formalized either in an unsupervised framework, where the latent space is described by discrete or continuous variables, or in a supervised framework, where weakly preserved labels add explicit information to continuous latent representations.

Original: To overcome inherent limitations, we propose a novel objective function for processing multi-agent trajectories based on semi-supervised variational autoencoders, where equivariance and interaction of agents are captured via customized graph networks.
Revised: To overcome inherent limitations, we propose a novel objective function for processing multi-agent trajectories based on semi-supervised variational autoencoders, where equivariance and interaction of agents are captured via customized graph networks.

Original: The resulting architecture disentangles discrete and continuous latent effects and provides a natural solution for injecting expensive domain knowledge into interactive sequential systems.
Revised: The resulting architecture disentangles discrete and continuous latent effects and provides a natural solution for injecting expensive domain knowledge into interactive sequential systems.

Original: Empirically, our model not only outperforms various state-of-the-art baselines in trajectory forecasting, but also learns to effectively leverage unsupervised multi-agent sequences for classiﬁcation tasks on interactive real-world sports datasets.
Revised: Empirically, our model not only outperforms various state-of-the-art baselines in trajectory forecasting, but also learns to effectively leverage unsupervised multi-agent sequences for classiﬁcation tasks on interactive real-world sports datasets.

Original:  Introduction
Revised:  Introduction

Original: Analyzing the spatiotemporal behavior of multiple agents bears a great deal of value in many domains such as autonomous driving [51, 16], public transportation [32, 61], migration [41, 24] or sports analytics [56, 12].
Revised: Analyzing the spatiotemporal behavior of multiple agents bears a great deal of value in many domains such as autonomous driving [45, 15], public transportation [29, 54], migration [37, 23] or sports analytics [50, 11].

Original: Particularly, the detection of collective patterns across time and space is of interest to many communities but non-trivial inter-dependencies between agents render estimating the inherent multi-modal distributions often difﬁcult.
Revised: Particularly, the detection of collective patterns across time and space is of interest to many communities but non-trivial inter-dependencies between agents render estimating the inherent multi-modal distributions often difﬁcult.

Original: Existing generative approaches to modeling sequential multi-agent data thus capitalize on variants from graph networks [58, 15] and (sequential) variational autoencoders [28, 9]; being purely unsupervised, their functionality is grounded in uncovering the implicit modular structure in agent-wise latent variables directly from the observed trajectories via approximate inference [30, 54, 62, 18, 17, 35].
Revised: Existing generative approaches to modeling sequential multi-agent data thus capitalize on variants from graph networks [52, 14] and (sequential) variational autoencoders [25, 8]; being purely unsupervised, their functionality is grounded in uncovering the implicit modular structure in agent-wise latent variables directly from the observed trajectories via approximate inference [27, 48, 55, 17, 16, 32].

Original: Although a generative view on the problem is very appealing, approaches in this regime either lack interpretable and controllable latent factors (e.g., [62]), thus being limited in practical settings, or put a strong focus on relationship discovery (e.g., [30]), which limits their generative capacity.
Revised: Although a generative view on the problem is very appealing, approaches in this regime either lack interpretable and controllable latent factors (e.g., [55]), thus being limited in practical settings, or put a strong focus on relationship discovery (e.g., [27]), which limits their generative capacity.

Original: An alternative is offered by supervised approaches that aim to counteract these issues by incorporating discrete behavioral indicators into the generation process [67, 68, 55].
Revised: An alternative is offered by supervised approaches that aim to counteract these issues by incorporating discrete behavioral indicators into the generation process [60, 61, 49].

Original: Ideally, these labels encode implicit expert knowledge that would be learned by the model and substantially expand the scope of information captured in latent space.
Revised: Ideally, these labels encode implicit expert knowledge that would be learned by the model and substantially expand the scope of information captured in latent space.

Original: Instead of arguing in favor of expensive and tedious manual annotations of spatiotemporal data, however, these approaches propose to incorporate heuristic surrogates as makeshift labels into the problem [69, 67, 42].
Revised: Instead of arguing in favor of expensive and tedious manual annotations of spatiotemporal data, however, these approaches propose to incorporate heuristic surrogates as makeshift labels into the problem [62, 60, 38].

Original: While reporting impressive predictive accuracies, they require the existence of label sequences for all training instances and cannot include inexpensive and possibly abundant unlabeled data due to their purely supervised nature.
Revised: While reporting impressive predictive accuracies, they require the existence of label sequences for all training instances and cannot include inexpensive and possibly abundant unlabeled data due to their purely supervised nature.

Original: To close this gap, we propose semi-supervised variational autoencoders for spatiotemporal multiagent problems to process (weakly) labeled as well as unlabeled data.
Revised: To close this gap, we propose semi-supervised variational autoencoders for spatiotemporal multiagent problems to process (weakly) labeled as well as unlabeled data.

Original: Our contribution generalizes the class of semi-supervised variational autoencoders [29] to spatiotemporal domains by using ideas from variational recurrent neural networks [9].
Revised: Our contribution generalizes the class of semi-supervised variational autoencoders [26] to spatiotemporal domains by using ideas from variational recurrent neural networks [8].

Original: At a high level, the derived objective function subsumes previous approaches into a uniﬁed framework via a disentangled latent space that can incorporate supervision signals in arbitrary quantities for structuring the discrete latent subspace.
Revised: At a high level, the derived objective function subsumes previous approaches into a uniﬁed framework via a disentangled latent space that can incorporate supervision signals in arbitrary quantities for structuring the discrete latent subspace.

Original: Additional adaptive graph network layers provide order in- and equivariance of agents and render the proposed approach appropriate for multi-agent scenarios.
Revised: Additional adaptive graph network layers provide order in- and equivariance of agents and render the proposed approach appropriate for multi-agent scenarios.

Original: The resulting semi-supervised approach is applicable to a wide range of problems including the generation of collective movements as well as classifying situations of interest.
Revised: The resulting semi-supervised approach is applicable to a wide range of problems including the generation of collective movements as well as classifying situations of interest.

Original: Empirically, our contribution signiﬁcantly advances recent baselines in generation and classiﬁcation tasks on interactive real-world data.
Revised: Empirically, our contribution signiﬁcantly advances recent baselines in generation and classiﬁcation tasks on interactive real-world data.

Original:  Related Work
Revised:  Related Work

Original: Semi-supervised generative models Semi-supervised variational autoencoders have been originally studied by [29] who propose to incorporate additional labels in latent space.
Revised: Semi-supervised generative models Semi-supervised variational autoencoders have been originally studied by [26] who propose to incorporate additional labels in latent space.

Original: Related approaches introduce auxiliary variables that leave the original model unchanged but increase the ﬂexibility of the variational posterior [36].
Revised: Related approaches introduce auxiliary variables that leave the original model unchanged but increase the ﬂexibility of the variational posterior [33].

Original: [26] propose to explicitly capture label characteristics in latent space instead of the label values themselves and [43] study generalizations that allow for learning more complex dependency structures.
Revised: [24] propose to explicitly capture label characteristics in latent space instead of the label values themselves and [39] study generalizations that allow for learning more complex dependency structures.

Original: However, all the above approaches are introduced only for static domains.
Revised: However, all the above approaches are introduced only for static domains.

Original: Our contribution constitutes a generalization of the above approaches to spatiotemporal domains.
Revised: Our contribution constitutes a generalization of the above approaches to spatiotemporal domains.

Original: Sequential generative models Approaches designed for predicting agent trajectories are frequently based on ideas from sequential generative models to encode temporal dependencies and address the stochasticity inherent in future predictions.
Revised: Sequential generative models Approaches designed for predicting agent trajectories are frequently based on ideas from sequential generative models to encode temporal dependencies and address the stochasticity inherent in future predictions.

Original: Sequential extensions of variational autoencoders with dynamic latent variables include [4, 9, 14, 1].
Revised: Sequential extensions of variational autoencoders with dynamic latent variables include [4, 8, 13, 1].

Original: Particularly, the VRNN [9] is related to our contribution, as both deploy a VAE instance per time step conditioned on a recurrent state.
Revised: Particularly, the VRNN [8] is related to our contribution, as both deploy a VAE instance per time step conditioned on a recurrent state.

Original: However, their work lacks modeling of a social dimension as well as means to integrate (potentially expensive) discrete behavioral indicators.
Revised: However, their work lacks modeling of a social dimension as well as means to integrate (potentially expensive) discrete behavioral indicators.

Original: Other approaches associate a single global latent variable with each sequence [5, 19, 11, 10].
Revised: Other approaches associate a single global latent variable with each sequence [5, 18, 10, 9].

Original: Graph neural networks Deep graph-based (GNNs) approaches [52] are a natural methodological choice for applications requiring modeling datasets composed of sets.
Revised: Graph neural networks Deep graph-based (GNNs) approaches [46] are a natural methodological choice for applications requiring modeling datasets composed of sets.

Original: GNNs operate by learning a chain of hidden representations for each node through an iterative process that relies on aggregating messages along edges in each network layer.
Revised: GNNs operate by learning a chain of hidden representations for each node through an iterative process that relies on aggregating messages along edges in each network layer.

Original: Prominent instantiations such as GCN [31], GraphSAGE [20], GAT [58], and others [15, 57, 22, 3, 66, 60] mainly differ in their notion of message passing and neighborhood aggregation [63].
Revised: Prominent instantiations such as GCN [28], GraphSAGE [19], GAT [52], and others [14, 51, 21, 3, 59, 53] mainly differ in their notion of message passing and neighborhood aggregation [56].

Original: GNN layers can be stacked and allow for context-based and adaptive neighborhoods for each node by incorporating particularly designed skip connections [60].
Revised: GNN layers can be stacked and allow for context-based and adaptive neighborhoods for each node by incorporating particularly designed skip connections [53].

Original: See [7] for a discussion on the function approximation capabilities of this neural network family.
Revised: See [7] for a discussion on the function approximation capabilities of this neural network family.

Original: Generative models for sequential multi-agent data Given the potential beneﬁt across various domains, a great deal of recent publications focus on movements of pedestrians or self-driving cars [39, 51, 16, 40, 64, 50, 2, 23, 65, 34].
Revised: Generative models for sequential multi-agent data Given the potential beneﬁt across various domains, a great deal of recent publications focus on movements of pedestrians or self-driving cars [35, 45, 15, 36, 57, 44, 2, 22, 58, 31].

Original: [38], however, show that these standard benchmark datasets generally exhibit weak social interaction patterns, so in the remainder, we focus on methods using team sport data.
Revised: [34], however, show that these standard benchmark datasets generally exhibit weak social interaction patterns, so in the remainder, we focus on methods using team sport data.

Original: Related approaches show that learning and computing realistic rollouts signiﬁcantly beneﬁt from the incorporation of makeshift annotations and heuristic surrogates into the problem [69, 67, 42].
Revised: Related approaches show that learning and computing realistic rollouts signiﬁcantly beneﬁt from the incorporation of makeshift annotations and heuristic surrogates into the problem [62, 60, 38].

Original: [13, 35] propose an architecture based on conditional VAEs [53] to model distributions of future movements of basketball players.
Revised: [12, 32] propose an architecture based on conditional VAEs [47] to model distributions of future movements of basketball players.

Original: [62, 54] combine ideas from graph networks [15] and
Revised: [55, 48] combine ideas from graph networks [14] and

Original: VRNNs [9] into a uniﬁed framework aiming at modeling data from basketball and soccer.
Revised: VRNNs [8] into a uniﬁed framework aiming at modeling data from basketball and soccer.

Original: Other related approaches [33, 30, 18, 17] aim to explicitly infer discrete latent variables to represent interaction types while executing a trajectory prediction task.
Revised: Other related approaches [30, 27, 17, 16] aim to explicitly infer discrete latent variables to represent interaction types while executing a trajectory prediction task.

Original: Hence, existing generative models for spatiotemporal pattern detection focus on multi-agent trajectory prediction in fully-(un)supervised settings.
Revised: Hence, existing generative models for spatiotemporal pattern detection focus on multi-agent trajectory prediction in fully-(un)supervised settings.

Original: Our contribution represents the ﬁrst semi-supervised generalization of variational autoencoders for spatiotemporal domains, which allows addressing tasks beyond representational or generative modeling involving expensive annotation processes.
Revised: Our contribution represents the ﬁrst semi-supervised generalization of variational autoencoders for spatiotemporal domains, which allows addressing tasks beyond representational or generative modeling involving expensive annotation processes.

Original:  Methods 3.1 Preliminaries
Revised:  Methods 3.1 Preliminaries

Original: Variational autoencoders (VAEs) [28, 45] allow to capture intrinsic conditional dependencies and are often used in structured problems.
Revised: Variational autoencoders (VAEs) [25, 40] allow to capture intrinsic conditional dependencies and are often used in structured problems.

Original: Essentially, they consist of a θ -parameterized generative model p θ ( x , z ) =
Revised: Essentially, they consist of a θ -parameterized generative model p θ ( x , z ) =

Original: p ( z ) p θ ( x | z ) and an associated φ -parameterized variational distribution q φ ( z | x ) .
Revised: p ( z ) p θ ( x | z ) and an associated φ -parameterized variational distribution q φ ( z | x ) .

Original: Optimal parameters { θ, φ } are obtained via maximizing the variational lower bound on the marginal likelihood, given by
Revised: Optimal parameters { θ, φ } are obtained via maximizing the variational lower bound on the marginal likelihood, given by

Original: [Equation]
Revised: [Equation]

Original: It is often helpful to augment partially observed discrete labels y into the model to guide the generating process [29].
Revised: It is often helpful to augment partially observed discrete labels y into the model to guide the generating process [26].

Original: The resulting model p ( x , z , y ) =
Revised: The resulting model p ( x , z , y ) =

Original: p ( x | z , y ) p ( y ) p ( z ) then acts on labeled and unlabeled instances.
Revised: p ( x | z , y ) p ( y ) p ( z ) then acts on labeled and unlabeled instances.

Original: Consequentially, the variational distribution q φ ( y, z | x ) =
Revised: Consequentially, the variational distribution q φ ( y, z | x ) =

Original: q φ ( y | x ) q φ ( z | y, x ) needs to be deﬁned over both quantities ( z and y ).
Revised: q φ ( y | x ) q φ ( z | y, x ) needs to be deﬁned over both quantities ( z and y ).

Original: Semi-supervised VAEs (SSVAE) are trained by maximizing
Revised: Semi-supervised VAEs (SSVAE) are trained by maximizing

Original: [Equation]
Revised: [Equation]

Original: where L ( x , y )
Revised: where L ( x , y )

Original: =
Revised: =

Original: E q φ ( z | x ,y )
Revised: E q φ ( z | x ,y )

Original: [log p θ ( x | y, z ) + log p ( y ) + log p ( z ) − log q φ ( z | x , y )] and U ( x )
Revised: [log p θ ( x | y, z ) + log p ( y ) + log p ( z ) − log q φ ( z | x , y )] and U ( x )

Original: = (cid:80)
Revised: = (cid:80)

Original: y q φ ( y | x )( L ( x , y ))
Revised: y q φ ( y | x )( L ( x , y ))

Original: + H ( q φ ( y | x )) denote (variational) lower bounds on labeled and unlabeled instances, respectively, and λ is a balancing term.
Revised: + H ( q φ ( y | x )) denote (variational) lower bounds on labeled and unlabeled instances, respectively, and λ is a balancing term.

Original: The original motivation for this framework was inspired by semi-supervised classiﬁcation tasks.
Revised: The original motivation for this framework was inspired by semi-supervised classiﬁcation tasks.

Original: However, SSVAEs are also frequently employed for learning meaningful representations and generating new data, thus rendering it a suitable methodological foundation for performing diverse sets of tasks.
Revised: However, SSVAEs are also frequently employed for learning meaningful representations and generating new data, thus rendering it a suitable methodological foundation for performing diverse sets of tasks.

Original: The dependency structures of q φ and p θ are displayed visually as part of Figure 1.
Revised: The dependency structures of q φ and p θ are displayed visually as part of Figure 1.

Original: 3.2 Problem Formulation
Revised: 3.2 Problem Formulation

Original: We are interested in modeling spatiotemporal multi-agent scenarios and focus on positions x
Revised: We are interested in modeling spatiotemporal multi-agent scenarios and focus on positions x

Original: ( a ) t ∈ Rof an agent a ∈
Revised: ( a ) t ∈ Rof an agent a ∈

Original: A over time 1 ≤ t ≤ T , denoted by x ( a ) ≤ T .
Revised: A over time 1 ≤ t ≤ T , denoted by x ( a ) ≤ T .

Original: We assume that a non-empty subset of agents in A is present in any observed segment and collect their trajectories (in random order) to form collective movements x ≤ T :=
Revised: We assume that a non-empty subset of agents in A is present in any observed segment and collect their trajectories (in random order) to form collective movements x ≤ T :=

Original: { x ( a ) ≤ T | a ∈ A} .
Revised: { x ( a ) ≤ T | a ∈ A} .

Original: We will make use of velocities d x /dt and linearized motion ∆ x t = x t (cid:48) − x t , for t (cid:48) >
Revised: We will make use of velocities d x /dt and linearized motion ∆ x t = x t (cid:48) − x t , for t (cid:48) >

Original: t , of agents in the remainder.
Revised: t , of agents in the remainder.

Original: In addition, we introduce discrete labels y
Revised: In addition, we introduce discrete labels y

Original: ( a ) t ∈ Y associated with agent
Revised: ( a ) t ∈ Y associated with agent

Original: a ∈ A at time t .
Revised: a ∈ A at time t .

Original: The collection of sequences y ≤ T :=
Revised: The collection of sequences y ≤ T :=

Original: { y ( a ) ≤ T | a ∈ A} may be best thought of as discriminative causes of variation in the corresponding trajectories x ≤ T .
Revised: { y ( a ) ≤ T | a ∈ A} may be best thought of as discriminative causes of variation in the corresponding trajectories x ≤ T .

Original: In practice, y ≤ T may contain anything that aids in generating multi-agent rollouts, including behavioral indicators, long-term goals, or current tasks.
Revised: In practice, y ≤ T may contain anything that aids in generating multi-agent rollouts, including behavioral indicators, long-term goals, or current tasks.

Original: Finally, observations x ≤ T accompanied by labels y ≤ T assemble the labeled share of the data, denoted by D L = { ( x ≤ T , y ≤ T ) n } 1 ≤ n ≤ N , while unlabeled observations are collected in
Revised: Finally, observations x ≤ T accompanied by labels y ≤ T assemble the labeled share of the data, denoted by D L = { ( x ≤ T , y ≤ T ) n } 1 ≤ n ≤ N , while unlabeled observations are collected in

Original: D U
Revised: D U

Original: = { ( x ≤ T ) m } 1 ≤ m ≤ M .
Revised: = { ( x ≤ T ) m } 1 ≤ m ≤ M .

Original: We refer to all data by D = D L ∪ D U .
Revised: We refer to all data by D = D L ∪ D U .

Original: We aim at estimating the underlying distribution that generated the observations in D .
Revised: We aim at estimating the underlying distribution that generated the observations in D .

Original: A straight forward solution is to maximize the log likelihood given by
Revised: A straight forward solution is to maximize the log likelihood given by

Original: [Equation]
Revised: [Equation]

Original: However, human movement is inherently non-determenistic and multi-modal.
Revised: However, human movement is inherently non-determenistic and multi-modal.

Original: Thus, it is common to model the stochasticity inherent in future agent behaviors via latent representations of VAEs.
Revised: Thus, it is common to model the stochasticity inherent in future agent behaviors via latent representations of VAEs.

Original: We propose to lower-bound Eq. 1 as follows.
Revised: We propose to lower-bound Eq. 1 as follows.

Original: 3.3 SSVAEs for Modeling Multi-Agent Trajectories
Revised: 3.3 SSVAEs for Modeling Multi-Agent Trajectories

Original: We associate disentangled latent realizations { z t , y t } with each time step t of the segment while injecting the dependency structures introduced in Section 3.1 for the generative and inference parts.
Revised: We associate disentangled latent realizations { z t , y t } with each time step t of the segment while injecting the dependency structures introduced in Section 3.1 for the generative and inference parts.

Original: To model temporal dependencies, we additionally condition the model components on the past observations x <t and latent variables z <t and y <t (cmp.
Revised: To model temporal dependencies, we additionally condition the model components on the past observations x <t and latent variables z <t and y <t (cmp.

Original: The joint distribution factorizes into p θ ( x ≤ T , z ≤ T , y ≤ [Equation]
Revised: The joint distribution factorizes into p θ ( x ≤ T , z ≤ T , y ≤ T ) =

Original: ) = (cid:89)
Revised: (cid:89)

Original: [Equation]
Revised: [Equation]

Original: with generating distribution p θ ( x t | x <t , z ≤ t , y ≤ t ) and prior distributions p θ ( z t | x <t , z <t , y <t ) and p θ ( y t | x <t , z <t , y <t ) for latents z t and y t .
Revised: with generating distribution p θ ( x t | x <t , z ≤ t , y ≤ t ) and prior distributions p θ ( z t | x <t , z <t , y <t ) and p θ ( y t | x <t , z <t , y <t ) for latents z t and y t .

Original: Note that in our sequential setting, the priors are included in the optimization rather than being represented by ﬁxed distributions to account for latent dynamics and effective sampling at inference time.
Revised: Note that in our sequential setting, the priors are included in the optimization rather than being represented by ﬁxed distributions to account for latent dynamics and effective sampling at inference time.

Original: To derive the variational distributions, we differentiate between observed labels y and latent variables ˜
Revised: To derive the variational distributions, we differentiate between observed labels y and latent variables ˜

Original: y in the remainder.
Revised: y in the remainder.

Original: Using the factorization introduced in Section 3.1 yields
Revised: Using the factorization introduced in Section 3.1 yields

Original: [Equation]
Revised: [Equation]

Original: Labeled instances render reasoning over latents ˜ y ≤ T unnecessary since the true y ≤ T is already known.
Revised: Labeled instances render reasoning over latents ˜ y ≤ T unnecessary since the true y ≤ T is already known.

Original: Hence, the right-hand side in Eqn (2) can be ignored when sampling from D L .
Revised: Hence, the right-hand side in Eqn (2) can be ignored when sampling from D L .

Original: We arrive at the following results and refer to Appendix A for the proofs.
Revised: We arrive at the following results and refer to Appendix ?? for the proofs.

Original: Theorem 1.
Revised: Theorem 1.

Original: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by
Revised: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by

Original: [Equation]
Revised: [Equation]

Original: Correspondingly, for unlabeled instances drawn from D U , sequential label information need to be estimated as shown in the following theorem.
Revised: Correspondingly, for unlabeled instances drawn from D U , sequential label information need to be estimated as shown in the following theorem.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let H ( β ) be the entropy of quantity β .
Revised: Let H ( β ) be the entropy of quantity β .

Original: A lower bound on log p θ ( x ≤ T ) in Eqn (1) is given by
Revised: A lower bound on log p θ ( x ≤ T ) in Eqn (1) is given by

Original: [Equation]
Revised: [Equation]

Original: Eqn (4) encodes the behavior of object q φ ( y ≤ T | x ≤ T ) when dealing with unsupervised data.
Revised: Eqn (4) encodes the behavior of object q φ ( y ≤ T | x ≤ T ) when dealing with unsupervised data.

Original: Since data likelihood log p θ ( x t | x <t , , z ≤ t , y ≤ t ) tends to comprise the largest factor in L ( x t , y t ) , taking the expectation wrt.
Revised: Since data likelihood log p θ ( x t | x <t , , z ≤ t , y ≤ t ) tends to comprise the largest factor in L ( x t , y t ) , taking the expectation wrt.

Original: q φ (˜ y t | x ≤ t , z <t , ˜ y <t ) encourages the model to realize the highest probability mass for the latent label valuel y ∈ Y that incurs the smallest reconstruction loss compared to other candidates in label space.
Revised: q φ (˜ y t | x ≤ t , z <t , ˜ y <t ) encourages the model to realize the highest probability mass for the latent label valuel y ∈ Y that incurs the smallest reconstruction loss compared to other candidates in label space.

Original: This is a desirable property that reﬂects our assumption of data generation.
Revised: This is a desirable property that reﬂects our assumption of data generation.

Original: The full evidence lower bound (ELBO) is obtained by combining the lower bounds for all data D ,
Revised: The full evidence lower bound (ELBO) is obtained by combining the lower bounds for all data D ,

Original: [Equation]
Revised: [Equation]

Original: This formalization focuses on either fully labeled or unlabeled observations.
Revised: This formalization focuses on either fully labeled or unlabeled observations.

Original: However, a more general formulation can be obtained to also allow for partially labeled data points.
Revised: However, a more general formulation can be obtained to also allow for partially labeled data points.

Original: 3.4 Extension to Classiﬁcation Settings
Revised: 3.4 Extension to Classiﬁcation Settings

Original: Intrinsic to our contribution is an encoding module that argues over the label space, q φ ( y ≤ T | x ≤ T ) , and hence can be used as a classiﬁcation network annotating new observations.
Revised: Intrinsic to our contribution is an encoding module that argues over the label space, q φ ( y ≤ T | x ≤ T ) , and hence can be used as a classiﬁcation network annotating new observations.

Original: However, the
Revised: However, the

Original: ELBO in Eqn (5) is oblivious to classiﬁcation and simply ignores relevant gradient updates for
Revised: ELBO in Eqn (5) is oblivious to classiﬁcation and simply ignores relevant gradient updates for

Original: J S − MAT ( x t , y t ) by discarding Eqn (2).
Revised: J S − MAT ( x t , y t ) by discarding Eqn (2).

Original: This is clearly inappropriate for semi-supervised learning where the overall goal is to learn an unknown mapping x
Revised: This is clearly inappropriate for semi-supervised learning where the overall goal is to learn an unknown mapping x

Original: (cid:55)→ y .
Revised: (cid:55)→ y .

Original: Following [29], we circumvent this by heuristically incorporating Eqn (2) into the label-dependent objective.
Revised: Following [26], we circumvent this by heuristically incorporating Eqn (2) into the label-dependent objective.

Original: Hence, the full training criterion to learn { θ, φ } is given by
Revised: Hence, the full training criterion to learn { θ, φ } is given by

Original: [Equation]
Revised: [Equation]

Original: where λ 1 balances generative and discriminative learning and λ 

Revised: where λ 1 balances generative and discriminative learning and λ

Original: the contribution of labeled and unlabeled data, respectively.
Revised: the contribution of labeled and unlabeled data, respectively.

Original: Thus, J MAT contains fully (un)supervised modeling tasks as special cases via adjusting λ 1 , λ 2 accordingly.
Revised: Thus, J MAT contains fully (un)supervised modeling tasks as special cases via adjusting λ 1 , λ 2 accordingly.

Original: Note that the resulting supervised part in Eqn 6 (right part) differs from its unsupervised counterpart only in observing y t instead of factorizing over Y , which results in computing the log-likelihood (auxiliary loss) log q φ ( y t | x ≤ t , z <t , y <t ) instead of the entropy H ( q φ ( y t | x ≤ t , z <t , y <t )) .
Revised: Note that the resulting supervised part in Eqn 6 (right part) differs from its unsupervised counterpart only in observing y t instead of factorizing over Y , which results in computing the log-likelihood (auxiliary loss) log q φ ( y t | x ≤ t , z <t , y <t ) instead of the entropy H ( q φ ( y t | x ≤ t , z <t , y <t )) .

Original: 3.5 Design Choices
Revised: 3.5 Design Choices

Original: As described in Section 3.2, we aim to learn a distribution over possible sequences of agent sets that may differ in share of discrete annotations (and cardinality).
Revised: As described in Section 3.2, we aim to learn a distribution over possible sequences of agent sets that may differ in share of discrete annotations (and cardinality).

Original: However, the previously derived objective functions only account for temporal dependencies, so that independence of the agent dimension is required to realize permutation-invariant models.
Revised: However, the previously derived objective functions only account for temporal dependencies, so that independence of the agent dimension is required to realize permutation-invariant models.

Original: This assumption is trivially inappropriate for interactive systems where future agent movements need to be coordinated with other agents.
Revised: This assumption is trivially inappropriate for interactive systems where future agent movements need to be coordinated with other agents.

Original: Thus, it is common to propose some form of graph encoding strategy [52] to account for both, equivariance and agent interactions, respectively.
Revised: Thus, it is common to propose some form of graph encoding strategy [46] to account for both, equivariance and agent interactions, respectively.

Original: Intuitively, optimal interaction modeling allows to adaptively accommodate structural information from different levels of granularity for each agent.
Revised: Intuitively, optimal interaction modeling allows to adaptively accommodate structural information from different levels of granularity for each agent.

Original: In team sports, for example, agent nodes should be able to capture immediate player inﬂuences as well as holistic team strategies dependent on the interaction structure best suited for the task at-hand.
Revised: In team sports, for example, agent nodes should be able to capture immediate player inﬂuences as well as holistic team strategies dependent on the interaction structure best suited for the task at-hand.

Original: Motivated by the previous considerations, we customize an GNN architecture that aims to mimic the desired behavior.
Revised: Motivated by the previous considerations, we customize an GNN architecture that aims to mimic the desired behavior.

Original: Speciﬁcally, we stack attention-based GNN layers [58] with particularly designed skip-connections [60] and construct the assumed graph structure using the k (spatially) closest agents.
Revised: Speciﬁcally, we stack attention-based GNN layers [52] with particularly designed skip-connections [53] and construct the assumed graph structure using the k (spatially) closest agents.

Original: There are various possibilities to incorporate graph-based approaches into the overall scheme 1 .
Revised: There are various possibilities to incorporate graph-based approaches into the overall scheme 1 .

Original: An efﬁcient way to model the distributions deﬁning J MAT is via hidden agent states h ( a ) t conditioned on representations
Revised: An efﬁcient way to model the distributions deﬁning J MAT is via hidden agent states h ( a ) t conditioned on representations

Original: o ( a ) t − 1 ,
Revised: o ( a ) t − 1 ,

Original: [Equation]
Revised: [Equation]

Original: with o
Revised: with o

Original: [Equation]
Revised: [Equation]

Original: where f rnn denotes an RNN transition function, GNN is the described GAT-based GNN and o t =
Revised: where f rnn denotes an RNN transition function, GNN is the described GAT-based GNN and o t =

Original: { o ( a ) t | a ∈ A} refers to the set of updated agent representations 2 .
Revised: { o ( a ) t | a ∈ A} refers to the set of updated agent representations 2 .

Original: Since the elements in set o t aggregate neighboring information of the RNN outputs h t via graph networks, the inferred feature vectors describe the entire interactive past of individual agents.
Revised: Since the elements in set o t aggregate neighboring information of the RNN outputs h t via graph networks, the inferred feature vectors describe the entire interactive past of individual agents.

Original: While it is reasonable to additionally enforce intra-timestep dependencies on the latent variables using encoding and decoding
Revised: While it is reasonable to additionally enforce intra-timestep dependencies on the latent variables using encoding and decoding

Original: GNN modules, we argue that capturing past spatiotemporal patterns is sufﬁciently informative for high-frequency data [48].
Revised: GNN modules, we argue that capturing past spatiotemporal patterns is sufﬁciently informative for high-frequency data [43].

Original: Hence, assuming conditional independence within t , the joint movement distribution factorizes across the agent dimension and is given by
Revised: Hence, assuming conditional independence within t , the joint movement distribution factorizes across the agent dimension and is given by

Original: [Equation]
Revised: [Equation]

Original: where ψ t = { ψ ( a ) t | a ∈ A} are the distribution parameters and ψ ( a ) t = ϕ
Revised: where ψ t = { ψ ( a ) t | a ∈ A} are the distribution parameters and ψ ( a ) t = ϕ

Original: ( z ( a ) t , y ( a ) t , o ( a ) t − 1 ) .
Revised: ( z ( a ) t , y ( a ) t , o ( a ) t − 1 ) .

Original: The computational logic of the full architecture is schematically depicted in Fig.
Revised: The computational logic of the full architecture is schematically depicted in Fig.

Original: 1 and 2.
Revised: 1 and 2.

Original:  Empirical Evaluation
Revised:  Empirical Evaluation

Original: For evaluation, we focus on team sports as the coordination of players renders these tasks more difﬁcult than other domains [38].
Revised: For evaluation, we focus on team sports as the coordination of players renders these tasks more difﬁcult than other domains [34].

Original: Hence, we experiment on STATS SportVU NBA 4 for comparison, and tracking data from soccer games of the german national team.
Revised: Hence, we experiment on STATS SportVU NBA 4 for comparison, and tracking data from soccer games of the german national team.

Original: As detailed in Section 3.2, we use agent velocities as input to all models and assume linear motion between consecutive observations.
Revised: As detailed in Section 3.2, we use agent velocities as input to all models and assume linear motion between consecutive observations.

Original: For NBA, we adopt the experimental setup and processing strategy from [42] 5 .
Revised: For NBA, we adopt the experimental setup and processing strategy from [38] 5 .

Original: The STATS SportVU NBA data comprises tracking positions of offensive plays from the 2016 NBA regular season covering more than 1200 different games where game segment are given by sequences of length 50 and contain two-dimensional positions of all agents ( 10 players and ball) sampled atframes per second.
Revised: The STATS SportVU NBA data comprises tracking positions of offensive plays from the 2016 NBA regular season covering more than 1200 different games where game segment are given by sequences of length 50 and contain two-dimensional positions of all agents ( 10 players and ball) sampled atframes per second.

Original: The data is split into 60% training, 20% validation, and 20% test sets.
Revised: The data is split into 60% training, 20% validation, and 20% test sets.

Original: All data is translated so that the origin of the underlying coordinate system is mapped onto the top-left corner.
Revised: All data is translated so that the origin of the underlying coordinate system is mapped onto the top-left corner.

Original: The soccer data consists of 12 matches where positions of players and ball are sampled at 25 frames per second.
Revised: The soccer data consists of 12 matches where positions of players and ball are sampled at 25 frames per second.

Original: The tracking data is accompanied by manual event annotations that we will also make use of in the remainder.
Revised: The tracking data is accompanied by manual event annotations that we will also make use of in the remainder.

Original: Models are trained on eight matches, the remaining four are distributed evenly into validation and test data (two matches each).
Revised: Models are trained on eight matches, the remaining four are distributed evenly into validation and test data (two matches each).

Original: 4.1 Baselines
Revised: 4.1 Baselines

Original: We evaluate versus several baselines on two distinct tasks: trajectory forecasting (Section 4.2 and 4.4) and spatiotemporal classiﬁcation (Section 4.3 and 4.4).
Revised: We evaluate versus several baselines on two distinct tasks: trajectory forecasting (Section 4.2 and 4.4) and spatiotemporal classiﬁcation (Section 4.3 and 4.4).

Original: Since our approach constitutes the ﬁrst semisupervised model for label predictions in multi-agent scenarios, we need to rely on self-constructed supervised baselines for scenarios where only a few labels are available, which we discuss in more detail in the relevant sections.
Revised: Since our approach constitutes the ﬁrst semisupervised model for label predictions in multi-agent scenarios, we need to rely on self-constructed supervised baselines for scenarios where only a few labels are available, which we discuss in more detail in the relevant sections.

Original: [Table]
Revised: [Table]

Original: By contrast, trajectory forecasting is an active topic in different ﬁelds (see, e.g., [47] for an overview), which allows comparisons with a wide range of baseline models.
Revised: By contrast, trajectory forecasting is an active topic in different ﬁelds (see, e.g., [42] for an overview), which allows comparisons with a wide range of baseline models.

Original: It is difﬁcult to determine the current state-of-the-art in generative methods for sports tracking data because there is no consistent setting in terms of forecasting horizon or data (see e.g., [35, 42]).
Revised: It is difﬁcult to determine the current state-of-the-art in generative methods for sports tracking data because there is no consistent setting in terms of forecasting horizon or data (see e.g., [32, 38]).

Original: While some approaches are restricted to making predictions for a predetermined horizon, our method learns spatiotemporal representations autoregressively and thus can be used for the full range of prediction scenarios.
Revised: While some approaches are restricted to making predictions for a predetermined horizon, our method learns spatiotemporal representations autoregressively and thus can be used for the full range of prediction scenarios.

Original: To include a variety of methods, we conduct two sets of experiments over short ( 10 timesteps) and long ( 40 timesteps) prediction horizons.
Revised: To include a variety of methods, we conduct two sets of experiments over short ( 10 timesteps) and long ( 40 timesteps) prediction horizons.

Original: We benchmark against the most recent and methodologically important work described below.
Revised: We benchmark against the most recent and methodologically important work described below.

Original: Fully supervised generative models such as DAG-Net [42] and Weak-Sup [67] use weak labels in the form of agent objectives (heuristically inferred prior to model training) to improve trajectory prediction.
Revised: Fully supervised generative models such as DAG-Net [38] and Weak-Sup [60] use weak labels in the form of agent objectives (heuristically inferred prior to model training) to improve trajectory prediction.

Original: In these settings, an agent’s objective at each timestep t is to reach the next area where the observed movement speed falls below a predeﬁned threshold.
Revised: In these settings, an agent’s objective at each timestep t is to reach the next area where the observed movement speed falls below a predeﬁned threshold.

Original: Adopting the experimental framework of [42] allows us to make further comparisons with Social-Ways [2] and STGAT [23] without additional experiments.
Revised: Adopting the experimental framework of [38] allows us to make further comparisons with Social-Ways [2] and STGAT [22] without additional experiments.

Original: STGAT [23] augments a standard GAT with an LSTM [21] to capture both spatial and temporal dependencies of social interactions while Social-Ways uses Info-GAN [6] to learn multimodal predictive distributions.
Revised: STGAT [22] augments a standard GAT with an LSTM [20] to capture both spatial and temporal dependencies of social interactions while Social-Ways uses Info-GAN [6] to learn multimodal predictive distributions.

Original: The majority of generative models for sequential multiagent systems operate in a fully-unsupervised fashion.
Revised: The majority of generative models for sequential multiagent systems operate in a fully-unsupervised fashion.

Original: GVRNN [62] is a graph extension of the VRNN for explicitly modeling joint agent movements for basketball and soccer data.
Revised: GVRNN [55] is a graph extension of the VRNN for explicitly modeling joint agent movements for basketball and soccer data.

Original: dNRI [18] dynamically extracts interaction types that determine the parametrization of the decoder.
Revised: dNRI [17] dynamically extracts interaction types that determine the parametrization of the decoder.

Original: Finally, GRIN [35] recovers (static) interactions via attention using a disentangled latent space.
Revised: Finally, GRIN [32] recovers (static) interactions via attention using a disentangled latent space.

Original: 4.2 Exploring Boundary Cases for Trajectory Prediction
Revised: 4.2 Exploring Boundary Cases for Trajectory Prediction

Original: Methods We study the purely (un)supervised generation of trajectories by maximizing the respective lower bounds on NBA.
Revised: Methods We study the purely (un)supervised generation of trajectories by maximizing the respective lower bounds on NBA.

Original: For the fully-supervised variant, the weak labels described in Section 4.1 can be naturally integrated into the overall scheme via treating them as semantic concepts y ( a ) t .
Revised: For the fully-supervised variant, the weak labels described in Section 4.1 can be naturally integrated into the overall scheme via treating them as semantic concepts y ( a ) t .

Original: These target areas naturally vary over time so that the sequential label y ≤ T always denotes the next desired position for every involved agent at time t .
Revised: These target areas naturally vary over time so that the sequential label y ≤ T always denotes the next desired position for every involved agent at time t .

Original: Following related work [42, 67], we obtaindifferent areas/class labels.
Revised: Following related work [38, 60], we obtaindifferent areas/class labels.

Original: The underlying data distribution is estimated by maximizing Eqn (3).
Revised: The underlying data distribution is estimated by maximizing Eqn (3).

Original: The supervised variant of our contribution is denoted by S-MAT .
Revised: The supervised variant of our contribution is denoted by S-MAT .

Original: Although done in [42, 67], direct comparison of unsupervised and supervised generative models is inappropriate inasmuch as the latter use ground-truth labels over the entire observation period (which include future information) at prediction time.
Revised: Although done in [38, 60], direct comparison of unsupervised and supervised generative models is inappropriate inasmuch as the latter use ground-truth labels over the entire observation period (which include future information) at prediction time.

Original: Thus, we additionally benchmark against a fully unsupervised instantiation of our proposed framework lacking ground-truth information for structuring the discrete latent subspace.
Revised: Thus, we additionally benchmark against a fully unsupervised instantiation of our proposed framework lacking ground-truth information for structuring the discrete latent subspace.

Original: To encourage the model to learn semantically meaningful concepts describing distinct movement patterns, we parametrize the decoder dependent on the predicted agent class ˜ y
Revised: To encourage the model to learn semantically meaningful concepts describing distinct movement patterns, we parametrize the decoder dependent on the predicted agent class ˜ y

Original: ( a ) t .
Revised: ( a ) t .

Original: In this context, ˜ y ( a ) t may be best thought of dynamically assigned agent roles that accurately explain the observed trajectories.
Revised: In this context, ˜ y ( a ) t may be best thought of dynamically assigned agent roles that accurately explain the observed trajectories.

Original: We maximize Eqn (4) using 3 agent types and refer to the unsupervised variant as U-MAT .
Revised: We maximize Eqn (4) using 3 agent types and refer to the unsupervised variant as U-MAT .

Original: Further details for S-MAT and U-MAT are given in
Revised: Further details for S-MAT and U-MAT are given in

Original: Appendix C.
Revised: Appendix ?? .

Original: Metrics We measure the quality of the learned multimodal distribution using common standard metrics: the minimum over 20 generated samples of the average and ﬁnal L 2 distance between predicted ˆ x ( a )≤
Revised: Metrics We measure the quality of the learned multimodal distribution using common standard metrics: the minimum over 20 generated samples of the average and ﬁnal L 2 distance between predicted ˆ x ( a )

Original: [Table]
Revised: [Table]

Revised: T

Original: [Table] 
Revised: [Table] [Table]

Original: Results Tables 1 and 2 show the results.
Revised: Results Tables 1 and 2 show the results.

Original: The proposed approaches clearly outperform their peers and realize the lowest average L 2 and ﬁnal L 2 distances.
Revised: The proposed approaches clearly outperform their peers and realize the lowest average L 2 and ﬁnal L 2 distances.

Original: The MAT variants effectively accommodate not only mutual inﬂuences across agents, but also discrete generative factors, leading to better approximations of the underlying multi-modal distributions.
Revised: The MAT variants effectively accommodate not only mutual inﬂuences across agents, but also discrete generative factors, leading to better approximations of the underlying multi-modal distributions.

Original: Unsurprisingly, Table 1 also shows that defending players exhibit more structure and are easier to predict than offensive players.
Revised: Unsurprisingly, Table 1 also shows that defending players exhibit more structure and are easier to predict than offensive players.

Original: The results of an ablation study are contained in Appendix E.
Revised: The results of an ablation study are contained in Appendix ?? .

Original: The right part of Fig.
Revised: The right part of Fig.

Original: 3 displays exemplary rollouts that underline models’ ability to capture highly complex changes in movement directions.
Revised: 3 displays exemplary rollouts that underline models’ ability to capture highly complex changes in movement directions.

Original: The results indicate that the proposed framework denotes a valuable contribution to the large body of research that explicitly addresses modeling multi-agent data accurately.
Revised: The results indicate that the proposed framework denotes a valuable contribution to the large body of research that explicitly addresses modeling multi-agent data accurately.

Original: 4.3 Semi-Supervised Classiﬁcation
Revised: 4.3 Semi-Supervised Classiﬁcation

Original: We now extend the experimental protocol by targeting accurate label discovery in semi-supervised scenarios.
Revised: We now extend the experimental protocol by targeting accurate label discovery in semi-supervised scenarios.

Original: We refer to the full model simply as MAT where λ 0 , λ 1 > 0 and incorporate the fully supervised S-MAT ( λ 0 = 0 ) as an additional baseline.
Revised: We refer to the full model simply as MAT where λ 0 , λ 1 > 0 and incorporate the fully supervised S-MAT ( λ 0 = 0 ) as an additional baseline.

Original: We also compare to an RNN-based classiﬁcation network that addresses inter-agent dependencies by inferring hidden states of agents by the GNN architecture described in Section 3.5.
Revised: We also compare to an RNN-based classiﬁcation network that addresses inter-agent dependencies by inferring hidden states of agents by the GNN architecture described in Section 3.5.

Original: A ﬁnal softmax layer is used for classiﬁcation, and the model is optimized by minimizing the negative log-likelihood.
Revised: A ﬁnal softmax layer is used for classiﬁcation, and the model is optimized by minimizing the negative log-likelihood.

Original: We refer to this baseline as MA-RNN .
Revised: We refer to this baseline as MA-RNN .

Original: Figure 4 shows the performance of the models on NBA when using 50% labeled and 50% unlabeled examples (left side) and when using varying portions of labeled data (right side), while the supervised baselines have access only to the labeled part.
Revised: Figure 4 shows the performance of the models on NBA when using 50% labeled and 50% unlabeled examples (left side) and when using varying portions of labeled data (right side), while the supervised baselines have access only to the labeled part.

Original: The semi-supervised MAT clearly emerges as the strongest classiﬁer as it converges to the lowest generalization error during a training run and the performance beneﬁt increases via reduction of labeled data.
Revised: The semi-supervised MAT clearly emerges as the strongest classiﬁer as it converges to the lowest generalization error during a training run and the performance beneﬁt increases via reduction of labeled data.

Original: Comparing the numbers to its supervised peer ( S-MAT ) highlights the value of incorporating unlabeled data that is effectively utilized by MAT to inform classiﬁcation decisions.
Revised: Comparing the numbers to its supervised peer ( S-MAT ) highlights the value of incorporating unlabeled data that is effectively utilized by MAT to inform classiﬁcation decisions.

Original: Comparing the fully supervised approaches S-MAT and MA-RNN , the importance of an effective regularization mechanism becomes obvious; MA-RNN quickly overﬁts while the lower-bound in S-MAT acts as a regularizer and guarantees gradually increasing predictive accuracies on validation sets.
Revised: Comparing the fully supervised approaches S-MAT and MA-RNN , the importance of an effective regularization mechanism becomes obvious; MA-RNN quickly overﬁts while the lower-bound in S-MAT acts as a regularizer and guarantees gradually increasing predictive accuracies on validation sets.

Original: [Table]
Revised: [Table]

Original: 4.4 Combining Generation and Classiﬁcation in Semi-Supervised Settings
Revised: 4.4 Combining Generation and Classiﬁcation in Semi-Supervised Settings

Original: In our last set of experiments
Revised: In our last set of experiments

Original: [Table]
Revised: [Table]

Original: we study the combination of trajectory generation and classiﬁcation and resort to tracking data from elite soccer.
Revised: we study the combination of trajectory generation and classiﬁcation and resort to tracking data from elite soccer.

Original: The reason for favoring this data over NBA is that the soccer tracking data is manually annotated by experts from the data provider.
Revised: The reason for favoring this data over NBA is that the soccer tracking data is manually annotated by experts from the data provider.

Original: We focus on labels
Revised: We focus on labels

Original: Y = { pass , other ball action , shot , none }
Revised: Y = { pass , other ball action , shot , none }

Original: [Table]
Revised: [Table]

Original: where class none denotes the absence of all other labels in a frame and use 20% annotated data.
Revised: where class none denotes the absence of all other labels in a frame and use 20% annotated data.

Original: Every label is propagated to the previous ﬁve and the subsequent 30 frames.
Revised: Every label is propagated to the previous ﬁve and the subsequent 30 frames.

Original: We generate a balanced training set where half of the segments carry label none .
Revised: We generate a balanced training set where half of the segments carry label none .

Original: The main purpose of this set of experiments is to quantitatively validate the beneﬁts of including expensive human annotations in the proposed modeling framework for structuring the discrete latent subspace.
Revised: The main purpose of this set of experiments is to quantitatively validate the beneﬁts of including expensive human annotations in the proposed modeling framework for structuring the discrete latent subspace.

Original: This approach has low additional costs because the method operates semi-supervised, so only a small subset of multi-agent segments needs to be annotated.
Revised: This approach has low additional costs because the method operates semi-supervised, so only a small subset of multi-agent segments needs to be annotated.

Original: Given the higher sampling rate and less interactive nature of soccer dynamics, we also aim to evaluate the extent to which social dependency modeling modules achieve empirical beneﬁts.
Revised: Given the higher sampling rate and less interactive nature of soccer dynamics, we also aim to evaluate the extent to which social dependency modeling modules achieve empirical beneﬁts.

Original: We design our baselines accordingly.
Revised: We design our baselines accordingly.

Original: Generation For trajectory prediction, we compare MAT to VRNN [9], an interactive version of thereof with GNNs on the hidden states ( VRNN+GNN ), GVRNN [62], S-MAT with long-term goals (introduced in Section 4.2), and a diagonal MAT version, MAT-Diag .
Revised: Generation For trajectory prediction, we compare MAT to VRNN [8], an interactive version of thereof with GNNs on the hidden states ( VRNN+GNN ), GVRNN [55], S-MAT with long-term goals (introduced in Section 4.2), and a diagonal MAT version, MAT-Diag .

Original: Note that the VRNN constitutes a strong competitor for the task at-hand, as it effectively corresponds to a diagonal GVRNN, which has been shown to produce competitive results on soccer data [62].
Revised: Note that the VRNN constitutes a strong competitor for the task at-hand, as it effectively corresponds to a diagonal GVRNN, which has been shown to produce competitive results on soccer data [55].

Original: The left part of Table 3 shows the results.
Revised: The left part of Table 3 shows the results.

Original: Although the results are generally more similar compared to the basketball experiments, the semi-supervised MAT generates trajectories that are closest in both metrics to the observed ones.
Revised: Although the results are generally more similar compared to the basketball experiments, the semi-supervised MAT generates trajectories that are closest in both metrics to the observed ones.

Original: We hypothesize that decreasing the frequency of the data and extracting multi-agent segments that merely consist of highly interactive plays yields an signiﬁcant increase in performance gap.
Revised: We hypothesize that decreasing the frequency of the data and extracting multi-agent segments that merely consist of highly interactive plays yields an signiﬁcant increase in performance gap.

Original: Classiﬁcation MAT learns a model over the label space Y simultaneously to the generative model.
Revised: Classiﬁcation MAT learns a model over the label space Y simultaneously to the generative model.

Original: To evaluate classiﬁcation performance, we turn the output of the corresponding softmax for a given frame into a class label whenever it exceeded a predeﬁned threshold.
Revised: To evaluate classiﬁcation performance, we turn the output of the corresponding softmax for a given frame into a class label whenever it exceeded a predeﬁned threshold.

Original: We compare the prediction performance to the classiﬁcation baselines introduced in Section 4.4.
Revised: We compare the prediction performance to the classiﬁcation baselines introduced in Section 4.4.

Original: The resulting accuracies and F1 scores are summarized in Table 3.
Revised: The resulting accuracies and F1 scores are summarized in Table 3.

Original: Once more, MAT clearly beats the baselines and stands out by the highest accuracy and F1 scores.
Revised: Once more, MAT clearly beats the baselines and stands out by the highest accuracy and F1 scores.

Original: The result impressively demonstrates the beneﬁt of including unlabeled data for the task.
Revised: The result impressively demonstrates the beneﬁt of including unlabeled data for the task.

Original: Surprisingly, MA-RNN performs better than
Revised: Surprisingly, MA-RNN performs better than

Original: S-MAT .
Revised: S-MAT .

Original: This ﬁnding is in contrast to the results on NBA shown in Figure 4.
Revised: This ﬁnding is in contrast to the results on NBA shown in Figure 4.

Original: However, soccer is played on a much larger pitch and movements are likely more linear compared to basketball, which leads to less multimodality of the distributions involved and a consequently decreased beneﬁt of variational methods.
Revised: However, soccer is played on a much larger pitch and movements are likely more linear compared to basketball, which leads to less multimodality of the distributions involved and a consequently decreased beneﬁt of variational methods.

Original: Together with a signiﬁcantly reduced label space and sufﬁcient amounts of data this may lead to a simpler learning task where MA-RNN is less prone to overﬁtting.
Revised: Together with a signiﬁcantly reduced label space and sufﬁcient amounts of data this may lead to a simpler learning task where MA-RNN is less prone to overﬁtting.

Original: The left part of Fig.
Revised: The left part of Fig.

Original: 3 shows exemplary prediction probabilities for two segments and three possible events.
Revised: 3 shows exemplary prediction probabilities for two segments and three possible events.

Original: In both segments, the algorithm is highly conﬁdent no shot-action will occur in the near future.
Revised: In both segments, the algorithm is highly conﬁdent no shot-action will occur in the near future.

Original: The segment on top contains an event of the class other ball action followed by an event of class pass .
Revised: The segment on top contains an event of the class other ball action followed by an event of class pass .

Original: Both are correctly identiﬁed by MAT .
Revised: Both are correctly identiﬁed by MAT .

Original: The former is clearly indicated by a peak in the corresponding probability chart, which then decreases to give rise to the following pass action.
Revised: The former is clearly indicated by a peak in the corresponding probability chart, which then decreases to give rise to the following pass action.

Original: The segment on the bottom shows solely a single pass event that is also clearly identiﬁable by a peak at the correct frame.
Revised: The segment on the bottom shows solely a single pass event that is also clearly identiﬁable by a peak at the correct frame.

Original:  Conclusion
Revised:  Conclusion

Original: We presented semi-supervised variational autoencoders for spatiotemporal multi-agent scenarios.
Revised: We presented semi-supervised variational autoencoders for spatiotemporal multi-agent scenarios.

Original: The proposed approach effectively combined ideas from semi-supervised variational autoencoders, variational recurrent neural networks, and graph neural networks.
Revised: The proposed approach effectively combined ideas from semi-supervised variational autoencoders, variational recurrent neural networks, and graph neural networks.

Original: Empirically, our approach clearly outperformed previous state-of-the-art models in interactive sequential generation and (semi-supervised) classiﬁcation tasks in all experiments.
Revised: Empirically, our approach clearly outperformed previous state-of-the-art models in interactive sequential generation and (semi-supervised) classiﬁcation tasks in all experiments.

Original: The performance underlines the beneﬁt of including unlabeled data in spatiotemporal problems where labeled sequences are either scarce or assembled from weak makeshift signals.
Revised: The performance underlines the beneﬁt of including unlabeled data in spatiotemporal problems where labeled sequences are either scarce or assembled from weak makeshift signals.

Original: Potential societal impact We have a clear focus on team sports.
Revised: Potential societal impact We have a clear focus on team sports.

Original: Though dual use of the proposed technique is certainly possible (e.g., military domains), it may be a bit far fetched as we have no expertise in these domains and would only be able to deliver uneducated guesses about whether this is realistic or not.
Revised: Though dual use of the proposed technique is certainly possible (e.g., military domains), it may be a bit far fetched as we have no expertise in these domains and would only be able to deliver uneducated guesses about whether this is realistic or not.

Original: Generally, the data and tasks at-hand are privacy-sensitive as the targets are human individuals.
Revised: Generally, the data and tasks at-hand are privacy-sensitive as the targets are human individuals.

Original: While the soccer and basketball players are probably identiﬁable in the data, they/their contract/their clubs agreed in recording the data.
Revised: While the soccer and basketball players are probably identiﬁable in the data, they/their contract/their clubs agreed in recording the data.

Original:  Acknowledgments
Revised:  Acknowledgments

Original: We would like to thank the German Football Association (DFB) for providing data for this study.
Revised: We would like to thank the German Football Association (DFB) for providing data for this study.

Original: Checklist
Revised: Checklist

Original: For all authors..
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Since this work constitutes the ﬁrst semi-supervised approach for spatiotemporal multi-agent problems, it is difﬁcult to discuss techical limitations at this stage.
Revised: [Yes] Since this work constitutes the ﬁrst semi-supervised approach for spatiotemporal multi-agent problems, it is difﬁcult to discuss techical limitations at this stage.

Original: Application-wise, there are clear limitations e.g., by using weak makeshift labels.
Revised: Application-wise, there are clear limitations e.g., by using weak makeshift labels.

Original: We discuss these issues across the paper (e.g., Section 4.)
Revised: We discuss these issues across the paper (e.g., Section 4.)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We have a clear focus on team sports.
Revised: [No] We have a clear focus on team sports.

Original: Though dual use of the proposed technique is certainly possible (e g., military domains), it may be a bit far fetched as we have no expertise in these domains and would only be able to deliver uneducated guesses about whether this is realistic or not.
Revised: Though dual use of the proposed technique is certainly possible (e.g., military domains), it may be a bit far fetched as we have no expertise in these domains and would only be able to deliver uneducated guesses about whether this is realistic or not.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 4.1. – 4.3.
Revised: [Yes] See Section 4.1. – 4.3.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] All proofs are contained in the Appendix.
Revised: [Yes] All proofs are contained in the Appendix.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The NBA data set is publicly available and we provide a URL in the paper.
Revised: [Yes] The NBA data set is publicly available and we provide a URL in the paper.

Original: The soccer data is private and must not be shared with third parties.
Revised: The soccer data is private and must not be shared with third parties.

Original: We will release the code and include a URL in the paper as well after the paper has been accepted.
Revised: We will release the code and include a URL in the paper as well after the paper has been accepted.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 4, more details are provided in the Appendixl.
Revised: [Yes] See Section 4, more details are provided in the Appendixl.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix4.
Revised: [Yes] See Appendix

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] See supplemental material (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] See supplemental material (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] The NBA data is publicly available and free, for soccer we have the permission of the data provider.
Revised: [Yes] The NBA data is publicly available and free, for soccer we have the permission of the data provider.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [Yes] The soccer and basketball players are probably identiﬁable in the data, but they/their contract/their clubs agreed in recording the data.
Revised: [Yes] The soccer and basketball players are probably identiﬁable in the data, but they/their contract/their clubs agreed in recording the data.

Original: The soccer data is private and cannot be shared.
Revised: The soccer data is private and cannot be shared.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] A Proofs of the Theorems
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: We begin with Theorem 2 for simplicity and sketch the proof of Theorem 1 afterwards.

Original: Theorem 2.

Original: Let H ( β ) be the entropy of quantity β .

Original: A lower bound on log p θ ( x ≤ T ) in Eqn (1) is given by

Original: [Equation]

Original: Proof.

Original: For unlabeled training instances, deﬁning the marginal likelihood according to the generative structure introduced in Section 3.3 yields

Original: [Equation]

Original: To derive the lower bound on the log likelihood

Original: [Equation]

Original: we incorporate the variational information into the above deﬁnition and apply Jensen’s inequality:

Original: [Equation]

Original: t | x ≤ t , z <t , y <t ) (cid:107) p θ ( y t | x <t , z <t , y <t )]

Original: [Equation]

Original: Thus, we can write

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: , which is identical to Eqn 3.

Original: Theorem 1.

Original: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by log p θ ( x ≤ T , y ≤ T ) ≥ (cid:88) t

Original: [Equation]

Original: |

Original: [Equation]

Original: z

Original: [Equation]

Original: t

Original: [Equation]

Original: y <t )] ≡

Original: [Equation]

Original: y

Original: [Equation]

Original: Proof.

Original: [Equation]

Original: B Implementation Details

Original: For our experiments, we deﬁne the model components as follows.

Original: We individually capture the spatiotemporal dependencies of the label-speciﬁc patterns y ≤ T and the data-speciﬁc patterns x ≤ T , z ≤ T , resulting in two separate variables o ( a ) , l t and o ( a ) , d t .

Original: For ease of exposition, we refer to the concatenation of the two parts as o ( a ) t in the remainder of this paper.

Original: The encoders are modeled as q φ (˜ y ( a ) | x ≤ t , z <t , y <t ) =

Original: Cat ( y ( a ) t | f yenc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t )]) and q φ ( z ( a ) t | x ≤ t , z <t , y ≤ t ) =

Original: N ( z ( a ) t | f

Original: z enc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t ) , ϕ y ( y ( a ) t )]) for uncovering discrete and continuous latent information, respectively.

Original: The prior distributions are computed analogously omitti

Original: All functions f and feature extractors ϕ are two-layer MLPs with LeakyRelu

Original: [59] activations.

Original: We use 2 -layer GRU networks [8] for recurrence.

Original: Moreover, we deﬁne the graph network using 3 GAT layers implemented according to [58] and the graph structure wrt.the k

Original: = 2 and k = 5 spatially closest agents for basketball and soccer, respectively.

Original: We implement skip-connections as described in [60] and use concatenation followed by a linear transformation to aggregate intermediate layer embeddings to the GNN model output.

Original: For generative tasks, we use λ 1 = 0 , however, including the auxiliary loss yields negligible deterioration.

Original: For classiﬁcation, we use λ 0 = 1 and α = 10 ∗ (1 / % of labeled data ) .

Original: The model operates solely on agent velocities (input and output).

Original: However, we inject position tuples to the all model components via teacher forcing.

Original: We use the gumbel-softmax trick [37, 25] when sampling from categorical distributions.

Original: Training was executed on an Nvidia V100 GPU and took about 16 hours to complete 300 epochs while comsuming ∼ 15 Gb for basketball.

Original: All models are optimized using Adam [27] with a learning rate of 0 , 001 and gradient clipping using a max norm of 10 .

Original: All models are implemented using PyTorch [44].

Original: Baselines As stated in Section 4.1, we report against the values reported in [42].

Original: For [18, 35, 39], we adapt the source code from their ofﬁcial repositories to our experimental setting 6 For GVRNN [62], we re-implemented the model according to the descriptions in their paper and designed the overall architecture such that it is comparable in parameter number to our method.

Original: Soccer application Since the labels denote ball-centric events, we use the output of the ball node for loss computation and evaluation.

Original: The F 1 score is computed as follows.

Original: We annotate a multiagent segment when the derived probability estimates exceed an externally deﬁned threshold value.

Original: We obtain TP values (FP values) when the predicted event coincides (disagrees) with the ground truth annotation.

Original: FN values are deﬁned by anotated segments that remain undetected.

Original: We compute

Original: F1-scores for 100 distinct threshold values in the range between 0 . 5 and 0 .

Original: 98 and only report the maximum F1-score.

Original: However, threshold optimization yields only negligible improvement over simply using 0 .

Original: C More Details on the MAT Models

Original: In Section 3.1, we deﬁne y ≤ T as arbitrary discrete (possibly latent) behavioral indicators.

Original: To validate this general formulation (and our proposed architecture), we vary the speciﬁc deﬁnition of the variable across experiments.

Original: Accordingly, we propose different framework instantiations that are described in more detail in this section.

Original: S-MAT First, we observe in Section 4.2 that existing SOTA trajectory prediction approaches (Section 4.1) use heuristically generated labels for trajectory prediction that encode agents’ intents or goals over a discretized position space.

Original: We then note that our formalization allows to naturally integrate such long-term goals into the overall scheme via treating them as discrete semantic concepts.

Original: Since these labels are generated heuristically based on the trajectory input prior to model training, this model instantiation is fully-supervised and is referred to as S-MAT .

Original: We refer to [67] for more details on how to produce the weak labels used for training S-MAT.

Original: U-MAT

Original: However, we found (although done in [ ? 42]) that benchmarking against unsupervised baselines is inappropriate for the reasons described in Section 4.1.

Original: Since most SOTA approaches are unsupervised generative models, we additionally propose a fully-unsupervised instantiation of our framework ( U-MAT ) employing no external guidance for discrete latent structuring.

Original: From the predicted (latent) categorical distribution q φ , we sample a label value and exploit separate motion predictors (different decoder parameterizations) p θ dependent on the realized value.

Original: Intuitively, this encourages the model to learn categories describing fundamentally different movement patterns, which can be interpreted as dynamic “agent roles”.

Original: Thus, the concept of agent roles here is merely an intuitive explanation and realized via an inductive bias that increases the “scope” of latent information encoded without utilizing any supervision.

Original: To the best of our knowledge, parameterizing the generation module based on inferred agent categories is novel and could provide valuable insights for practitioners.

Original: D Experiments with Drone Data

Original: [Table]

Original: To showcase applicability to scenarios that exhibt variable numbers of agents, we also report results on the Stanford Drone Data (SDD) [46].

Original: SDD is a collection of videos recorded by drones at eight locations at Stanford.

Original: While pedestrians predominate as interacting agents, cyclists, skateboarders, cars, buses, and golf carts are also present.

Original: We use the TrajNet benchmark [49] of the data, providing sequential two-dimensional real-world coordinates at a frame rate of 2.5 frames per second.

Original: We follow [42] regarding the data processing strategy 7 .

Original: We incorporate PECNet [39] for comparison as it is considered state-of-the-art for SDD [38].

Original: PECNet models stochasticity in the ﬁnal position of the pedestrians conditioned on the past motion history.

Original: To generate weak labels, we move a time window through each trajectory, with the respective end cells acting as agent targets y ( a ) t .

Original: Static time windows allow us to use the fully-supervised variant for all comparisons (including fully unsupervised models).

Original: The scene at hand is thereby discretized into 960 areas and results are shown in Table 4.

Original: We observe the same general pattern as for the basketball experiments.

Original: [Table]
Revised: [N/A]

Original: Table 5 validates the proposed architecture by showing results of an ablation study.

Original: We test a fully connected graph ( S-MAT-Full ), an independent version with diagonal adjacency matrices ( S-MATDiag ), a variant that employs GNNs for variational, generative, and prior distribution parameters, but ignores interactive updates for the hidden states ( S-MAT-GRVNN ) similar to [62], and a variant that is identical to the latter but additionally includes a GNN for the hidden state ( S-MAT-GRVNN-Hidden ).

Original: The table provides supporting evidence for our design choices: though the GVRNNs are theoretically able to capture intra-timestep dependencies, we observe signiﬁcant drops in performance compared to the other competitors.

Original: The S-MAT-Diag and MAT-Full experiments suggest that our model captures interaction patterns among agents very well; S-MAT denotes a valuable contribution to the large body of research that explicitly addresses modeling multi-agent data accurately.