Original: Latent Variable Models for Multi-Agent TrajectoriesAnonymous Author(s)
Revised: Semi-Supervised Generative Models for Multi-Agent Trajectories

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Analyzing the spatiotemporal behavior of multiple agents is of great interest tomany communities.
Revised: Analyzing the spatiotemporal behavior of multiple agents is of great interest to many communities.

Original: Existing probabilistic models in this regime employ eitherunsupervised generative settings, in which the latent space is described fully bydiscrete or continuous representations, or, are alternatively formalized in a (fully)supervised framework where weakly preserved labels add explicit information to acontinuous latent representation learned from the data.
Revised: Existing probabilistic models in this realm are formalized either in an unsupervised framework, where the latent space is described by discrete or continuous variables, or in a supervised framework, where weakly preserved labels add explicit information to continuous latent representations.

Original: To overcome the resultinglimitations, we propose a novel objective function for processing multi-agent trajectories based on semi-supervised variational autoencoders, where equivariance andinteraction of agents are modeled via costumized graph networks.
Revised: To overcome inherent limitations, we propose a novel objective function for processing multi-agent trajectories based on semi-supervised variational autoencoders, where equivariance and interaction of agents are captured via customized graph networks.

Original: Our formulationdisentangles discrete and continuous  effects andallows discrete behavioral indicators in arbitrary quantity and annotation type to guide the generation process. Thislifts applicability to relevant prediction problems beyond the generation of collective movements and provides an effective solution to incoporate expensive domainknowledge into interactive multi-agent systems.
Revised: The resulting architecture disentangles discrete and continuous latent effects and provides an effective solution for injecting expensive domain knowledge into interactive sequential systems.

Original: Empirically, our model  outperforms various state-of-the-art baselines in generating future agent movements oninteractive real-world datasets. We alsoshow that our approach effectively learns toleverage unsupervised multi-agent sequences to improve classiﬁcation of long-termlocations as well as manually annotated situations on  sports tracking data.
Revised: Empirically, our model not only outperforms various state-of-the-art baselines in trajectory forecasting, but also learns to effectively leverage unsupervised multi-agent sequences for classiﬁcation tasks on interactive real-world sports datasets.

Original:  Introduction
Revised:  Introduction

Original: Analyzing the spatiotemporal behavior of multiple agents bears a great deal of value in many domainssuch as autonomous driving [49, 13], public transportation [30, 58], migration [39, 22] and sportsanalytics [53].
Revised: Analyzing the spatiotemporal behavior of multiple agents bears a great deal of value in many domains such as autonomous driving [51, 16], public transportation [32, 60], migration [41, 24] or sports analytics [55, 12].

Original: Particularly, the detection of collective patterns across time and space is of interest tomany communities but non-trivial inter-dependencies between agents render estimating the inherentmulti-modal distributions often difﬁcult.
Revised: Particularly, the detection of collective patterns across time and space is of interest to many communities but non-trivial inter-dependencies between agents render estimating the inherent multi-modal distributions often difﬁcult.

Original: Existing generative approaches to modeling sequential multi-agent data thus capitalize on variantsfrom graph networks [55, 12] and (sequential) variational autoencoders [26, 7]; being purely unsupervised, their functionality is grounded in uncovering the implicit modular structure in agent-wise latentvariables directly from the observed trajectories via approximate inference [28, 52, 59, 16, 14, 33].
Revised: Existing generative approaches to modeling sequential multi-agent data thus capitalize on variants from graph networks [57, 15] and (sequential) variational autoencoders [28, 9]; being purely unsupervised, their functionality is grounded in uncovering the implicit modular structure in agent-wise latent variables directly from the observed trajectories via approximate inference [30, 54, 61, 18, 17, 35].

Original: Although a generative view on the problem is very appealing, approaches in this regime either lackinterpretable and controllable latent factors (e.g., [59]), and are thus limited in practical applicability,or put a strong focus on relationship discovery (e.g., [28]), which limits their generative capacity.
Revised: Although a generative view on the problem is very appealing, approaches in this regime either lack interpretable and controllable latent factors (e.g., [61]), and are thus limited in practical applicability, or put a strong focus on relationship discovery (e.g., [30]), which limits their generative capacity.

Original: An alternative is offered by supervised approaches that aim to counteract these issues by incorporatingdomain-knowledge in form of discrete labels into the generation process [64].
Revised: An alternative is offered by supervised approaches that aim to counteract these issues by incorporating discrete behavioral indicators into the generation process [66].

Original: Ideally, these labelsencode implicit expert knowledge that would be learned by the model and substantially expand thescope of information captured in latent space.
Revised: Ideally, these labels encode implicit expert knowledge that would be learned by the model and substantially expand the scope of information captured in latent space.

Original: Instead of arguing in favor of expensive and tediousmanual annotations of spatiotemporal data, however, recent approaches propose to incorporateheuristic surrogates as makeshift labels into the problem [65, 64, 40].
Revised: Instead of arguing in favor of expensive and tedious manual annotations of spatiotemporal data, however, these approaches propose to incorporate heuristic surrogates as makeshift labels into the problem [67, 66, 42].

Original: While reporting impressivepredictive accuracies, they require the existence of label sequences for all training instances and cannot include inexpensive and possibly abundant unlabeled data due to their purely supervisednature.
Revised: While reporting impressive predictive accuracies, they require the existence of label sequences for all training instances and cannot include inexpensive and possibly abundant unlabeled data due to their purely supervised nature.

Original: To close this gap, we propose semi-supervised variational autoencoders for spatiotemporal multi-agentproblems to process (weakly) labeled as well as unlabeled data.
Revised: To close this gap, we propose semi-supervised variational autoencoders for spatiotemporal multiagent problems to process (weakly) labeled as well as unlabeled data.

Original: Our contribution generalizes theclass of semi-supervised variational autoencoders [27] to spatiotemporal domains by using ideas fromvariational recurrent neural networks [7].
Revised: Our contribution generalizes the class of semi-supervised variational autoencoders [29] to spatiotemporal domains by using ideas from variational recurrent neural networks [9].

Original: At a high level, the derived objective function subsumesprevious formalizations into a uniﬁed framework via a disentangled latent space that can incorporatesupervision signals in arbitrary quantities for structuring the discrete latent subspace.
Revised: At a high level, the derived objective function subsumes previous approaches into a uniﬁed framework via a disentangled latent space that can incorporate supervision signals in arbitrary quantities for structuring the discrete latent subspace.

Original: Additionaladaptive graph network layers provide order in- and equivariance of agents and render the proposedapproach appropriate for multi-agent scenarios.
Revised: Additional adaptive graph network layers provide order in- and equivariance of agents and render the proposed approach appropriate for multi-agent scenarios.

Original: The resulting semi-supervised approach is applicableto a wide range of problems including the generation of collective movements as well as classifyingsituations of interest.
Revised: The resulting semi-supervised approach is applicable to a wide range of problems including the generation of collective movements as well as classifying situations of interest.

Original: Empirically, our contribution signiﬁcantly advances the state-of-the art ingeneration and classiﬁcation tasks on interactive real-world data.
Revised: Empirically, our contribution signiﬁcantly advances recent baselines in generation and classiﬁcation tasks on interactive real-world data.

Original:  Related Work
Revised:  Related Work

Original: Semi-supervised generative models Semi-supervised variational autoencoders have been originally studied by [27] who propose to incorporate additional labels in latent space.
Revised: Semi-supervised generative models Semi-supervised variational autoencoders have been originally studied by [29] who propose to incorporate additional labels in latent space.

Original: Related approachesintroduce auxiliary variables that leave the original model unchanged but increase the ﬂexibility ofthe variational posterior [34].
Revised: Related approaches introduce auxiliary variables that leave the original model unchanged but increase the ﬂexibility of the variational posterior [36].

Original: [24] propose to explicitly capture label characteristics in latent spaceinstead of the label values themselves and [41] study generalizations that allow for learning morecomplex dependency structures.
Revised: [26] propose to explicitly capture label characteristics in latent space instead of the label values themselves and [43] study generalizations that allow for learning more complex dependency structures.

Original: However, all the above approaches are introduced only for staticdomains.
Revised: However, all the above approaches are introduced only for static domains.

Original: Our contribution constitutes a generalization of the above approaches to spatiotemporaldomains.
Revised: Our contribution constitutes a generalization of the above approaches to spatiotemporal domains.

Original: Sequential generative models Approaches designed for predicting agent trajectories are frequentlybased on ideas from sequential generative models to encode temporal dependencies and addressthe stochasticity inherent in future predictions.
Revised: Sequential generative models Approaches designed for predicting agent trajectories are frequently based on ideas from sequential generative models to encode temporal dependencies and address the stochasticity inherent in future predictions.

Original: Sequential extensions of variational autoencoderswith dynamic latent variables include [3, 7, 11, 15].
Revised: Sequential extensions of variational autoencoders with dynamic latent variables include [4, 9, 14, 1].

Original: Particularly, the VRNN [7] is related to ourcontribution as both deploy a VAE instance per time step conditioned on a recurrent state.
Revised: Particularly, the VRNN [9] is related to our contribution, as both deploy a VAE instance per time step conditioned on a recurrent state.

Original: However,their work lacks modeling of a social dimension as well as means to integrate (potentially expensive)discrete behavioral indicators.
Revised: However, their work lacks modeling of a social dimension as well as means to integrate (potentially expensive) discrete behavioral indicators.

Original: Other approaches associate a single global latent variable with eachsequence [4, 17, 9, 8].
Revised: Other approaches associate a single global latent variable with each sequence [5, 19, 11, 10].

Original: Graph neural networks Deep graph-based (GNNs) approaches [50] are a natural methodologicalchoice for applications requiring modeling datasets composed of sets.
Revised: Graph neural networks Deep graph-based (GNNs) approaches [52] are a natural methodological choice for applications requiring modeling datasets composed of sets.

Original: GNNs operate by learning achain of hidden representations for each node through an iterative process that relies on aggregatingmessages along edges in each network layer.
Revised: GNNs operate by learning a chain of hidden representations for each node through an iterative process that relies on aggregating messages along edges in each network layer.

Original: Prominent instantiations such as GCN [29], GraphSAGE[18], GAT [55], and others [12, 54, 20, 2, 63, 57] mainly differ in their notion of message passing andneighborhood aggregation [60].
Revised: Prominent instantiations such as GCN [31], GraphSAGE [20], GAT [57], and others [15, 56, 22, 3, 65, 59] mainly differ in their notion of message passing and neighborhood aggregation [62].

Revised: GNN layers can be stacked and allow for context-based and adaptive neighborhoods for each node by incorporating particularly designed skip connections [59].

Original: See [5] for a discussion on the function approximation capabilitiesof this neural network family.
Revised: See [7] for a discussion on the function approximation capabilities of this neural network family.

Original: Generative models for sequential multi-agent data Given the potential beneﬁt across variousdomains, a great deal of recent publications focus on movements of pedestrians or self-driving cars[37, 49, 13, 38, 61, 48, 1, 21, 62, 32].
Revised: Generative models for sequential multi-agent data Given the potential beneﬁt across various domains, a great deal of recent publications focus on movements of pedestrians or self-driving cars [39, 51, 16, 40, 63, 50, 2, 23, 64, 34].

Original: [36], however, show that these standard benchmark datasetsgenerally exhibit weak social interaction patterns, so in the remainder, we focus on methods usingteam sport data.
Revised: [38], however, show that these standard benchmark datasets generally exhibit weak social interaction patterns, so in the remainder, we focus on methods using team sport data.

Original: Related approaches show that learning and computing realistic rollouts signiﬁcantlybeneﬁt from the incorporation of makeshift annotations and heuristic surrogates into the problem[65, 64, 40].
Revised: Related approaches show that learning and computing realistic rollouts signiﬁcantly beneﬁt from the incorporation of makeshift annotations and heuristic surrogates into the problem [67, 66, 42].

Original: [10, 33] propose an architecture based on conditional VAEs [51] to model distributionsof future movements of basketball players.
Revised: [13, 35] propose an architecture based on conditional VAEs [53] to model distributions of future movements of basketball players.

Original: [59, 52] combine ideas from graph networks [12] and
Revised: [61, 54] combine ideas from graph networks [15] and

Original: VRNNs [7] into a uniﬁed framework aiming at modeling data from basketball and soccer.
Revised: VRNNs [9] into a uniﬁed framework aiming at modeling data from basketball and soccer.

Original: Otherrelated approaches [31, 28, 16, 14] aim to explicitly infer discrete latent variables to representinteraction types while executing a trajectory prediction task.
Revised: Other related approaches [33, 30, 18, 17] aim to explicitly infer discrete latent variables to represent interaction types while executing a trajectory prediction task.

Original: Hence, existing generative models for spatiotemporal pattern detection focus on multi-agent trajectory prediction.
Revised: Hence, existing generative models for spatiotemporal pattern detection focus on multi-agent trajectory prediction in fully-(un)supervised settings.

Original: Our contribution constitutes the ﬁrst semi-supervised generalization of variationalautoencoders for spatiotemporal domains that gives rise to addressing tasks beyond representationalor generative modeling.
Revised: Our contribution represents the ﬁrst semi-supervised generalization of variational autoencoders for spatiotemporal domains, which allows addressing tasks beyond representational or generative modeling involving expensive annotation processes.

Original:  Methods 
Revised:  Methods 3.1 Preliminaries

Original: 3.1 Preliminaries

Original: Variational autoencoders (VAEs) [26, 43] allow to capture intrinsic conditional dependencies andare often used in structured problems.
Revised: Variational autoencoders (VAEs) [28, 45] allow to capture intrinsic conditional dependencies and are often used in structured problems.

Original: Essentially, they consist of a θ -parameterized generativemodel p θ ( x , z ) =
Revised: Essentially, they consist of a θ -parameterized generative model p θ ( x , z ) =

Original: p ( z ) p θ ( x | z ) and an associated φ -parameterized variational distribution q φ ( z | x ) .
Revised: p ( z ) p θ ( x | z ) and an associated φ -parameterized variational distribution q φ ( z | x ) .

Original: Optimal parameters { θ, φ } are obtained via maximizing the variational lower bound on the marginallikelihood, given by
Revised: Optimal parameters { θ, φ } are obtained via maximizing the variational lower bound on the marginal likelihood, given by

Original: [Equation]
Revised: [Equation]

Original: It is often helpful to augment  discrete labels y into the model to guide the generating process [27].
Revised: It is often helpful to augment partially observed discrete labels y into the model to guide the generating process [29].

Original: The resulting model p ( x , z , y ) =
Revised: The resulting model p ( x , z , y ) =

Original: p ( x | z , y ) p ( y ) p ( z ) then acts on labeled and unlabeled instances.
Revised: p ( x | z , y ) p ( y ) p ( z ) then acts on labeled and unlabeled instances.

Original: Consequentially, the variational distribution q φ ( y, z | x ) =
Revised: Consequentially, the variational distribution q φ ( y, z | x ) =

Original: q φ ( y | x ) q φ ( z | y, x ) needs to be deﬁnedover both quantities ( z and y ).
Revised: q φ ( y | x ) q φ ( z | y, x ) needs to be deﬁned over both quantities ( z and y ).

Original: Semi-supervised VAEs (SSVAE) are trained by maximizing
Revised: Semi-supervised VAEs (SSVAE) are trained by maximizing

Original: [Equation]
Revised: [Equation]

Original: where L ( x , y )
Revised: where L ( x , y )

Original: =
Revised: =

Original: E q φ ( z | x ,y )
Revised: E q φ ( z | x ,y )

Original: [log p θ ( x | y, z ) + log p ( y ) + log p ( z ) − log q φ ( z | x , y )] and U ( x ) =102 (cid:80)
Revised: [log p θ ( x | y, z ) + log p ( y ) + log p ( z ) − log q φ ( z | x , y )] and U ( x ) = (cid:80)

Original: y q φ ( y | x )( L ( x , y ))
Revised: y q φ ( y | x )( L ( x , y ))

Original: + H ( q φ ( y | x )) denote (variational) lower bounds on labeled and unlabeledinstances, respectively, and λ is a balancing term.
Revised: + H ( q φ ( y | x )) denote (variational) lower bounds on labeled and unlabeled instances, respectively, and λ is a balancing term.

Original: In fact, the original motivation for this frameworkwas inspired by semi-supervised classiﬁcation tasks.
Revised: The original motivation for this framework was inspired by semi-supervised classiﬁcation tasks.

Original: However, their contribution is also regularly [Figure] used for learning meaningful representations and generating new data, thus rendering it a suitablemethodological foundation for performing tasks with all sorts of data.
Revised: However, SSVAEs are also frequently employed for learning meaningful representations and generating new data, thus rendering it a suitable methodological foundation for performing diverse sets of tasks.

Original: The dependency structures ofq φ and p θfor SSVAEs are displayed visually as part of Figure 1 (right side).
Revised: The dependency structures of q φ and p θ are displayed visually as part of Figure 1.

Original: 3.2 Problem Formulation
Revised: 3.2 Problem Formulation

Original: We are interested in modeling spatiotemporal multi-agent scenarios and focus on positions x ( a ) t ∈ R 2of an agent a ∈
Revised: We are interested in modeling spatiotemporal multi-agent scenarios and focus on positions x ( a ) t ∈ Rof an agent a ∈

Original: A over time 1 ≤ t ≤ T , denoted by x ( a ) ≤ T .
Revised: A over time 1 ≤ t ≤ T , denoted by x ( a ) ≤ T .

Original: We assume that a non-empty subset ofagents in A is present in any observed segment and collect their trajectories (in random order) to formcollective movements x ≤ T :=
Revised: We assume that a non-empty subset of agents in A is present in any observed segment and collect their trajectories (in random order) to form collective movements x ≤ T :=

Original: { x ( a ) ≤ T | a ∈ A} .
Revised: { x ( a ) ≤ T | a ∈ A} .

Original: We will make use of velocities d x /dt and linearizedmotion ∆ x t = x t (cid:48) − x t , for t (cid:48) >
Revised: We will make use of velocities d x /dt and linearized motion ∆ x t = x t (cid:48) − x t , for t (cid:48) >

Original: t , of agents in the remainder.
Revised: t , of agents in the remainder.

Original: In addition, we introduce discrete labels y
Revised: In addition, we introduce discrete labels y

Original: ( a ) t ∈ Y associated with agent
Revised: ( a ) t ∈ Y associated with agent

Original: a ∈ A at time t .
Revised: a ∈ A at time t .

Original: Thecollection of sequences y ≤ T :=
Revised: The collection of sequences y ≤ T :=

Original: { y ( a ) ≤ T | a ∈ A} may be best thought of as discriminative causesof variation in the corresponding trajectories x ≤ T .
Revised: { y ( a ) ≤ T | a ∈ A} may be best thought of as discriminative causes of variation in the corresponding trajectories x ≤ T .

Original: In practice, y ≤ T may contain anything thataids in generating multi-agent rollouts, including behavioral indicators, long-term goals, or currenttasks.
Revised: In practice, y ≤ T may contain anything that aids in generating multi-agent rollouts, including behavioral indicators, long-term goals, or current tasks.

Original: Finally, observations x ≤ T accompanied by labels y ≤ T assemble the labeled share of thedata, denoted by D L = { ( x ≤ T , y ≤ T ) n } 1 ≤ n ≤ N , while unlabeled observations are collected in
Revised: Finally, observations x ≤ T accompanied by labels y ≤ T assemble the labeled share of the data, denoted by D L = { ( x ≤ T , y ≤ T ) n } 1 ≤ n ≤ N , while unlabeled observations are collected in

Original: D U
Revised: D U

Original: = { ( x ≤ T ) m } 1 ≤ m ≤ M .
Revised: = { ( x ≤ T ) m } 1 ≤ m ≤ M .

Original: We refer to all data by D = D L ∪ D U .
Revised: We refer to all data by D = D L ∪ D U .

Original: We aim at estimating the underlying distribution that generated the observations in D .
Revised: We aim at estimating the underlying distribution that generated the observations in D .

Original: A straightforward solution is to maximize the log likelihood given by
Revised: A straight forward solution is to maximize the log likelihood given by

Original: [Equation]
Revised: [Equation]

Original: However, human movement is inherently non-determenistic and multi-modal.
Revised: However, human movement is inherently non-determenistic and multi-modal.

Original: Thus, it is common tomodel the stochasticity inherent in future agent behaviors via latent representations of VAEs.
Revised: Thus, it is common to model the stochasticity inherent in future agent behaviors via latent representations of VAEs.

Original: Wepropose to lower-bound Eq. 1 as follows.
Revised: We propose to lower-bound Eq. 1 as follows.

Original: 3.3 SSVAEs for  Multi-Agent Trajectories
Revised: 3.3 SSVAEs for Modeling Multi-Agent Trajectories

Original: We associate disentangled latent realizations { z t , y t } with each time step t of the segment whileinjecting the dependency structures introduced in Section 3.1 for the generative and inference parts.
Revised: We associate disentangled latent realizations { z t , y t } with each time step t of the segment while injecting the dependency structures introduced in Section 3.1 for the generative and inference parts.

Original: To model temporal dependencies, wepropose to additionally condition the model components on thepast observations x <t and latent variables z <t and y <t (cmp.
Revised: To model temporal dependencies, we additionally condition the model components on the past observations x <t and latent variables z <t and y <t (cmp.

Original: The joint distribution factorizesinto
Revised: The joint distribution factorizes into p θ ( x ≤ T , z ≤ T , y ≤ [Equation] ) = (cid:89)

Original: [Equation]
Revised: [Equation]

Original: with generating distribution p θ ( x t | x <t , z ≤ t , y ≤ t ) and prior distributions p θ ( z t | x <t , z <t , y <t ) andp θ ( y t | x <t , z <t , y <t ) for latents z t and y t .
Revised: with generating distribution p θ ( x t | x <t , z ≤ t , y ≤ t ) and prior distributions p θ ( z t | x <t , z <t , y <t ) and p θ ( y t | x <t , z <t , y <t ) for latents z t and y t .

Original: Note thatwe differentiate between observed labels y andlatent variables ˜ y to derive the variational distributions. Also note that in our sequential setting, the priors are included in the optimization rather than beingrepresented by ﬁxed distributions to account for latent dynamics and effective sampling at inferencetime.
Revised: Note that in our sequential setting, the priors are included in the optimization rather than being represented by ﬁxed distributions to account for latent dynamics and effective sampling at inference time. To derive the variational distributions, we differentiate between observed labels y and latent variables ˜

Original: Hence, the surrogate posteriors q φ also need to condition on previous observations and randomvariables to allow for accurate approximations of the true posterior and provide a tractable lowerbound on the log-likelihood in Eqn (1).
Revised: y in the remainder.

Original: Using the factorization introduced in Section 3.1 yields
Revised: Using the factorization introduced in Section 3.1 yields

Original: [Equation]
Revised: [Equation]

Original: Labeled instances render reasoning over latents ˜ y ≤ T unnecessary since the true y ≤ T is alreadyknown.
Revised: Labeled instances render reasoning over latents ˜ y ≤ T unnecessary since the true y ≤ T is already known.

Original: Hence, the right-hand side in Eqn (2) can be ignored when sampling from D L .
Revised: Hence, the right-hand side in Eqn (2) can be ignored when sampling from D L .

Original: We arrive atthe following results and refer to Appendix A for the proofs.
Revised: We arrive at the following results and refer to Appendix A for the proofs.

Original: Theorem 1.
Revised: Theorem 1.

Original: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by
Revised: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by

Original: [Equation]
Revised: [Equation]

Original: Correspondingly, for unlabeled instances drawn from D U , sequential label information need to beestimated as shown in the following theorem.
Revised: Correspondingly, for unlabeled instances drawn from D U , sequential label information need to be estimated as shown in the following theorem.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let H ( β ) be the entropy of quantity β .
Revised: Let H ( β ) be the entropy of quantity β .

Original: A lower bound on log p θ ( x ≤ T ) in Eqn (1) isgiven by
Revised: A lower bound on log p θ ( x ≤ T ) in Eqn (1) is given by

Original: [Equation]
Revised: [Equation]

Revised: Eqn (4) encodes the behavior of object q φ ( y ≤ T | x ≤ T ) when dealing with unsupervised data.

Revised: Since data likelihood log p θ ( x t | x <t , , z ≤ t , y ≤ t ) tends to comprise the largest factor in L ( x t , y t ) , taking the expectation wrt.

Revised: q φ (˜ y t | x ≤ t , z <t , ˜ y <t ) encourages the model to realize the highest probability mass for the latent label valuel y ∈ Y that incurs the smallest reconstruction loss compared to other candidates in label space.

Revised: This is a desirable property that reﬂects our assumption of data generation.

Original: The full evidence lower bound (ELBO) is obtained by combining the lower bounds for all data D ,
Revised: The full evidence lower bound (ELBO) is obtained by combining the lower bounds for all data D ,

Original: [Equation]
Revised: [Equation]

Original: U T (cid:88) t = −U ( x t ) . This formalization focuses on either fully labeled or unlabeled observations, however, a more generalformulation can be obtained to also allow for partially labeled observations.
Revised: This formalization focuses on either fully labeled or unlabeled observations, however, a more general formulation can be obtained to also allow for partially labeled sequential data points [Equation]

Original: 3.4 Semi-supervised Sequential Learning
Revised: 3.4 Extension to Classiﬁcation Settings

Original: Intrinsic to our contribution is an encoding module that argues over the label space, q φ ( y ≤ T | x ≤ T ) ,and hence can be used as a classiﬁcation network annotating new observations.
Revised: Intrinsic to our contribution is an encoding module that argues over the label space, q φ ( y ≤ T | x ≤ T ) , and hence can be used as a classiﬁcation network annotating new observations.

Original: However, the ELBOin Eqn (5) is oblivious to classiﬁcation and simply ignores relevant gradient updates for L ( x t , y t )by discarding Eqn (2).
Revised: However, the ELBO in Eqn (5) is oblivious to classiﬁcation and simply ignores relevant gradient updates for L ( x t , y t ) by discarding Eqn (2).

Original: This is clearly inappropriate for semi-supervised learning where the overallgoal is to learn an unknown mapping x
Revised: This is clearly inappropriate for semi-supervised learning where the overall goal is to learn an unknown mapping x

Original: (cid:55)→ y .
Revised: (cid:55)→ y .

Original: Following [27], we circumvent this by heuristicallyincorporating Eqn (2) into the label-dependent objective.
Revised: Following [29], we circumvent this by heuristically incorporating Eqn (2) into the label-dependent objective.

Original: Hence, the full training criterion to learn{ θ, φ } is given by
Revised: Hence, the full training criterion to learn { θ, φ } is given by

Original: [Equation]
Revised: [Equation]

Original:  where λ 1 balances generative and discriminative learning and λ 

Revised: [Equation] [Equation] 6) where λ 1 balances generative and discriminative learning and λ

Original: the contribution of labeled andunlabeled data, respectively.
Revised: the contribution of labeled and unlabeled data, respectively.

Original: Thus, the semi-supervised variant contains fully (un)supervised modelingtasks as special cases via adjusting λ 1 , λ 2 accordingly.
Revised: Thus, J MAT contains fully (un)supervised modeling tasks as special cases via adjusting λ 1 , λ 2 accordingly.

Original: The resulting supervised target differs fromits unsupervised counterpart only in observing y t instead of factorizing over Y , which results incomputing the log-likelihood (auxiliary loss) instead of the entropy H ( q φ ( y t | x ≤ t , z <t , y <t)) . Eqn (4) encodes the behavior of classiﬁer q φ ( y ≤ T | x ≤ T ) when dealing with unsupervised data. Sincedata likelihood log p θ ( x t | x <t , , z ≤ t , y ≤ t ) tends to comprise the largest factor in L (x t , y t ) , takingthe expectation wrt. q φ (˜ y t | x ≤ t , z <t ,˜ y <t ) encourages the model to realize the highest probabilitymass for the latent label valuel y ∈ Y that incurs the smallest reconstruction loss compared to othercandidates in label space.
Revised: Note that the resulting supervised part in Eqn 6 (right part) differs from its unsupervised counterpart only in observing y t instead of factorizing over Y , which results in computing the log-likelihood (auxiliary loss) log q φ ( y t | x ≤ t , z <t , y <t ) instead of the entropy H ( q φ ( y t | x ≤ t , z <t , y <t )) .

Original: This is a desirable property that reﬂects our assumption of data generation.

Original: 3.5 Design Choices
Revised: 3.5 Design Choices

Original: As described in Section 3.2, we aim to learn a distribution over possible sequences of agent sets thatmay differ in share of discrete annotations (and cardinality).
Revised: As described in Section 3.2, we aim to learn a distribution over possible sequences of agent sets that may differ in share of discrete annotations (and cardinality).

Original: However, the previously derived objectivefunctions only account for temporal dependencies, so that independence of the agent dimensionis required to realize permutation-invariant models.
Revised: However, the previously derived objective functions only account for temporal dependencies, so that independence of the agent dimension is required to realize permutation-invariant models.

Original: This assumption is trivially inappropriate forinteractive systems where future agent movements need to be coordinated with other agents.
Revised: This assumption is trivially inappropriate for interactive systems where future agent movements need to be coordinated with other agents.

Original: Thus, itis common to propose some form of graph encoding strategy [50] to account for both, equivarianceand agent interactions, respectively.
Revised: Thus, it is common to propose some form of graph encoding strategy [52] to account for both, equivariance and agent interactions, respectively.

Original: Intuitively, optimal interaction modeling allows to adaptively accomodate structural information fromdifferent levels of granularity for each agent.
Revised: Intuitively, optimal interaction modeling allows to adaptively accommodate structural information from different levels of granularity for each agent.

Original: In team sports, for example, agent nodes should be ableto capture immediate player inﬂuences as well as holistic team strategies dependent on the interactionstructure best suited for the task at-hand.
Revised: In team sports, for example, agent nodes should be able to capture immediate player inﬂuences as well as holistic team strategies dependent on the interaction structure best suited for the task at-hand.

Original: Motivated by the previous considerations, we costumize an GNN architecture that aims to mimic the descired behavior.
Revised: Motivated by the previous considerations, we customize an GNN architecture that aims to mimic the desired behavior.

Original: Speciﬁcally, we stack attention-based GNN layers [55] with particularly designed skip-connections [57] and construct the underlying graphstructure using the k (spatially) closest agents.
Revised: Speciﬁcally, we stack attention-based GNN layers [57] with particularly designed skip-connections [59] and construct the assumed graph structure using the k (spatially) closest agents.

Original: There are various possibilities to incorporate graph-based approaches into the overall scheme 1 .
Revised: There are various possibilities to incorporate graph-based approaches into the overall scheme 1 .

Original: Anefﬁcient way to model the distributions deﬁning Eqn (6) is via hidden agent states h ( a ) t conditionedon representations o ( a ) t − 1 , 
Revised: An efﬁcient way to model the distributions deﬁning J MAT is via hidden agent states h ( a ) t conditioned on representations o ( a ) t − 1 , [Equation] with o

Original: [Equation]
Revised: [Equation]

Original: , where f rnn denotes an RNN transition function, GNN is the described GAT-based GNN and o t ={ o ( a ) t | a ∈ A} refers to the set of updated agent representations.
Revised: where f rnn denotes an RNN transition function, GNN is the described GAT-based GNN and o t = { o ( a ) t | a ∈ A} refers to the set of updated agent representations 2 .

Original: Since the elements in set o taggregate neighboring information of the RNN outputs h t via graph networks, the inferred featurevectors describe the entire interactive past of individual agents.
Revised: Since the elements in set o t aggregate neighboring information of the RNN outputs h t via graph networks, the inferred feature vectors describe the entire interactive past of individual agents.

Original: While it is reasonable to additionallyenforce intra-timestep dependencies on the latent variables using encoding and decoding GNNmodules, we argue that capturing past spatiotemporal patterns is sufﬁciently informative for highfrequency data [46].
Revised: While it is reasonable to additionally enforce intra-timestep dependencies on the latent variables using encoding and decoding GNN modules, we argue that capturing past spatiotemporal patterns is sufﬁciently informative for high-frequency data [48].

Original: Hence, assuming conditional independence within t , the joint movement distribution factorizes across the agent dimension and is given by[Equation] a ∈A p [Equation]
Revised: Hence, assuming conditional independence within t , the joint movement distribution factorizes across the agent dimension and is given by

Original: x

Original: [Equation]
Revised: [Equation]

Original: a [Equation] where ψ t = { ψ ( a ) t | a ∈ A} are the distribution parameters and ψ ( a ) t = ϕ
Revised: where ψ t = { ψ ( a ) t | a ∈ A} are the distribution parameters and ψ ( a ) t = ϕ

Original: ( z ( a ) t , y ( a ) t , o ( a ) t − 1 ) .
Revised: ( z ( a ) t , y ( a ) t , o ( a ) t − 1 ) .

Original: Thecomputational logic of the full architecture is schematically depicted in Fig.
Revised: The computational logic of the full architecture is schematically depicted in Fig.

Revised: 1 and 2.

Original:  Empirical Evaluation
Revised:  Empirical Evaluation

Original: For evaluation, we focus on team sports as the coordination of players renders these tasks moredifﬁcult than other domains [36].
Revised: For evaluation, we focus on team sports as the coordination of players renders these tasks more difﬁcult than other domains [38].

Original: Hence, we experiment on STATS SportVU NBA 3 for comparison,and tracking data fromelite soccer 4 .
Revised: Hence, we experiment on STATS SportVU NBA 4 for comparison, and tracking data from soccer games of the german national team.

Original: As detailed in Section 3.2, we use agent velocities as inputto all models and assume linear motion between consecutive observations.
Revised: As detailed in Section 3.2, we use agent velocities as input to all models and assume linear motion between consecutive observations.

Original: For NBA, we adopt theexperimental setup and processing strategy from [40] 5 .
Revised: For NBA, we adopt the experimental setup and processing strategy from [42] 5 .

Original: The STATS SportVU NBA data comprises tracking positions of offensive plays from the 2016 NBAregular season covering more than 1200 different games where game segment are given by sequencesof length 50 and contain two-dimensional positions of all agents ( 10 players and ball) sampled at 5frames per second.
Revised: The STATS SportVU NBA data comprises tracking positions of offensive plays from the 2016 NBA regular season covering more than 1200 different games where game segment are given by sequences of length 50 and contain two-dimensional positions of all agents ( 10 players and ball) sampled atframes per second.

Original: The data is split into 60% training, 20% validation, and 20% test sets.
Revised: The data is split into 60% training, 20% validation, and 20% test sets.

Original: All data istranslated so that the origin of the underlying coordinate system is mapped onto the top-left corner.
Revised: All data is translated so that the origin of the underlying coordinate system is mapped onto the top-left corner.

Original: The soccer data consists of 12 matches where positions of players and ball are sampled at 25 framesper second.
Revised: The soccer data consists of 12 matches where positions of players and ball are sampled at 25 frames per second.

Original: The tracking data is accompanied by manual event annotations that we will also make useof in the remainder.
Revised: The tracking data is accompanied by manual event annotations that we will also make use of in the remainder.

Original: Models are trained on eight matches, the remaining four are distributed evenlyinto validation and test data (two matches each).
Revised: Models are trained on eight matches, the remaining four are distributed evenly into validation and test data (two matches each).

Original: 4.1 Baselines
Revised: 4.1 Baselines

Original: We evaluate versus several baselines on two distinct tasks: trajectory forecasting (Section 4.2 and 4.4)and spatiotemporal classiﬁcation (Section 4.3 and 4.4).
Revised: We evaluate versus several baselines on two distinct tasks: trajectory forecasting (Section 4.2 and 4.4) and spatiotemporal classiﬁcation (Section 4.3 and 4.4).

Original: Since our approach constitutes the ﬁrst semisupervised model for label predictions in multi-agent scenarios, we need to rely on self-constructedsupervised baselines for scenarios where only a few labels are available, which we discuss in moredetail in the relevant sections.
Revised: Since our approach constitutes the ﬁrst semisupervised model for label predictions in multi-agent scenarios, we need to rely on self-constructed supervised baselines for scenarios where only a few labels are available, which we discuss in more detail in the relevant sections.

Original:  By contrast, trajectory forecasting is an active topic in different ﬁelds (see, e.g., [45] for an overview),which allows comparisons with a wide range of baseline models.
Revised: [Table] By contrast, trajectory forecasting is an active topic in different ﬁelds (see, e.g., [47] for an overview), which allows comparisons with a wide range of baseline models.

Revised: It is difﬁcult to determine the current state-of-the-art in generative methods for sports tracking data because there is no consistent setting in terms of forecasting horizon or data (see e.g., [35, 42]).

Revised: While some approaches are restricted to making predictions for a predetermined horizon, our method learns spatiotemporal representations autoregressively and thus can be used for the full range of prediction scenarios.

Revised: To include a variety of methods, we conduct two sets of experiments over short ( 10 timesteps) and long ( 40 timesteps) prediction horizons.

Revised: We benchmark against the most recent and methodologically important work described below.

Original: Fully supervised generativemodels such as DAG-Net [40] and Weak-Sup [64] use weak labels in the form of agent objectives(heuristically inferred prior to model training) to improve trajectory prediction.
Revised: Fully supervised generative models such as DAG-Net [42] and Weak-Sup [66] use weak labels in the form of agent objectives (heuristically inferred prior to model training) to improve trajectory prediction.

Original: In these settings, anagent’s objective at each timestep t is to reach the next area where the observed movement speedfalls below a predeﬁned threshold.
Revised: In these settings, an agent’s objective at each timestep t is to reach the next area where the observed movement speed falls below a predeﬁned threshold.

Original: Adopting the experimental framework of [40] allows us to makefurther comparisons with Social-Ways [1] and STGAT [21] without additional experiments.
Revised: Adopting the experimental framework of [42] allows us to make further comparisons with Social-Ways [2] and STGAT [23] without additional experiments.

Original: STGAT[21] augments a standard GAT with an LSTM [19] to capture both spatial and temporal dependenciesof social interactions while Social-Ways uses Info-GAN  to learn multimodal predictive distributions.
Revised: STGAT [23] augments a standard GAT with an LSTM [21] to capture both spatial and temporal dependencies of social interactions while Social-Ways uses Info-GAN [6] to learn multimodal predictive distributions.

Original: It is difﬁcult to determine the current state-of-the-art in generative baselines for sports tracking databecause there is no consistent setting interms of forecasting horizon or data (see e.g., [33, 40]). Whilesome baselines are restricted to making predictions for a predetermined horizon, our method learnsspatiotemporal representations autoregressively and thus can be used for the full range of predictionscenarios. Hence, we conduct two sets of experiments over both short ( 10 timesteps) and long ( 40timesteps) and additionally benchmark against the most recent and methodologically most relevantwork described below.
Revised: The majority of generative models for sequential multiagent systems operate in a fully-unsupervised fashion.

Original: GVRNN [59] is a graph extension of the VRNN for explicitly modeling jointagent movements for basketball and soccer data.
Revised: GVRNN [61] is a graph extension of the VRNN for explicitly modeling joint agent movements for basketball and soccer data.

Original: dNRI [16] dynamically extracts interaction typesthat determine the parametrization of the decoder.
Revised: dNRI [18] dynamically extracts interaction types that determine the parametrization of the decoder.

Original: Finally, GRIN [33] recovers (static) interactionsvia attention using a disentangled latent space.[Table] [Table]
Revised: Finally, GRIN [35] recovers (static) interactions via attention using a disentangled latent space.

Original: 4.2 Exploring Boundary Cases for Trajectory Prediction
Revised: 4.2 Exploring Boundary Cases for Trajectory Prediction

Original: Methods We study the purely (un)supervised generation of trajectories by maximizing the respective lower bounds on NBA.
Revised: Methods We study the purely (un)supervised generation of trajectories by maximizing the respective lower bounds on NBA.

Original: For the fully-supervised variant, the weak labels described in Section4.1 can be naturally integrated into the overall scheme via treating them as semantic concepts y ( a ) t .
Revised: For the fully-supervised variant, the weak labels described in Section 4.1 can be naturally integrated into the overall scheme via treating them as semantic concepts y ( a ) t .

Original: These target areas naturally vary over time so that the sequential label y ≤ Tdenotes always  the nextdesired position for every involved agent at time t .
Revised: These target areas naturally vary over time so that the sequential label y ≤ T always denotes the next desired position for every involved agent at time t .

Original: Following related work [40, 64], we obtain 90different areas/class labels.
Revised: Following related work [42, 66], we obtaindifferent areas/class labels.

Original: The underlying data distribution is estimated by maximizing Eqn [-(3) viasetting λ 
-]
Revised: The underlying data distribution is estimated by maximizing Eqn [-(3) viasetting λ 
-](3).

Original: = λ 1 = 0 in Eqn (6).

Original: The supervised variant of our contribution is denoted by S-MAT .
Revised: The supervised variant of our contribution is denoted by S-MAT .

Original: Although done in [40, 64], direct comparison of unsupervised and supervised generative modelsis inappropriate inasmuch as the latter use ground-truth labels over the entire observation period(which include future information) at prediction time.
Revised: Although done in [42, 66], direct comparison of unsupervised and supervised generative models is inappropriate inasmuch as the latter use ground-truth labels over the entire observation period (which include future information) at prediction time.

Original: Thus, we additionally benchmark against afully unsupervised instantiation of our proposed framework lacking ground-truth information forstructuring the discrete latent subspace.
Revised: Thus, we additionally benchmark against a fully unsupervised instantiation of our proposed framework lacking ground-truth information for structuring the discrete latent subspace.

Original: To encourage the model to learn semantically meaningfulconcepts describing distinct movement patterns, we parametrize the decoder dependent on thepredicted agent class ˜ y
Revised: To encourage the model to learn semantically meaningful concepts describing distinct movement patterns, we parametrize the decoder dependent on the predicted agent class ˜ y

Original: ( a ) t .
Revised: ( a ) t .

Original: In this context, ˜ y ( a ) t may be best thought of dynamically assigned agentroles that accurately explain the observed trajectories.
Revised: In this context, ˜ y ( a ) t may be best thought of dynamically assigned agent roles that accurately explain the observed trajectories.

Original: We maximize Eqn (4) using 3 agent typesand refer to the unsupervised variant as U-MAT .
Revised: We maximize Eqn (4) using 3 agent types and refer to the unsupervised variant as U-MAT .

Original: Further details for S-MAT and U-MAT are given in
Revised: Further details for S-MAT and U-MAT are given in

Original: Appendix C.
Revised: Appendix C.

Original: Metrics We measure the quality of the learned multimodal distribution using common standardmetrics: the minimum over 20 generated samples of the average and ﬁnal L 2 distance betweenpredicted ˆ x ( a ) ≤ T and observed positions x ( a ) ≤ T .
Revised: Metrics We measure the quality of the learned multimodal distribution using common standard metrics: the minimum over 20 generated samples of the average and ﬁnal L 2 distance between predicted ˆ x ( a ) ≤ [Table] [Table]

Original: Results Tables 1 and 2 show the results.
Revised: Results Tables 1 and 2 show the results.

Original: The proposed approaches clearly outperform their peersand realize the lowest average L 2 and ﬁnal L 2 distances.
Revised: The proposed approaches clearly outperform their peers and realize the lowest average L 2 and ﬁnal L 2 distances.

Original: The MAT variants effectively accommodatenot only mutual inﬂuences across agents, but also discrete generative factors, leading to betterapproximations of the underlying multi-modal distributions.
Revised: The MAT variants effectively accommodate not only mutual inﬂuences across agents, but also discrete generative factors, leading to better approximations of the underlying multi-modal distributions.

Original: Unsurprisingly, Table 1 also shows thatdefending players exhibit more structure and are easier to predict than offensive players.
Revised: Unsurprisingly, Table 1 also shows that defending players exhibit more structure and are easier to predict than offensive players.

Original: The resultsof an ablation study are contained in
Revised: The results of an ablation study are contained in

Original: Appendix E.
Revised: Appendix E.

Original: The right part of Fig.
Revised: The right part of Fig.

Original: 2 displays exemplary rolloutsthat underline models’ ability to capture highly complex changes in movement directions.
Revised: 3 displays exemplary rollouts that underline models’ ability to capture highly complex changes in movement directions.

Original: 4.3 Semi-Supervised Classiﬁcation
Revised: 4.3 Semi-Supervised Classiﬁcation

Original: We now extend the experimental protocol by targeting accurate label discovery in semi-supervisedscenarios.
Revised: We now extend the experimental protocol by targeting accurate label discovery in semi-supervised scenarios.

Original: We refer to the full model simply as MAT where λ 0 , λ 1 > 0 and incorporate the fully supervised S-MAT ( λ 0 = 0 ) as an additional baseline.
Revised: We refer to the full model simply as MAT where λ 0 , λ 1 > 0 and incorporate the fully supervised S-MAT ( λ 0 = 0 ) as an additional baseline.

Original: We also compare to an RNN-based classiﬁcationnetwork that addresses inter-agent dependencies by inferring hidden states of agents by the GNN architecture described in Section 3.5.
Revised: We also compare to an RNN-based classiﬁcation network that addresses inter-agent dependencies by inferring hidden states of agents by the GNN architecture described in Section 3.5.

Original: A ﬁnal softmax layer is used for classiﬁcation and the model isoptimized by minimizing the negative log-likelihood.
Revised: A ﬁnal softmax layer is used for classiﬁcation, and the model is optimized by minimizing the negative log-likelihood.

Original: We refer to this baseline as MA-RNN .
Revised: We refer to this baseline as MA-RNN .

Original: Figure 3 shows the performance of the models on NBA when using 50% labeled and 50% unlabeledexamples (left side) and when using varying portions of labeled data (right side), while the supervisedbaselines have access only to the labeled part.
Revised: Figure 4 shows the performance of the models on NBA when using 50% labeled and 50% unlabeled examples (left side) and when using varying portions of labeled data (right side), while the supervised baselines have access only to the labeled part.

Original: The semi-supervised MAT clearly emerges as thestrongest classiﬁer as it converges to the lowest generalization error during a training run and theperformance beneﬁt increases via reduction of labeled data.
Revised: The semi-supervised MAT clearly emerges as the strongest classiﬁer as it converges to the lowest generalization error during a training run and the performance beneﬁt increases via reduction of labeled data.

Original: Comparing the numbers to its supervisedpeer ( S-MAT ) highlights the value of incorporating unlabeled data that is effectively utilized by MATto inform classiﬁcation decisions.
Revised: Comparing the numbers to its supervised peer ( S-MAT ) highlights the value of incorporating unlabeled data that is effectively utilized by MAT to inform classiﬁcation decisions.

Original: Comparing the fully supervised approaches S-MAT and MA-RNN ,the importance of an effective regularization mechanism becomes obvious; MA-RNN quickly overﬁtswhile the lower-bound in S-MAT acts as a regularizer and guarantees gradually increasing predictiveaccuracies on validation sets. 
Revised: Comparing the fully supervised approaches S-MAT and MA-RNN , the importance of an effective regularization mechanism becomes obvious; MA-RNN quickly overﬁts while the lower-bound in S-MAT acts as a regularizer and guarantees gradually increasing predictive accuracies on validation sets. [Table]

Original: 4.4 Combining Generation and Classiﬁcation in Semi-Supervised Settings
Revised: 4.4 Combining Generation and Classiﬁcation in Semi-Supervised Settings

Original: In our last set of experiments, we study the combination of trajectory generation and classiﬁcationand resort to tracking data from elite soccer.
Revised: In our last set of experiments [Table] we study the combination of trajectory generation and classiﬁcation and resort to tracking data from elite soccer.

Original: The reason for favoring this data over NBA is thatthe soccer tracking data is manually annotated by experts from the data provider.
Revised: The reason for favoring this data over NBA is that the soccer tracking data is manually annotated by experts from the data provider.

Original: Since the datarequires human annotations for further processing, a successful semi-supervised application bearsa great deal of practical relevance by reducing the expensive labeling effort.

Original: We focus on labels
Revised: We focus on labels

Original: Y = { pass , other ball action , shot , none } , where class none denotes the absence of all other labels ina frame and use 20% annotated data.
Revised: Y = { pass , other ball action , shot , none } [Table] where class none denotes the absence of all other labels in a frame and use 20% annotated data.

Original: Every label is propagated to the previous ﬁve and the subsequent30 frames.
Revised: Every label is propagated to the previous ﬁve and the subsequent 30 frames.

Original: We generate a balanced training set where half of the segments carry label none and studythe combined task of trajectory generation and classiﬁcation.
Revised: We generate a balanced training set where half of the segments carry label none .

Original: This set of experiments mainly serves the purpose to quantitatively validate the beneﬁts of incorporating expensive human annotations as a generation guide or for structuring the discrete posterior. This
Revised: The main purpose of this set of experiments is to quantitatively validate the beneﬁts of including expensive human annotations in the proposed modeling framework for structuring the discrete latent subspace.

Original: [Table] approach comes with little additional costs since the method operates semi-supervised and thereforerequires only a small subset of multi-agent segments  to be annotated.
Revised: This approach has low additional costs because the method operates semi-supervised, so only a small subset of multi-agent segments needs to be annotated.

Original: Given the higher sampling rateand less involved plays in soccer dynamics, we also intend to evaluate the extent to which interactionmodules achieve empirical beneﬁts.
Revised: Given the higher sampling rate and less interactive nature of soccer dynamics, we also aim to evaluate the extent to which social dependency modeling modules achieve empirical beneﬁts.

Original: We design our baselines accordinly.
Revised: We design our baselines accordingly.

Original: Generation For trajectory prediction, we compare MAT to VRNN [7] that implicitly assumesindependence across agent dimensions, an interactive version of thereof with GNNs on the hiddenstates ( VRNN+GNN ), GVRNN [59], S-MAT with long-term goals (introduced in Section 4.2, anda diagonal MAT version, MAT-Diag .
Revised: Generation For trajectory prediction, we compare MAT to VRNN [9], an interactive version of thereof with GNNs on the hidden states ( VRNN+GNN ), GVRNN [61], S-MAT with long-term goals (introduced in Section 4.2), and a diagonal MAT version, MAT-Diag .

Original: Note that the VRNN constitutes a strong competitor for thetask at-hand, as it effectively corresponds to a diagonal GVRNN, which has been shown to producecompetitive results on soccer data [59].
Revised: Note that the VRNN constitutes a strong competitor for the task at-hand, as it effectively corresponds to a diagonal GVRNN, which has been shown to produce competitive results on soccer data [61].

Original: The left part of Table 3 shows the results.
Revised: The left part of Table 3 shows the results.

Original: Although the results are generally more similar comparedto the basketball experiments, the semi-supervised MAT generates trajectories that are closest in bothmetrics to the observed ones.
Revised: Although the results are generally more similar compared to the basketball experiments, the semi-supervised MAT generates trajectories that are closest in both metrics to the observed ones.

Original: We hypothesize that decreasing the frequency of the data and extractingmulti-agent segments that merely consist of highly interactive plays yields an signiﬁcant increase inperformance gap.
Revised: We hypothesize that decreasing the frequency of the data and extracting multi-agent segments that merely consist of highly interactive plays yields an signiﬁcant increase in performance gap.

Original: Classiﬁcation MAT learns a model over the label space Y simultaneously to the generative model.
Revised: Classiﬁcation MAT learns a model over the label space Y simultaneously to the generative model.

Original: To evaluate classiﬁcation performance, we turn the output of the corresponding softmax for a givenframe into a class label whenever it exceeded a predeﬁned threshold.
Revised: To evaluate classiﬁcation performance, we turn the output of the corresponding softmax for a given frame into a class label whenever it exceeded a predeﬁned threshold.

Original: We compare the predictionperformance again to MA-RNN , a diagonal variant MA-RNN-Diag , and S-MAT .
Revised: We compare the prediction performance to the classiﬁcation baselines introduced in Section 4.4.

Original: The resulting accuracies and F1 scores are summarized in Table 3.
Revised: The resulting accuracies and F1 scores are summarized in Table 3.

Original: Once more, MAT clearly beats thebaselines and stands out by the highest accuracy and F1 scores.
Revised: Once more, MAT clearly beats the baselines and stands out by the highest accuracy and F1 scores.

Original: The result impressively demonstratesthe beneﬁt of including unlabeled data for the task.
Revised: The result impressively demonstrates the beneﬁt of including unlabeled data for the task.

Original: Surprisingly, MA-RNN performs better than
Revised: Surprisingly, MA-RNN performs better than

Original: S-MAT .
Revised: S-MAT .

Original: This ﬁnding is in contrast to the results on NBA shown in Figure 3.
Revised: This ﬁnding is in contrast to the results on NBA shown in Figure 4.

Original: However, soccer isplayed on a much larger pitch and movements are likely more linear compared to basketball, whichleads to less multimodality of the distributions involved and a consequently decreased beneﬁt ofvariational methods.
Revised: However, soccer is played on a much larger pitch and movements are likely more linear compared to basketball, which leads to less multimodality of the distributions involved and a consequently decreased beneﬁt of variational methods.

Original: Together with a signiﬁcantly reduced label space and sufﬁcient amounts of datathis may lead to a simpler learning task where MA-RNN is less prone to overﬁtting.
Revised: Together with a signiﬁcantly reduced label space and sufﬁcient amounts of data this may lead to a simpler learning task where MA-RNN is less prone to overﬁtting.

Original: The left part of Fig.
Revised: The left part of Fig.

Original: 2 shows exemplary prediction probabilities for two segments and three possibleevents.
Revised: 3 shows exemplary prediction probabilities for two segments and three possible events.

Original: In both segments, the algorithm is highly conﬁdent no shot-action will occur in the near future.
Revised: In both segments, the algorithm is highly conﬁdent no shot-action will occur in the near future.

Original: The segment on top contains an event of the class other ball action followed by an event of class pass .
Revised: The segment on top contains an event of the class other ball action followed by an event of class pass .

Original: Both are correctly identiﬁed by MAT .
Revised: Both are correctly identiﬁed by MAT .

Original: The former is clearly indicated by a peak in the correspondingprobability chart which then decreases to give rise to the following pass action.
Revised: The former is clearly indicated by a peak in the corresponding probability chart, which then decreases to give rise to the following pass action.

Original: The segment on thebottom shows solely a single pass event that is also clearly identiﬁable by a peak at the correct frame.
Revised: The segment on the bottom shows solely a single pass event that is also clearly identiﬁable by a peak at the correct frame.

Original:  Conclusion
Revised:  Conclusion

Original: We presented semi-supervised variational autoencoders for spatiotemporal multi-agent scenarios.
Revised: We presented semi-supervised variational autoencoders for spatiotemporal multi-agent scenarios.

Original: The proposed approach effectively combined ideas from semi-supervised variational autoencoders,variational recurrent neural networks, and graph neural networks.
Revised: The proposed approach effectively combined ideas from semi-supervised variational autoencoders, variational recurrent neural networks, and graph neural networks.

Original: Empirically, our approach clearlyoutperformed the state-of-the-art  in  sequential generation and (semi-supervised) classiﬁcation tasks inall experiments.
Revised: Empirically, our approach clearly outperformed previous state-of-the-art models in interactive sequential generation and (semi-supervised) classiﬁcation tasks in all experiments.

Original: The performance underlines the beneﬁt of including unlabeled data in spatiotemporalproblems where labeled sequences are either scarce or assembled from weak makeshift signals.
Revised: The performance underlines the beneﬁt of including unlabeled data in spatiotemporal problems where labeled sequences are either scarce or assembled from weak makeshift signals.

Revised: Potential societal impact We have a clear focus on team sports.

Original: Checklist
Revised: Though dual use of the proposed technique is certainly possible (e.g., military domains), it may be a bit far fetched as we have no expertise in these domains and would only be able to deliver uneducated guesses about whether this is realistic or not. Generally, the data and tasks at-hand are privacy-sensitive as the targets are human individuals. While the soccer and basketball players are probably identiﬁable in the data, they/their contract/their clubs agreed in recording the data.  Acknowledgments

Revised: We would like to thank the German Football Association (DFB) for providing data for this study.

Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Since this work constitutes the ﬁrst semi-supervised approach for spatiotemporal multi-agent problems, it is difﬁcult to discuss techical limitations at this stage.
Revised: [Yes] Since this work constitutes the ﬁrst semi-supervised approach for spatiotemporal multi-agent problems, it is difﬁcult to discuss techical limitations at this stage.

Original: Application-wise, there are clear limitations e.g., by using weak makeshift labels.
Revised: Application-wise, there are clear limitations e.g., by using weak makeshift labels.

Original: We discuss these issues across the paper (e g., Section 
Revised: We discuss these issues across the paper (e.g., Section 4.)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We have a clear focus on team sports.
Revised: [No] We have a clear focus on team sports.

Original: Though dual use of the proposed technique is certainly possible (e.g., military domains), it may be a bit far fetched as we have no expertise in these domains and would only be able to deliver uneducated guesses about whether this is realistic or not.
Revised: Though dual use of the proposed technique is certainly possible (e.g., military domains), it may be a bit far fetched as we have no expertise in these domains and would only be able to deliver uneducated guesses about whether this is realistic or not.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section4.1.
Revised: [Yes] See Section 4.1. – 4.3.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] All proofs are contained in the Appendix.
Revised: [Yes] All proofs are contained in the Appendix.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The NBA data set is publicly available and we provide a URL in the paper.
Revised: [Yes] The NBA data set is publicly available and we provide a URL in the paper.

Original: The soccer data is private and must not be shared with third parties.
Revised: The soccer data is private and must not be shared with third parties.

Original: We will release the code and include a URL in the paper as well after the paper has been accepted.
Revised: We will release the code and include a URL in the paper as well after the paper has been accepted.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section  more details are provided in the Appendixl.
Revised: [Yes] See Section 4, more details are provided in the Appendixl.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix4.
Revised: [Yes] See Appendix 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...[Table] (a) If your work uses existing assets, did you cite the creators?[Yes]
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Table] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] See supplemental material[Table] (c) Did you include any new assets either in the supplemental material or as a URL?[Yes]
Revised: [Yes] See supplemental material (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re[Table] using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] The NBA data is publicly available and free, for soccer we have[Table] the permission of the data provider.[Table]
Revised: [Yes] The NBA data is publicly available and free, for soccer we have the permission of the data provider.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable[Table] information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [Yes] The soccer and basketball players are probably[Table] identiﬁable in the data, but they/their contract/their clubs agreed in recording the data.[Table]
Revised: [Yes] The soccer and basketball players are probably identiﬁable in the data, but they/their contract/their clubs agreed in recording the data.

Original: The soccer data is private and cannot be shared.
Revised: The soccer data is private and cannot be shared.

Original: [Table] If you used crowdsourcing or conducted research with human subjects...[Table] (a) Did you include the full text of instructions given to participants and screenshots, if 
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [Table]

Original: applicable?

Original: [N/A]

Original: [Table]

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review[Table] Board (IRB) approvals, if applicable?[N/A]
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount[Table] spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: A Proofs of the Theorems
Revised: A Proofs of the Theorems

Original: We begin with Theorem 2 for simplicity and sketch the proof of Theorem 1 afterwards.
Revised: We begin with Theorem 2 for simplicity and sketch the proof of Theorem 1 afterwards.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let H ( β ) be the entropy of quantity β .
Revised: Let H ( β ) be the entropy of quantity β .

Original: A lower bound on log p θ ( x ≤ T ) in Eqn (1) isgiven by
Revised: A lower bound on log p θ ( x ≤ T ) in Eqn (1) is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For unlabeled training instances, deﬁning the marginal likelihood according to the generativestructure introduced in Section 3.3 yields
Revised: For unlabeled training instances, deﬁning the marginal likelihood according to the generative structure introduced in Section 3.3 yields

Original: [Equation]
Revised: [Equation]

Original: To derive the lower bound on the log likelihood
Revised: To derive the lower bound on the log likelihood

Original: [Equation]
Revised: [Equation]

Original: we incorporate the variational information into theabove deﬁnition and apply Jensen’s inequality:
Revised: we incorporate the variational information into the above deﬁnition and apply Jensen’s inequality:

Original: [Equation]
Revised: [Equation]

Original:  , y <t 
Revised: t | x ≤ t , z <t , y <t ) (cid:107) p θ ( y t | x <t , z <t , y <t )]

Original: [Equation]
Revised: [Equation]

Original: Thus, we can write
Revised: Thus, we can write

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original:  which is identical to Eqn 3.
Revised: , which is identical to Eqn 3.

Original: Theorem 1.
Revised: Theorem 1.

Original: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by 
Revised: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by log p θ ( x ≤ T , y ≤ T ) ≥ (cid:88) t [Equation] |

Original: [Equation]
Revised: [Equation]

Revised: z

Original: [Equation]
Revised: [Equation]

Original: ≤ T
Revised: t [Equation] y <t )] ≡ [Equation]

Revised: y

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: B Implementation Details
Revised: B Implementation Details

Original: All models are implemented using

Original: PyTorch [42].

Original: For our experiments, wedeﬁne the model components as follows.
Revised: For our experiments, we deﬁne the model components as follows.

Revised: We individually capture the spatiotemporal dependencies of the label-speciﬁc patterns y ≤ T and the data-speciﬁc patterns x ≤ T , z ≤ T , resulting in two separate variables o ( a ) , l t and o ( a ) , d t .

Revised: For ease of exposition, we refer to the concatenation of the two parts as o ( a ) t in the remainder of this paper.

Original: The encoders are modeled asq φ (˜ y ( a ) | x ≤ t , z <t , y <t ) =
Revised: The encoders are modeled as q φ (˜ y ( a ) | x ≤ t , z <t , y <t ) =

Original: Cat ( y ( a ) t | f y enc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t )]) and q φ ( z ( a ) t | x ≤ t , z <t , y ≤ t ) =
Revised: Cat ( y ( a ) t | f yenc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t )]) and q φ ( z ( a ) t | x ≤ t , z <t , y ≤ t ) =

Original: N ( z ( a ) t | f
Revised: N ( z ( a ) t | f

Original: z enc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t ) , ϕ y ( y ( a ) t )]) for uncovering discrete and continuous latent information,respectively.
Revised: z enc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t ) , ϕ y ( y ( a ) t )]) for uncovering discrete and continuous latent information, respectively.

Original: The prior distributions are computed analogously omitting input x t .
Revised: The prior distributions are computed analogously omitti

Original: Agent movementsare represented as p θ ( x ( a ) t | x <t , z ≤ t , y ≤ t ) =

Original: N ( z ( a ) t | f dec ([ o ( a ) t − 1 , ϕ x ( x ( a ) t ) , ϕ y ( y ( a ) t )]) .

Original: All functions f and feature extractors ϕ are two-layer MLPs with LeakyRelu [56] activations.
Revised: All functions f and feature extractors ϕ are two-layer MLPs with LeakyRelu [58] activations.

Original: We use2 -layer GRU networks [6] for recurrence.
Revised: We use 2 -layer GRU networks [8] for recurrence.

Original: Moreover, we deﬁne the graph network using 3 GAT layersimplemented according to [55] and the graph structure wrt.the k
Revised: Moreover, we deﬁne the graph network using 3 GAT layers implemented according to [57] and the graph structure wrt.the k

Original: = 2 and k = 5 spatially closestagents for basketball and soccer, respectively.
Revised: = 2 and k = 5 spatially closest agents for basketball and soccer, respectively.

Original: We implement skip-connections as described in [57]and use concatenation followed by a linear transformation to aggregate intermediate layer embeddingsto the GNN model output.
Revised: We implement skip-connections as described in [59] and use concatenation followed by a linear transformation to aggregate intermediate layer embeddings to the GNN model output.

Original: For generative tasks, we use λ 1 = 0 , however, including the auxiliary lossyields negligible deterioration.
Revised: For generative tasks, we use λ 1 = 0 , however, including the auxiliary loss yields negligible deterioration.

Original: For classiﬁcation, we use λ 0 = 1 and α = 10 ∗ (1 / % of labeled data ) .
Revised: For classiﬁcation, we use λ 0 = 1 and α = 10 ∗ (1 / % of labeled data ) .

Original: The model operates solely on agent velocities (input and output).
Revised: The model operates solely on agent velocities (input and output).

Original: However, we inject position tuplesto the all model components via teacher forcing.
Revised: However, we inject position tuples to the all model components via teacher forcing.

Original: We use the gumbel-softmax trick [35, 23] whensampling from categorical distributions.
Revised: We use the gumbel-softmax trick [37, 25] when sampling from categorical distributions.

Original: Training was executed on an Nvidia V100 GPU and tookabout 16 hours to complete 300 epochs while comsuming ∼ 15 Gb for basketball.
Revised: Training was executed on an Nvidia V100 GPU and took about 16 hours to complete 300 epochs while comsuming ∼ 15 Gb for basketball.

Original: All models areoptimized using Adam [25] with a learning rate of 0 , 001 and gradient clipping using a max norm of10 .
Revised: All models are optimized using Adam [27] with a learning rate of 0 , 001 and gradient clipping using a max norm of 10 .

Revised: All models are implemented using PyTorch [44].

Original: Baselines As stated in Section 4.1, we report against the values reported in [40].
Revised: Baselines As stated in Section 4.1, we report against the values reported in [42].

Original: For [16, 33, 37],we adapt the source code from their ofﬁcial repositories to our experimental setting 6 For GVRNN[59], we re-implemented the model according to the descriptions in their paper and designed theoverall architecture such that it is comparable in parameter number to our method.
Revised: For [18, 35, 39], we adapt the source code from their ofﬁcial repositories to our experimental setting 6 For GVRNN [61], we re-implemented the model according to the descriptions in their paper and designed the overall architecture such that it is comparable in parameter number to our method.

Original: Soccer application Since the labels denote ball-centric events, we use the output of the ball nodefor loss computation and evaluation.
Revised: Soccer application Since the labels denote ball-centric events, we use the output of the ball node for loss computation and evaluation.

Original: The F 1 score is computed as follows.
Revised: The F 1 score is computed as follows.

Original: We annotate a multiagent segment when the derived probability estimates exceed an externally deﬁned threshold value.
Revised: We annotate a multiagent segment when the derived probability estimates exceed an externally deﬁned threshold value.

Original: We obtain TP values (FP values) when the predicted event coincides (disagrees) with the groundtruth annotation.
Revised: We obtain TP values (FP values) when the predicted event coincides (disagrees) with the ground truth annotation.

Original: FN values are deﬁned by anotated segments that remain undetected.
Revised: FN values are deﬁned by anotated segments that remain undetected.

Original: We compute
Revised: We compute

Original: F1-scores for 100 distinct threshold values in the range between 0 . 5 and 0 .
Revised: F1-scores for 100 distinct threshold values in the range between 0 . 5 and 0 .

Original: 98 and only report themaximum F1-score.
Revised: 98 and only report the maximum F1-score.

Original: However, threshold optimization yields only negligible improvement over simplyusing 0 .
Revised: However, threshold optimization yields only negligible improvement over simply using 0 .

Original: C MAT Model Details 
Revised: C More Details on the MAT Models

Original: In Section 3.1, we deﬁne y ≤ T as arbitrary discrete (possibly latent) behavioral indicators.
Revised: In Section 3.1, we deﬁne y ≤ T as arbitrary discrete (possibly latent) behavioral indicators.

Original: To validatethis general formulation (and our proposed architecture), we vary the speciﬁc deﬁnition of the variable across experiments.
Revised: To validate this general formulation (and our proposed architecture), we vary the speciﬁc deﬁnition of the variable across experiments.

Original: Accordingly, we propose different framework instantiations that are described inmore detail in this section.
Revised: Accordingly, we propose different framework instantiations that are described in more detail in this section.

Original: S-MAT First, we observe in Section 4.2 that existing SOTA trajectory prediction approaches(Section 4.1) use heuristically generated labels for trajectory prediction that encode agents’ intentsor goals over a discretized position space.
Revised: S-MAT First, we observe in Section 4.2 that existing SOTA trajectory prediction approaches (Section 4.1) use heuristically generated labels for trajectory prediction that encode agents’ intents or goals over a discretized position space.

Original: We then note that our formalization allows to naturallyintegrate such long-term goals into the overall scheme via treating them as discrete semantic concepts.
Revised: We then note that our formalization allows to naturally integrate such long-term goals into the overall scheme via treating them as discrete semantic concepts.

Original: Since these labels are generated heuristically based on the trajectory input prior to model training,this model instantiation is fully-supervised and is referred to as S-MAT .
Revised: Since these labels are generated heuristically based on the trajectory input prior to model training, this model instantiation is fully-supervised and is referred to as S-MAT .

Original: We refer to [64] for moredetails on how to produce the weak labels used for training S-MAT.
Revised: We refer to [66] for more details on how to produce the weak labels used for training S-MAT.

Original: U-MAT
Revised: U-MAT

Original: However, we found (although done in [64, 40]) that benchmarking against unsupervisedbaselines is inappropriate for the reasons described in Section 4.1.
Revised: However, we found (although done in [66, 42]) that benchmarking against unsupervised baselines is inappropriate for the reasons described in Section 4.1.

Original: Since most SOTA approachesare unsupervised generative models, we additionally propose a fully-unsupervised instantiation ofour framework ( U-MAT ) employing no external guidance for discrete latent structuring.
Revised: Since most SOTA approaches are unsupervised generative models, we additionally propose a fully-unsupervised instantiation of our framework ( U-MAT ) employing no external guidance for discrete latent structuring.

Original: From thepredicted (latent) categorical distribution q φ , we sample a label value and exploit separate motionpredictors (different decoder parameterizations) p θ dependent on the realized value.
Revised: From the predicted (latent) categorical distribution q φ , we sample a label value and exploit separate motion predictors (different decoder parameterizations) p θ dependent on the realized value.

Original: Intuitively, thisencourages the model to learn categories describing fundamentally different movement patterns,which can be interpreted as dynamic “agent roles”.
Revised: Intuitively, this encourages the model to learn categories describing fundamentally different movement patterns, which can be interpreted as dynamic “agent roles”.

Original: Thus, the concept of agent roles here is merelyan intuitive explanation and realized via an inductive bias that increases the “scope” of latentinformation encoded without utilizing any supervision.
Revised: Thus, the concept of agent roles here is merely an intuitive explanation and realized via an inductive bias that increases the “scope” of latent information encoded without utilizing any supervision.

Original: To the best of our knowledge, parameterizingthe generation module based on inferred agent categories is novel and could provide valuable insightsfor practitioners.
Revised: To the best of our knowledge, parameterizing the generation module based on inferred agent categories is novel and could provide valuable insights for practitioners.

Original: D Experiments with Drone Data
Revised: D Experiments with Drone Data

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: To showcase applicability to scenarios that exhibt variable numbers of agents, we also report resultson the Stanford Drone Data (SDD) [44].
Revised: To showcase applicability to scenarios that exhibt variable numbers of agents, we also report results on the Stanford Drone Data (SDD) [46].

Original: SDD is a collection of videos recorded by drones at eightlocations at Stanford.
Revised: SDD is a collection of videos recorded by drones at eight locations at Stanford.

Original: While pedestrians predominate as interacting agents, cyclists, skateboarders,cars, buses, and golf carts are also present.
Revised: While pedestrians predominate as interacting agents, cyclists, skateboarders, cars, buses, and golf carts are also present.

Original: We use the TrajNet benchmark [47] of the data, providingsequential two-dimensional real-world coordinates at a frame rate of 2.5 frames per second.
Revised: We use the TrajNet benchmark [49] of the data, providing sequential two-dimensional real-world coordinates at a frame rate of 2.5 frames per second.

Original: Wefollow [40] regarding the data processing strategy 7 .
Revised: We follow [42] regarding the data processing strategy 7 .

Original: We incorporate PECNet [37] for comparison asit is considered state-of-the-art for SDD [36].
Revised: We incorporate PECNet [39] for comparison as it is considered state-of-the-art for SDD [38].

Original: PECNet models stochasticity in the ﬁnal position of thepedestrians conditioned on the past motion history.
Revised: PECNet models stochasticity in the ﬁnal position of the pedestrians conditioned on the past motion history.

Original: To generate weak labels, we move a time window through each trajectory, with the respective endcells acting as agent targets y ( a ) t .
Revised: To generate weak labels, we move a time window through each trajectory, with the respective end cells acting as agent targets y ( a ) t .

Original: Static time windows allow us to use the fully-supervised variant forall comparisons (including fully unsupervised models).
Revised: Static time windows allow us to use the fully-supervised variant for all comparisons (including fully unsupervised models).

Original: The scene at hand is thereby discretized into960 areas and results are shown in Table 4.
Revised: The scene at hand is thereby discretized into 960 areas and results are shown in Table 4.

Original: We observe the same general pattern as for the basketballexperiments.
Revised: We observe the same general pattern as for the basketball experiments.

Original: [Table]
Revised: [Table]

Original: E Ablation Study

Original: [Table]

Original: Table 5 validates the proposed architecture by showing results of an ablation study.
Revised: Table 5 validates the proposed architecture by showing results of an ablation study.

Original: We test a fullyconnected graph ( S-MAT-Full ), an independent version with diagonal adjacency matrices ( S-MATDiag ), a variant that employs GNNs for variational, generative, and prior distribution parameters, butignores interactive updates for the hidden states ( S-MAT-GRVNN ) similar to [59], and a variant that isidentical to the latter but additionally includes a GNN for the hidden state ( S-MAT-GRVNN-Hidden ).
Revised: We test a fully connected graph ( S-MAT-Full ), an independent version with diagonal adjacency matrices ( S-MATDiag ), a variant that employs GNNs for variational, generative, and prior distribution parameters, but ignores interactive updates for the hidden states ( S-MAT-GRVNN ) similar to [61], and a variant that is identical to the latter but additionally includes a GNN for the hidden state ( S-MAT-GRVNN-Hidden ).

Original: The table provides supporting evidence for our design choices: though the GVRNNs are theoreticallyable to capture intra-timestep dependencies, we observe signiﬁcant drops in performance compared tothe other competitors.
Revised: The table provides supporting evidence for our design choices: though the GVRNNs are theoretically able to capture intra-timestep dependencies, we observe signiﬁcant drops in performance compared to the other competitors.

Original: The S-MAT-Diag and MAT-Full experiments suggest that our model capturesinteraction patterns among agents very well; S-MAT denotes a valuable contribution to the large bodyof research that explicitly addresses modeling multi-agent data accurately.
Revised: The S-MAT-Diag and MAT-Full experiments suggest that our model captures interaction patterns among agents very well; S-MAT denotes a valuable contribution to the large body of research that explicitly addresses modeling multi-agent data accurately.