Original: Latent Variable Models for Multi-Agent Trajectories
Revised: Latent Variable Models for Multi-Agent Trajectories

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: The supervised identiﬁcation of spatiotemporal patterns usually requires a metric over time and space and a great deal of annotated situations.
Revised: Analyzing the spatiotemporal behavior of multiple agents is of great interest tomany communities. Existing probabilistic models in this regime employ eitherunsupervised generative settings, in which the latent space is described fully bydiscrete or continuous representations, or, are alternatively formalized in a (fully)supervised framework where weakly preserved labels add explicit information to acontinuous latent representation learned from the data.

Original: To overcomethese limitations, we present a generative framework for modeling and annotating trajectories of multiple agents over time and space. Our approach leveragessemi-supervised learning for variational autoencoders where equivariance and interaction of agents are appropriately represented by graph networks.
Revised: To overcome the resultinglimitations, we propose a novel objective function for processing multi-agent trajectories based on semi-supervised variational autoencoders, where equivariance andinteraction of agents are modeled via costumized graph networks.

Original: The proposedapproach is generally applicable to a wide range of problems, including generation, classiﬁcation, and pattern detection.
Revised: Our formulationdisentangles discrete and continuous effects and allows discrete behavioral indicators in arbitrary quantity and annotation type to guide the generation process. Thislifts applicability to relevant prediction problems beyond the generation of collective movements and provides an effective solution to incoporate expensive domainknowledge into interactive multi-agent systems.

Original: Empirically, our approach outperformsall competitors on various interactive spatiotemporal problems
Revised: Empirically, our model outperforms various state-of-the-art baselines in generating future agent movements oninteractive real-world datasets.

Revised: We also show that our approach effectively learns toleverage unsupervised multi-agent sequences to improve classiﬁcation of long-termlocations as well as manually annotated situations on sports tracking data.

Original:  Introduction
Revised:  Introduction

Original: Analyzing the spatiotemporal behavior of multiple agents bears a great deal of value in many domains such as autonomous driving [48, 13], public transportation [30, 57], migration [38, 22] andsports analytics [52].
Revised: Analyzing the spatiotemporal behavior of multiple agents bears a great deal of value in many domainssuch as autonomous driving [49, 13], public transportation [30, 58], migration [39, 22] and sportsanalytics [53].

Original: Particularly, the detection of collective patterns across time and space is ofinterest to many communities but non-trivial inter-dependencies between agents render estimatingthe involved multi-modal distributions often difﬁcult.
Revised: Particularly, the detection of collective patterns across time and space is of interest tomany communities but non-trivial inter-dependencies between agents render estimating the inherentmulti-modal distributions often difﬁcult.

Original: Existing  approaches to spatiotemporal pattern detection thus capitalize on variants from graph networks [54, 12] and  variational autoencoders [26, 7]; being purely unsupervised, their functionalityis grounded in uncovering the implicit modular structure in agent-wise latent variables directly fromthe observed trajectories via approximate inference [28, 51, 58, 16, 14, 32].
Revised: Existing generative approaches to modeling sequential multi-agent data thus capitalize on variantsfrom graph networks [55, 12] and (sequential) variational autoencoders [26, 7]; being purely unsupervised, their functionality is grounded in uncovering the implicit modular structure in agent-wise latentvariables directly from the observed trajectories via approximate inference [28, 52, 59, 16, 14, 33].

Original: Although a generativeview on the problem is very appealing, approaches in this regime either lack interpretable and controllable latent factors (e.g., [58]), and are thus limited in practical applicability, or put a strong focuson relationship discovery (e.g., [28]), which limits their generative capacity.
Revised: Although a generative view on the problem is very appealing, approaches in this regime either lackinterpretable and controllable latent factors (e.g., [59]), and are thus limited in practical applicability,or put a strong focus on relationship discovery (e.g., [28]), which limits their generative capacity.

Original: An alternative is offered by supervised approaches that aim to counteract these issues by incorporating domain-knowledge in form of discrete labels into the learning process [62].
Revised: An alternative is offered by supervised approaches that aim to counteract these issues by incorporatingdomain-knowledge in form of discrete labels into the generation process [64].

Original: Ideally, these labelsencode intents or goals of the agents. In practical applications, however, such annotations hardlyexist.
Revised: Ideally, these labelsencode implicit expert knowledge that would be learned by the model and substantially expand thescope of information captured in latent space.

Original: Instead of arguing in favor of expensive and tedious manual annotations of spatiotemporaldata, recent approaches propose to incorporate heuristic surrogates as makeshift labels into the problem and report impressive predictive accuracies [63, 62, 39].
Revised: Instead of arguing in favor of expensive and tediousmanual annotations of spatiotemporal data, however, recent approaches propose to incorporateheuristic surrogates as makeshift labels into the problem [65, 64, 40].

Original: However, these weakly supervisedapproaches require the existence of label sequences for all training instances and cannot includeinexpensive and possibly abundant unlabeled data due to their purely supervised nature.
Revised: While reporting impressivepredictive accuracies, they require the existence of label sequences for all training instances and cannot include inexpensive and possibly abundant unlabeled data due to their purely supervisednature.

Original: To close this gap, we propose semi-supervised variational autoencoders for spatiotemporal multiagent problems to process (weakly) labeled as well as unlabeled data.
Revised: To close this gap, we propose semi-supervised variational autoencoders for spatiotemporal multi-agentproblems to process (weakly) labeled as well as unlabeled data.

Original: Our contribution generalizesthe class of semi-supervised variational autoencoders [27] to spatiotemporal domains by using ideasfrom variational recurrent neural networks [7].
Revised: Our contribution generalizes theclass of semi-supervised variational autoencoders [27] to spatiotemporal domains by using ideas fromvariational recurrent neural networks [7].

Revised: At a high level, the derived objective function subsumesprevious formalizations into a uniﬁed framework via a disentangled latent space that can incorporatesupervision signals in arbitrary quantities for structuring the discrete latent subspace.

Original: Additional adaptive graph network layers provideorder in- and equivariance of agents and render the proposed approach appropriate for multi-agentscenarios.
Revised: Additionaladaptive graph network layers provide order in- and equivariance of agents and render the proposedapproach appropriate for multi-agent scenarios.

Original: The resulting semi-supervised approach is applicable to a wide range of problems including the generation of collective movements as well as classifying situations of interest.
Revised: The resulting semi-supervised approach is applicableto a wide range of problems including the generation of collective movements as well as classifyingsituations of interest.

Original: Empirically, our contribution signiﬁcantly advances the state-of-the art in generation and classiﬁcation tasks involving multiple agents.
Revised: Empirically, our contribution signiﬁcantly advances the state-of-the art ingeneration and classiﬁcation tasks on interactive real-world data.

Original:  Related Work
Revised:  Related Work

Original: Semi-supervised generative models Semi-supervised variational autoencoders have been originally studied byKingma et al. [27] who propose to incorporate additional labels in latent space.
Revised: Semi-supervised generative models Semi-supervised variational autoencoders have been originally studied by [27] who propose to incorporate additional labels in latent space.

Original: Related approaches introduce auxiliary variables that leave the original model unchanged but increase the ﬂexibility of the variational posterior [33].
Revised: Related approachesintroduce auxiliary variables that leave the original model unchanged but increase the ﬂexibility ofthe variational posterior [34].

Original: Joy et al.

Original: [24] propose to explicitly capturelabel characteristics in latent space instead of the label values themselves and Narayanaswamy et al.[40] study generalizations that allow for learning more complex dependency structures.
Revised: [24] propose to explicitly capture label characteristics in latent spaceinstead of the label values themselves and [41] study generalizations that allow for learning morecomplex dependency structures.

Original: However,all the above approaches are introduced only for static domains.
Revised: However, all the above approaches are introduced only for staticdomains.

Original: Our contribution constitutes ageneralization of the above approaches to spatiotemporal domains.
Revised: Our contribution constitutes a generalization of the above approaches to spatiotemporaldomains.

Original: Sequential generative models Approaches designed for predicting agent trajectories frequentlyleverage ideas from sequential generative models to address the stochasticity inherent in future predictions.
Revised: Sequential generative models Approaches designed for predicting agent trajectories are frequentlybased on ideas from sequential generative models to encode temporal dependencies and addressthe stochasticity inherent in future predictions.

Original: Extensions of variational autoencoders in this respect include STORN [3], VRNN [7], SRNN [11], or Z-Forcing [15].
Revised: Sequential extensions of variational autoencoderswith dynamic latent variables include [3, 7, 11, 15].

Original: Particularly, the VRNN [7] is related to our contribution as both deploy a VAE instance per time step conditioned on a recurrent state to capture temporal dependencies.
Revised: Particularly, the VRNN [7] is related to ourcontribution as both deploy a VAE instance per time step conditioned on a recurrent state.

Revised: However,their work lacks modeling of a social dimension as well as means to integrate (potentially expensive)discrete behavioral indicators.

Original: Other approaches associate a single global latent variable with each sequence [4, 17, 9, 8].
Revised: Other approaches associate a single global latent variable with eachsequence [4, 17, 9, 8].

Original:  neural networks Deep graph-based (GNNs) approaches [49] are a natural choice for applications requiring permutation invariance of the inputs.
Revised: Graph neural networks Deep graph-based (GNNs) approaches [50] are a natural methodologicalchoice for applications requiring modeling datasets composed of sets.

Original: These methods represent inputs as graphnodes where connecting edges capture interactions between the nodes.

Original: GNNs operate by learninga chain of hidden representations for each node through an iterative process that relies on aggregating messages along edges in each network layer.
Revised: GNNs operate by learning achain of hidden representations for each node through an iterative process that relies on aggregatingmessages along edges in each network layer.

Original: Prominent instantiations such as GCN [29], GraphSAGE [18], GAT [54], and others [12, 53, 20, 2, 61] mainly differ in their notion of messagepassing and neighborhood aggregation [59].
Revised: Prominent instantiations such as GCN [29], GraphSAGE[18], GAT [55], and others [12, 54, 20, 2, 63, 57] mainly differ in their notion of message passing andneighborhood aggregation [60].

Original: For example, GATs incorporate directoral biases intothe aggregation via attention mechanisms.

Original: GNN layers can be stacked and allow for context-basedand adaptive neighborhoods for each node by incorporating particularly designed skip connections[56].

Original: See Chen et al.

Original:  [5] for a discussion on the function approximation capabilities of this neuralnetwork family.
Revised: See [5] for a discussion on the function approximation capabilitiesof this neural network family.

Original: Generative models for sequential multi-agent data In the remainder, we empirically compareour approach with state-of-the-art methods in spatiotemporal pattern detection for multiple agents.

Original:  Given the potential beneﬁt across various domains, a great deal of recent publications focus onmovements of pedestrians or self-driving cars [36, 48, 13, 37, 60, 47, 1, 21]. Makansi et al.
Revised: Generative models for sequential multi-agent data Given the potential beneﬁt across variousdomains, a great deal of recent publications focus on movements of pedestrians or self-driving cars[37, 49, 13, 38, 61, 48, 1, 21, 62, 32].

Original: [35],however, show that these standard benchmark datasets generally exhibit weak social interactionpatterns, so in the remainder, we focus on methods using team sport data.
Revised: [36], however, show that these standard benchmark datasetsgenerally exhibit weak social interaction patterns, so in the remainder, we focus on methods usingteam sport data.

Original: Related approachesshow that learning and computing realistic rollouts signiﬁcantly beneﬁt from the incorporation ofmakeshift annotations and heuristic surrogates into the problem [63, 62, 39]. Felsen et al. [10] and Liet al.
Revised: Related approaches show that learning and computing realistic rollouts signiﬁcantlybeneﬁt from the incorporation of makeshift annotations and heuristic surrogates into the problem[65, 64, 40].

Original: [32] propose an architecture based on conditional VAEs [50] to model distributions of futuremovements of basketball players.
Revised: [10, 33] propose an architecture based on conditional VAEs [51] to model distributionsof future movements of basketball players.

Original: Yeh et al.

Original: [58] and Sun et al. [51] combine ideas from graphnetworks [12] and VRNNs [7] into a uniﬁed framework aiming at modeling data from basketballand soccer.
Revised: [59, 52] combine ideas from graph networks [12] and VRNNs [7] into a uniﬁed framework aiming at modeling data from basketball and soccer.

Original: Other related approaches [31, 28, 16, 14] aim to explicitly infer discrete latent variablesto represent interaction types while executing a trajectory prediction task.
Revised: Otherrelated approaches [31, 28, 16, 14] aim to explicitly infer discrete latent variables to representinteraction types while executing a trajectory prediction task.

Original: Hence, our contribution constitutes the ﬁrst semi-supervised generalization of variational autoencoders for spatiotemporal domains that lifts applicability to a wide range of prediction problems andgives rise to addressing tasks beyond representational or generative modeling.
Revised: Hence, existing generative models for spatiotemporal pattern detection focus on multi-agent trajectory prediction. Our contribution constitutes the ﬁrst semi-supervised generalization of variationalautoencoders for spatiotemporal domains that gives rise to addressing tasks beyond representationalor generative modeling.

Original:  Main Contribution3.1 Variational Autoencoders
Revised:  Methods

Revised: 3.1 Preliminaries

Original: Variational autoencoders (VAEs) [26, 42] allow to capture intrinsic conditional dependencies andare often used in structured problems.
Revised: Variational autoencoders (VAEs) [26, 43] allow to capture intrinsic conditional dependencies andare often used in structured problems.

Original: Essentially, they consist of a θ -parameterized generative
Revised: Essentially, they consist of a θ -parameterized generativemodel p θ ( x , z ) =

Revised: p ( z ) p θ ( x | z ) and an associated φ -parameterized variational distribution q φ ( z | x ) .

Original: Optimal parameters θ are computed by maximizing the variational lower bound on the marginallikelihood, given by
Revised: Optimal parameters { θ, φ } are obtained via maximizing the variational lower bound on the marginallikelihood, given by

Original: [Equation]
Revised: [Equation]

Original: It is often helpful to augment discrete labels y into the model to guide the generating process [27].
Revised: It is often helpful to augment discrete labels y into the model to guide the generating process [27].

Original: The resulting model p ( x , z , y ) =
Revised: The resulting model p ( x , z , y ) =

Original: p ( x j z , y ) p ( y ) p ( z ) then acts on labeled and unlabeled instances.
Revised: p ( x | z , y ) p ( y ) p ( z ) then acts on labeled and unlabeled instances.

Original: Consequentially, the variational distribution q ϕ ( y, z j x ) =
Revised: Consequentially, the variational distribution q φ ( y, z | x ) =

Original: q ϕ ( y j x ) q ϕ ( z j y, x ) needs to be deﬁnedover both quantities, latents z andlabels y .
Revised: q φ ( y | x ) q φ ( z | y, x ) needs to be deﬁnedover both quantities ( z and y ).

Original: Semi-supervised VAEs  are trained by maximizing
Revised: Semi-supervised VAEs (SSVAE) are trained by maximizing

Original: [Equation]
Revised: [Equation]

Original: where L ( x , y )
Revised: where L ( x , y )

Original: =
Revised: =

Original: E q ϕ ( z | x ,y )
Revised: E q φ ( z | x ,y )

Original: [log p θ ( x j y, z ) + log p ( y ) + log p ( z ) (cid:0) log q ϕ ( z j x , y )] and U ( x ) = [-
-]
Revised: [log p θ ( x | y, z ) + log p ( y ) + log p ( z ) − log q φ ( z | x , y )] and U ( x ) = [-
-]102 (cid:80)

Original: ∑ y q ϕ ( y j x )( L ( x , y ))
Revised: y q φ ( y | x )( L ( x , y ))

Original: + H ( q ϕ ( y j x )) denote (variational) lower bounds on labeled and unlabeledinstances, respectively, and λ is a balancing term.
Revised: + H ( q φ ( y | x )) denote (variational) lower bounds on labeled and unlabeledinstances, respectively, and λ is a balancing term.

Original: In fact, the original motivation for this framework was inspired by semi-supervised classiﬁcation tasks.
Revised: In fact, the original motivation for this frameworkwas inspired by semi-supervised classiﬁcation tasks.

Original: Meanwhile, their contribution is alsoused for learning meaningful representations and generating new data, thus rendering it a suitablemethodological foundation for performing tasks with all sorts of data.
Revised: However, their contribution is also regularly [Figure] used for learning meaningful representations and generating new data, thus rendering it a suitablemethodological foundation for performing tasks with all sorts of data.

Revised: The dependency structures ofq φ and p θ for SSVAEs are displayed visually as part of Figure 1 (right side).

Original: 3.2 Problem Formulation
Revised: 3.2 Problem Formulation

Original: We are interested in modeling spatiotemporal multi-agent scenarios and focus on positions x ( a ) t [-
-]
Revised: We are interested in modeling spatiotemporal multi-agent scenarios and focus on positions x ( a ) t [-
-]∈ R 2of an agent a ∈

Original: R 2 of an agent a 2 A over time 1 (cid:20) t (cid:20) T , denoted by x ( a ) ≤ T .
Revised: A over time 1 ≤ t ≤ T , denoted by x ( a ) ≤ T .

Original: We assume that a non-empty subsetof agents in A is present in any observed segment and collect their trajectories (in random order) toform a collective movement x ≤ T :=
Revised: We assume that a non-empty subset ofagents in A is present in any observed segment and collect their trajectories (in random order) to formcollective movements x ≤ T :=

Original: f x ( a ) ≤ T j a 2 Ag .
Revised: { x ( a ) ≤ T | a ∈ A} .

Original: We will make use of velocities d x /dt andlinearized motion ∆ x t = x t ′ (cid:0) x t , for t ′ > t , of agents in the remainder.
Revised: We will make use of velocities d x /dt and linearizedmotion ∆ x t = x t (cid:48) − x t , for t (cid:48) > t , of agents in the remainder.

Original: In addition, we introduce discrete labels y
Revised: In addition, we introduce discrete labels y

Original: ( a ) t 2 Y associated with agent a 2 A at time t .
Revised: ( a ) t ∈ Y associated with agent a ∈ A at time t .

Original: Thecollection of sequences y ≤ T :=
Revised: Thecollection of sequences y ≤ T :=

Original: f y ( a ) ≤ T j a 2 Ag may be best thought of as discriminative causesof variation in the corresponding trajectories x ≤ T .
Revised: { y ( a ) ≤ T | a ∈ A} may be best thought of as discriminative causesof variation in the corresponding trajectories x ≤ T .

Original: In practice, y ≤ T may contain anything thataids in generating multi-agent rollouts, including behavioral indicators, long-term goals, or currenttasks.
Revised: In practice, y ≤ T may contain anything thataids in generating multi-agent rollouts, including behavioral indicators, long-term goals, or currenttasks.

Original: Finally, observations x ≤ T accompanied by labels y ≤ T assemble the labeled share of the data,denoted by D L = f ( x ≤ T , y ≤ T ) n g 1 ≤ n ≤ N , while unlabeled observations are collected in D U
Revised: Finally, observations x ≤ T accompanied by labels y ≤ T assemble the labeled share of thedata, denoted by D L = { ( x ≤ T , y ≤ T ) n } 1 ≤ n ≤ N , while unlabeled observations are collected in D U

Original: = 110 f ( x ≤ T ) m g 1 ≤ m ≤ M .
Revised: = { ( x ≤ T ) m } 1 ≤ m ≤ M .

Original: We refer to all data by D = D L [ D U .
Revised: We refer to all data by D = D L ∪ D U .

Original: We aim at estimating the underlying distribution that generated the observations in D .
Revised: We aim at estimating the underlying distribution that generated the observations in D .

Original: A straightforward way is tosimply maximize the log likelihood given by
Revised: A straightforward solution is to maximize the log likelihood given by

Original: [Equation]
Revised: [Equation]

Original: However, human movement is inherently multi-modal [anonymized reference] which renders maximum likelihood clearly inappropriate.
Revised: However, human movement is inherently non-determenistic and multi-modal.

Original: It istherefore common to model the stochasticity inherent infuture agent behaviors via latent variables.
Revised: Thus, it is common tomodel the stochasticity inherent in future agent behaviors via latent representations of VAEs.

Original: To lower bound the objective, we need to derive lowerbounds on log p θ ( x ≤ T ) and log p θ ( x ≤ T , y ≤ T ) as follows.
Revised: Wepropose to lower-bound Eq. 1 as follows.

Original: 3.3 VAEs for Multi-Agent Trajectories
Revised: 3.3 SSVAEs for Multi-Agent Trajectories

Original: We associate disentangled latent realizations f z t , y t g with each time step t of the segment whileinjecting the dependency structures introduced in Section 3.1 for the generative and inference parts.
Revised: We associate disentangled latent realizations { z t , y t } with each time step t of the segment whileinjecting the dependency structures introduced in Section 3.1 for the generative and inference parts.

Original: To model temporal dependencies, we propose to additionally condition the model components on thepast observations x <t and latent variables z <t and
Revised: To model temporal dependencies, we propose to additionally condition the model components on thepast observations x <t and latent variables z <t and

Original: y <t (cmp.
Revised: y <t (cmp.

Original: The joint distribution factorizesinto
Revised: The joint distribution factorizesinto

Original: [Equation]
Revised: [Equation]

Original: with generating distribution p θ ( x t j x <t , z ≤ t , y ≤ t ) and prior distributions p θ ( z t j x <t , z <t , y <t ) andp θ ( y t j x <t , z <t , y <t ) for latents z t and˜ y t .
Revised: with generating distribution p θ ( x t | x <t , z ≤ t , y ≤ t ) and prior distributions p θ ( z t | x <t , z <t , y <t ) andp θ ( y t | x <t , z <t , y <t ) for latents z t and y t .

Original: Note that we differentiate between observed labels yand latent variables ˜ y .
Revised: Note that we differentiate between observed labels y andlatent variables ˜ y to derive the variational distributions.

Original: Also note that in our sequential setting, the priors must be included in the optimization rather thanbeing represented by ﬁxed distributions to capture temporal dependencies between subsequent latentrealizations.
Revised: Also note that in our sequential setting, the priors are included in the optimization rather than beingrepresented by ﬁxed distributions to account for latent dynamics and effective sampling at inferencetime.

Original: This enables the prior samples to provide valuable information on the agents’ futuremovements at inference time.

Original: Hence, the surrogate posteriors q ϕ also need to condition on previousobservations and random variables to allow for accurate approximations of the true posterior andprovide a tractable lower bound on the log-likelihood in Eqn (1).
Revised: Hence, the surrogate posteriors q φ also need to condition on previous observations and randomvariables to allow for accurate approximations of the true posterior and provide a tractable lowerbound on the log-likelihood in Eqn (1).

Original: Using the factorization introducedin Section 3.1 yields
Revised: Using the factorization introduced in Section 3.1 yields

Original: [Equation]
Revised: [Equation]

Original: Labeled instances render reasoning over latents ˜ y ≤ T unnecessary since the true y ≤ T is alreadyknown.
Revised: Labeled instances render reasoning over latents ˜ y ≤ T unnecessary since the true y ≤ T is alreadyknown.

Original: Hence, the right-hand side in Eqn (2) can be ignored when sampling from D L .
Revised: Hence, the right-hand side in Eqn (2) can be ignored when sampling from D L .

Original: We arrive atthe following results and refer to Appendix A for the proofs.
Revised: We arrive atthe following results and refer to Appendix A for the proofs.

Original: Theorem 1.
Revised: Theorem 1.

Original: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by
Revised: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by

Original: [Equation]
Revised: [Equation]

Original: Correspondingly, for unlabeled instances drawn from D U , sequential label information need to beestimated as shown in the following theorem.
Revised: Correspondingly, for unlabeled instances drawn from D U , sequential label information need to beestimated as shown in the following theorem.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let H ( β ) be the entropy of quantity β .
Revised: Let H ( β ) be the entropy of quantity β .

Original: A lower bound on log p θ ( x ≤ T ) in Eqn (1) isgiven by
Revised: A lower bound on log p θ ( x ≤ T ) in Eqn (1) isgiven by

Original: [Equation]
Revised: [Equation]

Original: The full evidence lower bound (ELBO) is obtained by combining the lower bounds for all data D ,
Revised: The full evidence lower bound (ELBO) is obtained by combining the lower bounds for all data D ,

Original: [Equation]
Revised: [Equation]

Original:  This formalization focuses on either fully labeled or unlabeled observations, however, a more generalformulation can be obtained to also allowing for partially labeled observations.
Revised: U T (cid:88) t = −U ( x t ) . This formalization focuses on either fully labeled or unlabeled observations, however, a more generalformulation can be obtained to also allow for partially labeled observations.

Original: 3.4 Semi-supervised Sequential Learning
Revised: 3.4 Semi-supervised Sequential Learning

Original: Similar toinferring label sequences for unlabeled examples, our contribution leverages to sequenceannotation tasks where posterior q ϕ ( y ≤ T j x ≤ T ) acts as a classiﬁcation network annotating new observations.
Revised: Intrinsic to our contribution is an encoding module that argues over the label space, q φ ( y ≤ T | x ≤ T ) ,and hence can be used as a classiﬁcation network annotating new observations.

Original: However, the ELBO in Eqn (5) is oblivious to classiﬁcation and simply ignoresrelevant gradient updates for L ( x t , y t ) by discarding Eqn (2).
Revised: However, the ELBOin Eqn (5) is oblivious to classiﬁcation and simply ignores relevant gradient updates for L ( x t , y t )by discarding Eqn (2).

Original: This is clearly inappropriate forsemi-supervised learning where the overall goal is to learn an unknown mapping x7!
Revised: This is clearly inappropriate for semi-supervised learning where the overallgoal is to learn an unknown mapping x

Original:  y .Following Kingma et al.
Revised: (cid:55)→ y .

Original:  [27], we circumvent this by heuristically incorporating Eqn (2) into the labeldependent objective.
Revised: Following [27], we circumvent this by heuristicallyincorporating Eqn (2) into the label-dependent objective.

Original: Hence, the full objective to learn f θ, ϕ g is given by
Revised: Hence, the full training criterion to learn{ θ, φ } is given by

Original: [Equation]
Revised: [Equation]

Original: where λ 1 balances generative and discriminative learning and λ 

Revised: where λ 1 balances generative and discriminative learning and λ

Original: the contribution of labeled and unlabeled data, respectively.
Revised: the contribution of labeled andunlabeled data, respectively.

Original: Thus, the semi-supervised variant contains fully (un)supervised modelingtasks as special cases via adjusting λ 1 , λ 2 accordingly.
Revised: Thus, the semi-supervised variant contains fully (un)supervised modelingtasks as special cases via adjusting λ 1 , λ 2 accordingly.

Original: Note that the supervised target differs fromits unsupervised counterpart only in observing y t instead of factorizing over Y , which results incomputing the log-likelihood (auxiliary loss) instead of the entropy H ( q ϕ ( y t j x ≤ t , z <t , y <t )) .
Revised: The resulting supervised target differs fromits unsupervised counterpart only in observing y t instead of factorizing over Y , which results incomputing the log-likelihood (auxiliary loss) instead of the entropy H ( q φ ( y t | x ≤ t , z <t , y <t )) .

Original: Eqn (4) encodes the behavior of classiﬁer q ϕ ( y ≤ T j x ≤ T ) when dealing with unsupervised data.
Revised: Eqn (4) encodes the behavior of classiﬁer q φ ( y ≤ T | x ≤ T ) when dealing with unsupervised data.

Original: Since log p θ ( x t j x <t , , z ≤ t , y ≤ t ) is a main contributing factor in L ( x t , y t ) , optimizing for ϕ encourages the model to choose a high probability mass for  label  y 2 Y that incurs the smallestreconstruction loss compared to other candidates in label space.
Revised: Sincedata likelihood log p θ ( x t | x <t , , z ≤ t , y ≤ t ) tends to comprise the largest factor in L ( x t , y t ) , takingthe expectation wrt. q φ (˜ y t | x ≤ t , z <t , ˜ y <t ) encourages the model to realize the highest probabilitymass for the latent label valuel y ∈ Y that incurs the smallest reconstruction loss compared to othercandidates in label space.

Original: This provides an effective way tolearn from unobserved data.
Revised: This is a desirable property that reﬂects our assumption of data generation.

Original: 3.5 Design Choices
Revised: 3.5 Design Choices

Original: In Section 3.2, we argued that not all agents need to be present in every example and we simplyadded agents to training instances in arbitrary and possibly random order.
Revised: As described in Section 3.2, we aim to learn a distribution over possible sequences of agent sets thatmay differ in share of discrete annotations (and cardinality).

Original: However,order-agnosticmethods rely on the  independence of  agent trajectories, which is clearly inappropriate for multiagent scenarios where future movements need to be coordinated with other agents. We thus requirethe generative model to be in- or equivariant wrt. agent permutations and employ deep graph-neuralnetworks (GNNs) [49] as a methodological remedy to account for both, equivariance and agentinteractions, respectively. We process every agent’s trajectory x ( a ) ≤ T individually and assume an underlying connecting graph
Revised: However, the previously derived objectivefunctions only account for temporal dependencies, so that independence of the agent dimensionis required to realize permutation-invariant models.

Original: G = ( A , E ) at each timestep, where edges in E encode direct agent dependencies. Since agent-levelpredictions mainly rely on (spatially) nearby agents, we neglect direct inﬂuences from distant peers;however, this choice may be problem dependent. Thus, while the set of agents A remains constantacross the temporal dimension, the composition of edges E changes dynamically, depending on thek -nearest agents x ( a ) t .
Revised: This assumption is trivially inappropriate forinteractive systems where future agent movements need to be coordinated with other agents.

Original: Note that eradicating remote agents in the input graph does not curtail their indirect inﬂuence on anagent of interest. By adding depth to the network through stacking up GNN layers, the receptiveﬁeld is increased and thus also the number of nodes that affect the computations. Skip-like connections proposed by Xu et al. [56] aggregate the ﬁnal feature representation over all agent embeddingsof previous layers.
Revised: Thus, itis common to propose some form of graph encoding strategy [50] to account for both, equivarianceand agent interactions, respectively.

Original: This mechanism adds to the expressiveness of the network as it allows to adaptively accommodate structural information from different levels of granularity for each agent.
Revised: Intuitively, optimal interaction modeling allows to adaptively accomodate structural information fromdifferent levels of granularity for each agent.

Original: Inteam sports, for example, agent nodes are able to capture immediate player inﬂuences as well asholistic team strategies/formations dependent on the interaction structure best suited at predictiontime.
Revised: In team sports, for example, agent nodes should be ableto capture immediate player inﬂuences as well as holistic team strategies dependent on the interactionstructure best suited for the task at-hand. Motivated by the previous considerations, we costumize an

Original: To further narrow down the training process towards important pieces of data, every GNNlayer uses GAT update equations [54], implicitly assigning different weights to different nodes in aneighborhood.
Revised: GNN architecture that aims to mimic the descired behavior. Speciﬁcally, we stack attention-based GNN layers [55] with particularly designed skip-connections [57] and construct the assumed graphstructure using the k (spatially) closest agents.

Original: There are various possibilities to incorporate previously described GNNs in the overall scheme 1 .
Revised: There are various possibilities to incorporate graph-based approaches into the overall scheme 1 .

Original: Anefﬁcient way to model the distributions deﬁning Eqn (6) is to employ hidden  statesfor agents h ( a ) tthat encapsulate the past realizations of all random variables as node representations,
Revised: Anefﬁcient way to model the distributions deﬁning Eqn (6) is via hidden agent states h ( a ) t conditionedon representations o ( a ) t − 1 ,

Original: [Equation]
Revised: [Equation]

Original:  where f rnn denotes an RNN transition function, GNN is the described GAT-based GNN, the φ ’sare nonlinear function approximators, and o t = f o ( a ) t 
Revised: , where f rnn denotes an RNN transition function, GNN is the described GAT-based GNN and o t ={ o ( a ) t | a ∈ A} refers to the set of updated agent representations.

Original: j a 2 Ag refers to the set of updated agentrepresentations. Since the elements of o t additionally encode neighboring information, the resulting individual feature vectors can be used to describe the entire interactive past.
Revised: Since the elements in set o taggregate neighboring information of the RNN outputs h t via graph networks, the inferred featurevectors describe the entire interactive past of individual agents.

Original: While it is reasonable to enforce a dependency structure on the latent variables at time t via encoding and decoding GNN modules, we argue that capturing past agent interactions is sufﬁciently informative for highfrequency data [45].
Revised: While it is reasonable to additionallyenforce intra-timestep dependencies on the latent variables using encoding and decoding GNNmodules, we argue that capturing past spatiotemporal patterns is sufﬁciently informative for highfrequency data [46].

Original: Hence, assuming conditional independence within a single timestep, the jointmovement distribution factorizes across  agent dimensions and is given by
Revised: Hence, assuming conditional independence within t , the joint movement distribution factorizes across the agent dimension and is given by

Original: [Equation]
Revised: [Equation]

Original: where ψ t = f ψ ( a ) t j a 2 Ag are the distribution parameters where ψ ( a ) t = φ ( z
Revised: a ∈A p [Equation] x

Original: [Equation]
Revised: [Equation]

Original: a)
Revised: a

Original: [Equation]
Revised: [Equation]

Revised: where ψ t = { ψ ( a ) t | a ∈ A} are the distribution parameters and ψ ( a ) t = ϕ

Original:  , y ( a ) t , o ( a ) t − 1 ) .
Revised: ( z ( a ) t , y ( a ) t , o ( a ) t − 1 ) .

Original: Theoverall logic  is schematically depicted in Fig.
Revised: Thecomputational logic of the full architecture is schematically depicted in Fig.

Original:  Empirical Evaluation
Revised:  Empirical Evaluation

Original: For evaluation, we focus on team sports as the coordination of players renders these tasks moredifﬁcult than other domains [35].
Revised: For evaluation, we focus on team sports as the coordination of players renders these tasks moredifﬁcult than other domains [36].

Original: Hence, we experiment on STATS SportVU NBA 3 for comparison,and tracking data from elite soccer 4 .
Revised: Hence, we experiment on STATS SportVU NBA 3 for comparison,and tracking data from elite soccer 4 .

Original: As detailed in Section 3.2, we use agent velocities as input toall models and assume linear motion between consecutive observations.
Revised: As detailed in Section 3.2, we use agent velocities as inputto all models and assume linear motion between consecutive observations.

Original: For NBA, we adopt theexperimental setup and processing strategy from Monti et al.
Revised: For NBA, we adopt theexperimental setup and processing strategy from [40] 5 .

Original: The STATS SportVU NBA data comprises tracking positions of offensive plays from the 2016 NBAregular season covering more than 1200 different games where game segment are given by sequencesof length 50 and contain two-dimensional positions of all agents ( 10 players and ball) sampled at 5frames per second.
Revised: The STATS SportVU NBA data comprises tracking positions of offensive plays from the 2016 NBAregular season covering more than 1200 different games where game segment are given by sequencesof length 50 and contain two-dimensional positions of all agents ( 10 players and ball) sampled at 5frames per second.

Original: The data is split into 60% training, 20% validation, and 20% test sets.
Revised: The data is split into 60% training, 20% validation, and 20% test sets.

Original: All data istranslated so that the origin of the underlying coordinate system is mapped onto the top-left corner.
Revised: All data istranslated so that the origin of the underlying coordinate system is mapped onto the top-left corner.

Original: The soccer data consists of 12 matches where positions of players and ball are sampled at 25 framesper second.
Revised: The soccer data consists of 12 matches where positions of players and ball are sampled at 25 framesper second.

Original: The tracking data is accompanied by manual event annotations that we will also makeuse of in the remainder.
Revised: The tracking data is accompanied by manual event annotations that we will also make useof in the remainder.

Original: Models are trained on eight matches, the remaining six are distributedevenly into validation and test data (three matches each).
Revised: Models are trained on eight matches, the remaining four are distributed evenlyinto validation and test data (two matches each).

Original: 4.1 Baselines
Revised: 4.1 Baselines

Original: We evaluate versus several baselines on two distinct tasks: trajectory forecasting (Section 4.2and4.4) and  spatiotemporal classiﬁcation (Section 4.3 and 4.4).
Revised: We evaluate versus several baselines on two distinct tasks: trajectory forecasting (Section 4.2 and 4.4)and spatiotemporal classiﬁcation (Section 4.3 and 4.4).

Original: Since our approach constitutes theﬁrst semi-supervised model for label predictions in multi-agent scenarios, we need to rely on selfconstructed supervised baselines for scenarios where only a few labels are available, which wediscuss in more detail in the relevant sections.
Revised: Since our approach constitutes the ﬁrst semisupervised model for label predictions in multi-agent scenarios, we need to rely on self-constructedsupervised baselines for scenarios where only a few labels are available, which we discuss in moredetail in the relevant sections.

Original: By contrast, trajectory forecasting is an active topic in different ﬁelds (see, e.g., [44] for an overview),which allows comparisons with a wide range of baseline models.
Revised: By contrast, trajectory forecasting is an active topic in different ﬁelds (see, e.g., [45] for an overview),which allows comparisons with a wide range of baseline models.

Original: Fully supervised generative modelssuch as DAG-Net [39] and Weak-Sup [62] use weak labels in the form of agent objectives to improvetrajectory prediction.
Revised: Fully supervised generativemodels such as DAG-Net [40] and Weak-Sup [64] use weak labels in the form of agent objectives(heuristically inferred prior to model training) to improve trajectory prediction.

Original: In these settings, an agent’s objective at each timestep t is to reach the next areawhere the observed movement speed falls below a predeﬁned threshold.
Revised: In these settings, anagent’s objective at each timestep t is to reach the next area where the observed movement speedfalls below a predeﬁned threshold.

Original: Adopting the experimentalframework of Monti et al. [39] allows us to make further comparisons with Social-Ways [1] and STGAT [21] without additional experiments.
Revised: Adopting the experimental framework of [40] allows us to makefurther comparisons with Social-Ways [1] and STGAT [21] without additional experiments.

Original: STGAT [21] augments a standard GAT with an LSTM[19] to capture both spatial and temporal dependencies of social interactions while Social-Ways uses Info-GAN to learn multimodal predictive distributions.
Revised: STGAT[21] augments a standard GAT with an LSTM [19] to capture both spatial and temporal dependenciesof social interactions while Social-Ways uses Info-GAN to learn multimodal predictive distributions.

Original: It is difﬁcult to determine the current state-of-the-art in generative baselines for sports trackingdata because there is no consistent setting in terms of forecasting horizon or data (see e.g., [32, 39]).
Revised: It is difﬁcult to determine the current state-of-the-art in generative baselines for sports tracking databecause there is no consistent setting in terms of forecasting horizon or data (see e.g., [33, 40]).

Revised: Whilesome baselines are restricted to making predictions for a predetermined horizon, our method learnsspatiotemporal representations autoregressively and thus can be used for the full range of predictionscenarios.

Original: Hence, we  additionally benchmark against the most recent and methodologically most relevant workdescribed below.
Revised: Hence, we conduct two sets of experiments over both short ( 10 timesteps) and long ( 40timesteps) and additionally benchmark against the most recent and methodologically most relevantwork described below.

Original: GVRNN [58] is a graph extension of the VRNN for explicitly modeling joint agentmovements for basketball and soccer data.
Revised: GVRNN [59] is a graph extension of the VRNN for explicitly modeling jointagent movements for basketball and soccer data.

Original: dNRI [16] dynamically extracts interaction types thatdetermine the parametrization of the decoder.
Revised: dNRI [16] dynamically extracts interaction typesthat determine the parametrization of the decoder.

Original: Finally, GRIN [32] recovers (static) interactions viaattention using a disentangled latent space.
Revised: Finally, GRIN [33] recovers (static) interactionsvia attention using a disentangled latent space.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4.2 Exploring Boundary Cases for Trajectory Prediction
Revised: 4.2 Exploring Boundary Cases for Trajectory Prediction

Original: Methods We study the purely (un)supervised generation of trajectories by maximizing the respective lower bounds on NBA.
Revised: Methods We study the purely (un)supervised generation of trajectories by maximizing the respective lower bounds on NBA.

Original: For the fully-supervised variant, the weak labels described in theprevious section can be naturally integrated into the overall scheme via treating them as semanticconcepts y ( a ) t .
Revised: For the fully-supervised variant, the weak labels described in Section4.1 can be naturally integrated into the overall scheme via treating them as semantic concepts y ( a ) t .

Original: These target areas naturally vary over time so that the sequential label y ≤ T denotesalways the next desired position for every involved agent at time t .
Revised: These target areas naturally vary over time so that the sequential label y ≤ T denotes always the nextdesired position for every involved agent at time t .

Original: Following related work, weobtain 90 different areas/class labels.
Revised: Following related work [40, 64], we obtain 90different areas/class labels.

Original: The underlying data distribution is estimated by maximizing {+Eqn (3) viasetting λ 
+}
Revised: The underlying data distribution is estimated by maximizing {+Eqn (3) viasetting λ 
+}

Original: Eqn (3) via setting λ 0 = λ 1 = 0 in Eqn (6).
Revised: = λ 1 = 0 in Eqn (6).

Original: The supervised variant of our contribution is denotedby S-MAT .
Revised: The supervised variant of our contribution is denoted by S-MAT .

Original: Although done in [39, 62], direct comparison of unsupervised and supervised generative modelsis inappropriate inasmuch as the latter use their (weakly) generated labels over the entire observation period (which include future information) at prediction time.
Revised: Although done in [40, 64], direct comparison of unsupervised and supervised generative modelsis inappropriate inasmuch as the latter use ground-truth labels over the entire observation period(which include future information) at prediction time.

Original: Thus, we additionally benchmarkagainst a fully unsupervised instantiation of our proposed framework lacking ground-truth information for structuring the discrete latent subspace.
Revised: Thus, we additionally benchmark against afully unsupervised instantiation of our proposed framework lacking ground-truth information forstructuring the discrete latent subspace.

Original: To encourage the model to learn semanticallymeaningful concepts describing distinct movement patterns, we parametrize the decoder dependenton the predicted agent class ˜ y
Revised: To encourage the model to learn semantically meaningfulconcepts describing distinct movement patterns, we parametrize the decoder dependent on thepredicted agent class ˜ y

Original: ( a ) t .
Revised: ( a ) t .

Original: In this context, ˜ y ( a ) t may be best thought of dynamically assignedagent roles that accurately explain the observed trajectories.
Revised: In this context, ˜ y ( a ) t may be best thought of dynamically assigned agentroles that accurately explain the observed trajectories.

Original: We maximize Eqn (4) using 3 agenttypes and refer to the unsupervised variant as U-MAT .
Revised: We maximize Eqn (4) using 3 agent typesand refer to the unsupervised variant as U-MAT .

Original: Metrics We measure the quality of the generated trajectories by their average L 2 distance betweenpredicted ˆ x ( a ) t and observed positions x ( a ) t over the entire sequence given by
Revised: Further details for S-MAT and U-MAT are given in

Original: [Equation]
Revised: Appendix C.

Original: and report the best result out of  20 generated samplesfor every instance. In addition, we give anaccount of the  ﬁnal L 2 distance which follows the same logic but evaluates only the difference ofthe ﬁnal timestep T in the test sequences. We report all metrics on un-normalized scales.
Revised: Metrics We measure the quality of the learned multimodal distribution using common standardmetrics: the minimum over 20 generated samples of the average and ﬁnal L 2 distance betweenpredicted ˆ x ( a ) ≤ T and observed positions x ( a ) ≤ T .

Original: Results Tables 1 and 2 show the results.
Revised: Results Tables 1 and 2 show the results.

Original: The proposed approaches clearly outperform their peersand realize the lowest average L 2 and ﬁnal L 2 distances.
Revised: The proposed approaches clearly outperform their peersand realize the lowest average L 2 and ﬁnal L 2 distances.

Original: The MAT variants effectively accommodate not only mutual inﬂuences across agents, but also discrete generative factors, leading to betterapproximations of the underlying multi-modal distributions.
Revised: The MAT variants effectively accommodatenot only mutual inﬂuences across agents, but also discrete generative factors, leading to betterapproximations of the underlying multi-modal distributions.

Original: Unsurprisingly, Table 1 also showsthat defending players exhibit more structure and are easier to predict than offensive players.
Revised: Unsurprisingly, Table 1 also shows thatdefending players exhibit more structure and are easier to predict than offensive players.

Original: Theresults of an ablation study are contained in Appendix D.
Revised: The resultsof an ablation study are contained in Appendix E.

Original: The right part of Fig.
Revised: The right part of Fig.

Original: 2 displays exemplaryrollouts.
Revised: 2 displays exemplary rolloutsthat underline models’ ability to capture highly complex changes in movement directions.

Original: 4.3 Semi-Supervised Classiﬁcation
Revised: 4.3 Semi-Supervised Classiﬁcation

Original: We now extend the experimental protocol by targeting accurate label discovery in semi-supervisedscenarios.
Revised: We now extend the experimental protocol by targeting accurate label discovery in semi-supervisedscenarios.

Original: We refer to the full model simply as MAT where λ 0 , λ 1 > 0 and incorporate the fullysupervised S-MAT ( λ 0 = 0 ) as an additional baseline.
Revised: We refer to the full model simply as MAT where λ 0 , λ 1 > 0 and incorporate the fully supervised S-MAT ( λ 0 = 0 ) as an additional baseline.

Original: We also compare to an RNN-based classiﬁcation network that addresses inter-agent dependencies by inferring hidden states of agents by the GNN architecture described in Section 3.5.
Revised: We also compare to an RNN-based classiﬁcationnetwork that addresses inter-agent dependencies by inferring hidden states of agents by the GNN architecture described in Section 3.5.

Original: A ﬁnal softmax layer is used for classiﬁcation and themodel is optimized by minimizing the negative log-likelihood.
Revised: A ﬁnal softmax layer is used for classiﬁcation and the model isoptimized by minimizing the negative log-likelihood.

Original: We refer to this baseline as MA-RNN .
Revised: We refer to this baseline as MA-RNN .

Original: The semi-supervised MAT is trained on  50% labeled and 50% unlabeled examples while the supervised baselines haveonly access  to the labeled part.Figure 3 shows the performance of the threemodels on NBA.
Revised: Figure 3 shows the performance of the models on NBA when using 50% labeled and 50% unlabeledexamples (left side) and when using varying portions of labeled data (right side), while the supervisedbaselines have access only to the labeled part.

Original: The semi-supervised MAT clearly emerges as the strongest classiﬁer and distinguishes by the lowestlong-term generalization error.
Revised: The semi-supervised MAT clearly emerges as thestrongest classiﬁer as it converges to the lowest generalization error during a training run and theperformance beneﬁt increases via reduction of labeled data.

Original: Comparing the numbers to its supervised peer S-MAT  highlights the value of the unlabeled data that is effectively utilized by MAT .
Revised: Comparing the numbers to its supervisedpeer ( S-MAT ) highlights the value of incorporating unlabeled data that is effectively utilized by MATto inform classiﬁcation decisions.

Original: Comparingthe fully supervised approaches S-MAT and MA-RNN , the importance of an effective regularizationmechanism becomes obvious; MA-RNN quickly overﬁts while the lower-bound in S-MAT acts as aregularizer and guarantees gradually increasing predictive accuracies on validation sets.
Revised: Comparing the fully supervised approaches S-MAT and MA-RNN ,the importance of an effective regularization mechanism becomes obvious; MA-RNN quickly overﬁtswhile the lower-bound in S-MAT acts as a regularizer and guarantees gradually increasing predictiveaccuracies on validation sets.

Original: 4.4 Combining Generation and Classiﬁcation in Semi-Supervised Settings
Revised: 4.4 Combining Generation and Classiﬁcation in Semi-Supervised Settings

Original: In our last set of experiments, we study the combination of trajectory generation and classiﬁcationand resort to tracking data from elite soccer.
Revised: In our last set of experiments, we study the combination of trajectory generation and classiﬁcationand resort to tracking data from elite soccer.

Original: The reason for favoring this data over NBA is thatthe soccer tracking data is manually annotated by experts from the data provider.
Revised: The reason for favoring this data over NBA is thatthe soccer tracking data is manually annotated by experts from the data provider.

Original: Since the datarequires human annotations for further processing, a successful semi-supervised application bears [Table] a great deal of practical relevance by reducing the expensive labeling effort.
Revised: Since the datarequires human annotations for further processing, a successful semi-supervised application bearsa great deal of practical relevance by reducing the expensive labeling effort.

Original: We focus on labels288 Y = f pass , other ball action , shot , none g , where class none denotes the absence of all other labelsin a frame.
Revised: We focus on labels Y = { pass , other ball action , shot , none } , where class none denotes the absence of all other labels ina frame and use 20% annotated data.

Original: Every label is propagated to the previous ﬁve and the subsequent 30 frames.
Revised: Every label is propagated to the previous ﬁve and the subsequent30 frames.

Original: We generatea balanced training set where half of the segments carry label none and study the combined task oftrajectory generation and classiﬁcation.
Revised: We generate a balanced training set where half of the segments carry label none and studythe combined task of trajectory generation and classiﬁcation.

Revised: This set of experiments mainly serves the purpose to quantitatively validate the beneﬁts of incorporating expensive human annotations as a generation guide or for structuring the discrete posterior.

Revised: This

Revised: [Table]

Revised: approach comes with little additional costs since the method operates semi-supervised and thereforerequires only a small subset of multi-agent segments to be annotated.

Revised: Given the higher sampling rateand less involved plays in soccer dynamics, we also intend to evaluate the extent to which interactionmodules achieve empirical beneﬁts.

Original:  Generation For trajectory prediction, we compare MAT against VRNN [7] that implicitly assumesindependence across agent dimensions, an interactive version of thereof with GNNs on the hiddenstates ( VRNN+GNN ), the supervised S-MAT with long-term goals, and a diagonal  MAT-Diag .
Revised: We design our baselines accordinly. Generation For trajectory prediction, we compare MAT to VRNN [7] that implicitly assumesindependence across agent dimensions, an interactive version of thereof with GNNs on the hiddenstates ( VRNN+GNN ), GVRNN [59], S-MAT with long-term goals (introduced in Section 4.2, anda diagonal MAT version, MAT-Diag .

Original: Notethat the VRNN constitutes a strong competitor for the task at-hand, as it effectively corresponds to adiagonal GVRNN, which has been shown to produce competitive results on soccer data [58].
Revised: Note that the VRNN constitutes a strong competitor for thetask at-hand, as it effectively corresponds to a diagonal GVRNN, which has been shown to producecompetitive results on soccer data [59].

Original: The left part of Table 3 shows the results.
Revised: The left part of Table 3 shows the results.

Original: Although all competitors realize similar performances,the semi-supervised MAT generates trajectories that are closest in both metrics to the observed ones.
Revised: Although the results are generally more similar comparedto the basketball experiments, the semi-supervised MAT generates trajectories that are closest in bothmetrics to the observed ones.

Original: MAT effectively exploits the additional expert annotations and performs signiﬁcantly better than its(weakly) supervised peers.
Revised: We hypothesize that decreasing the frequency of the data and extractingmulti-agent segments that merely consist of highly interactive plays yields an signiﬁcant increase inperformance gap.

Original: Classiﬁcation MAT learns a model over the label space Y simultaneously to the generative model.
Revised: Classiﬁcation MAT learns a model over the label space Y simultaneously to the generative model.

Original: To evaluate classiﬁcation performance, we turn the output of the corresponding softmax for a givenframe into a class label whenever it exceeded a predeﬁned threshold.
Revised: To evaluate classiﬁcation performance, we turn the output of the corresponding softmax for a givenframe into a class label whenever it exceeded a predeﬁned threshold.

Original: We compare the predictionperformance again to MA-RNN , a diagonal variant MA-RNN-Diag , and S-MAT .
Revised: We compare the predictionperformance again to MA-RNN , a diagonal variant MA-RNN-Diag , and S-MAT .

Original: The resulting accuracies and F1 scores are summarized in Table 3.
Revised: The resulting accuracies and F1 scores are summarized in Table 3.

Original: Once more, MAT clearly beats thebaselines and stands out by the highest accuracy and F1 scores.
Revised: Once more, MAT clearly beats thebaselines and stands out by the highest accuracy and F1 scores.

Original: The result impressively demonstratesthe beneﬁt of including unlabeled data for the task.
Revised: The result impressively demonstratesthe beneﬁt of including unlabeled data for the task.

Original: Surprisingly, MA-RNN performs better than SMAT .
Revised: Surprisingly, MA-RNN performs better than S-MAT .

Original: This ﬁnding is in contrast to the results on NBA shown in Figure 3.
Revised: This ﬁnding is in contrast to the results on NBA shown in Figure 3.

Original: However, soccer is playedon a much larger pitch and movements are likely more linear compared to basketball, which leads toless multimodality of the distributions involved and a consequently decreased beneﬁt of variationalmethods.
Revised: However, soccer isplayed on a much larger pitch and movements are likely more linear compared to basketball, whichleads to less multimodality of the distributions involved and a consequently decreased beneﬁt ofvariational methods.

Original: Together with a signiﬁcantly reduced label space and sufﬁcient amounts of data this maylead to a simpler learning task where MA-RNN is less prone to overﬁtting.
Revised: Together with a signiﬁcantly reduced label space and sufﬁcient amounts of datathis may lead to a simpler learning task where MA-RNN is less prone to overﬁtting.

Original: The right part of Fig.
Revised: The left part of Fig.

Original: 2 shows exemplary prediction probabilities for two segments and three possibleevents.
Revised: 2 shows exemplary prediction probabilities for two segments and three possibleevents.

Original: In both segments, the algorithm is highly conﬁdent no shot-action will occur in the nearfuture.
Revised: In both segments, the algorithm is highly conﬁdent no shot-action will occur in the near future.

Original: The segment on top contains an event of the class other ball action followed by an event ofclass pass .
Revised: The segment on top contains an event of the class other ball action followed by an event of class pass .

Original: Both are correctly identiﬁed by MAT .
Revised: Both are correctly identiﬁed by MAT .

Original: The former is clearly indicated by a peak in thecorresponding probability chart which then decreases to give rise to the following pass action.
Revised: The former is clearly indicated by a peak in the correspondingprobability chart which then decreases to give rise to the following pass action.

Original: Thesegment on the bottom shows solely a single pass event that is also clearly identiﬁable by a peak atthe correct frame.
Revised: The segment on thebottom shows solely a single pass event that is also clearly identiﬁable by a peak at the correct frame.

Original:  Conclusion
Revised:  Conclusion

Original: We presented semi-supervised variational autoencoders for spatiotemporal multi-agent scenarios.
Revised: We presented semi-supervised variational autoencoders for spatiotemporal multi-agent scenarios.

Original: The proposed approach effectively combined ideas from semi-supervised variational autoencoders,variational recurrent neural networks, and graph neural networks.
Revised: The proposed approach effectively combined ideas from semi-supervised variational autoencoders,variational recurrent neural networks, and graph neural networks.

Original: Empirically, our approach clearlyoutperformed the state-of-the-art in sequential generation and (semi-supervised) classiﬁcation tasksin all experiments.
Revised: Empirically, our approach clearlyoutperformed the state-of-the-art in sequential generation and (semi-supervised) classiﬁcation tasks inall experiments.

Original: The performance underlines the beneﬁt of including unlabeled data in spatiotemporal problems where labeled sequences are either scarce or assembled from weak makeshift signals.
Revised: The performance underlines the beneﬁt of including unlabeled data in spatiotemporalproblems where labeled sequences are either scarce or assembled from weak makeshift signals.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] Since this work constitutes theﬁrst semi-supervised approach for spatiotemporal multi-agent problems, it is difﬁcultto discuss techical limitations at this stage.
Revised: [Yes] Since this work constitutes the ﬁrst semi-supervised approach for spatiotemporal multi-agent problems, it is difﬁcult to discuss techical limitations at this stage.

Original: Application-wise, there are clear limitationse
Revised: Application-wise, there are clear limitations

Original: .g. , by using weak makeshift labels.
Revised: e.g., by using weak makeshift labels.

Original: We discuss these issues across the paper (e
Revised: We discuss these issues across the paper (e

Original: .g.
Revised: g.,

Original: Section
Revised: Section

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We have
Revised: [No] We have

Original: a clear focus on team sports.
Revised: a clear focus on team sports.

Original: Though dual use of the proposed technique is certainly
Revised: Though dual use of the proposed technique is certainly

Original: possible (e.g., military domains), it may be a bit far fetched as we have no expertise 
Revised: possible (e.g., military domains), it may be a bit far fetched as we have no expertise in

Original: in these domains and would only be able to deliver uneducated guesses about whether
Revised: these domains and would only be able to deliver uneducated guesses about whether

Original: this is realistic or not.
Revised: this is realistic or not.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section
Revised: [Yes] See Section4.1.

Original: 4.1. – 4.3.(b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] All proofs arecon-
Revised: [Yes] All proofs are

Original: tained in the Appendix.
Revised: contained in the Appendix.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] The NBA data
Revised: [Yes] The NBA data

Original: set is publicly available and we provide a URL in the paper.
Revised: set is publicly available and we provide a URL in the paper.

Original: The soccer data is private
Revised: The soccer data is private

Original: and must not be shared with third parties.
Revised: and must not be shared with third parties.

Original: We will release the code and include a URL
Revised: We will release the code and include a URL

Original: in the paper as well after the paper has been accepted.
Revised: in the paper as well after the paper has been accepted.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section4, more details are provided in the Appendixl.(c)
Revised: [Yes] See Section more details are provided in the Appendixl.

Original:  Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [No]
Revised: [No]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix4.
Revised: [Yes] See Appendix4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Revised: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Revised: [Table]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See supplemental material
Revised: [Yes] See supplemental material

Original:  (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Table] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data 
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: you’re using/curating?
Revised: [Table] using/curating?

Original: [Yes] The NBA data is publicly available and free, for soc-
Revised: [Yes] The NBA data is publicly available and free, for soccer we have

Original: [Table]
Revised: [Table]

Original: cer we have the permission of the data provider.
Revised: the permission of the data provider.

Original:  (e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
Revised: [Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable [Table]

Original: able information or offensive content?
Revised: information or offensive content?

Original: [Yes] The soccer and basketball players are 
Revised: [Yes] The soccer and basketball players are probably

Original: [Table]
Revised: [Table]

Original: probably identiﬁable in the data, but they/their contract/their clubs agreed in recording 
Revised: identiﬁable in the data, but they/their contract/their clubs agreed in recording the data.

Original: the data.
Revised: [Table]

Original: The soccer data is private and cannot be shared. 
Revised: The soccer data is private and cannot be shared. [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]
Revised: [Table]

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: A Proofs of the Theorems
Revised: A Proofs of the Theorems

Original: We begin with Theorem 2 for simplicity and sketch the proof of Theorem 1 afterwards.
Revised: We begin with Theorem 2 for simplicity and sketch the proof of Theorem 1 afterwards.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let H ( β ) be the entropy of quantity β .
Revised: Let H ( β ) be the entropy of quantity β .

Original: A lower bound on log p θ ( x ≤ T ) in Eqn (1) isgiven by
Revised: A lower bound on log p θ ( x ≤ T ) in Eqn (1) isgiven by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For unlabeled training instances, deﬁning the marginal likelihood according to the generativestructure introduced in Section 3.3 yields
Revised: For unlabeled training instances, deﬁning the marginal likelihood according to the generativestructure introduced in Section 3.3 yields

Original: [Equation]
Revised: [Equation]

Original: To derive the lower bound on the log likelihood
Revised: To derive the lower bound on the log likelihood

Original: [Equation]
Revised: [Equation]

Original: we incorporate the variational information into theabove deﬁnition and apply Jensen’s inequality:[Equation] = [Equation]
Revised: we incorporate the variational information into theabove deﬁnition and apply Jensen’s inequality:

Original: =

Original: [Equation]

Original: ∑

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [ q

Original: [Equation]

Original: ( y

Original: [Equation]

Original: y

Original: [Equation]
Revised: [Equation]

Original: p

Original: [Equation]

Original:  y t j x
Revised: , y <t

Original: [Equation]
Revised: [Equation]

Original: z

Original: [Equation]

Original: y <t )] ] .

Original: Thus, we can writelog
Revised: Thus, we can write

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: <

Original: [Equation]

Original: [Equation]

Original: which is identical to Eqn 3.
Revised: which is identical to Eqn 3.

Original: Theorem 1.
Revised: Theorem 1.

Original: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by 
Revised: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by [Equation] [Equation] ≤ T

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: B Implementation Details
Revised: B Implementation Details

Original: All models were implemented using
Revised: All models are implemented using

Original: PyTorch [41].
Revised: PyTorch [42].

Original: For our experiments wedeﬁne the model components as follows.
Revised: For our experiments, wedeﬁne the model components as follows.

Original: The encoders are modeled asq ϕ (˜ y ( a ) j x ≤ t , z <t , y <t ) =
Revised: The encoders are modeled asq φ (˜ y ( a ) | x ≤ t , z <t , y <t ) =

Original: Cat ( y ( a ) t j f yenc ([ o ( a ) t − 1 , φ x ( x ( a ) t )]) and q ϕ ( z ( a ) t j x ≤ t , z <t , y ≤ t ) =
Revised: Cat ( y ( a ) t | f y enc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t )]) and q φ ( z ( a ) t | x ≤ t , z <t , y ≤ t ) =

Original: N ( z ( a ) t j f z enc ([ o ( a ) t − 1 , φ x ( x ( a ) t ) , φ y ( y ( a ) t )]) for uncovering discrete and continuous latent information, respectively.
Revised: N ( z ( a ) t | f z enc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t ) , ϕ y ( y ( a ) t )]) for uncovering discrete and continuous latent information,respectively.

Original: The prior distributions are computed analogously omitting input x t .
Revised: The prior distributions are computed analogously omitting input x t .

Original: Agentmovements are represented as p θ ( x ( a ) t j x <t , z ≤ t , y ≤ t ) =
Revised: Agent movementsare represented as p θ ( x ( a ) t | x <t , z ≤ t , y ≤ t ) =

Original: N ( z ( a ) t j f dec ([ o ( a ) t − 1 , φ x ( x ( a ) t ) , φ y ( y ( a ) t )]) .
Revised: N ( z ( a ) t | f dec ([ o ( a ) t − 1 , ϕ x ( x ( a ) t ) , ϕ y ( y ( a ) t )]) .

Original: All functions f and feature extractors φ are two-layer MLPs with LeakyRelu [55] activations.
Revised: All functions f and feature extractors ϕ are two-layer MLPs with LeakyRelu [56] activations.

Original: Weuse 2 -layer GRU networks [6] for recurrence, and GNNs with 3 GAT layers implemented according according to [54].
Revised: We use2 -layer GRU networks [6] for recurrence. Moreover, we deﬁne the graph network using 3 GAT layersimplemented according to [55] and the graph structure wrt.the k

Revised: = 2 and k = 5 spatially closestagents for basketball and soccer, respectively.

Original: We implement skip-connections as described in [56] and use concatenationfollowed by a linear transformation to aggregate intermediate layer embeddings to the GNN modeloutput.
Revised: We implement skip-connections as described in [57]and use concatenation followed by a linear transformation to aggregate intermediate layer embeddingsto the GNN model output.

Original: For generative tasks, we used λ 1 = 0 , however, including the auxiliary loss yields negligibledeterioration.
Revised: For generative tasks, we use λ 1 = 0 , however, including the auxiliary lossyields negligible deterioration.

Original: For classiﬁcation, we use λ 0 = 1 and α = 10 (cid:3) (1 / % of labeled data ) 
Revised: For classiﬁcation, we use λ 0 = 1 and α = 10 ∗ (1 / % of labeled data ) .

Original: The model operates solely on agent velocities (input and output).
Revised: The model operates solely on agent velocities (input and output).

Original: However, we inject position tuplesto the all model components via teacher forcing.
Revised: However, we inject position tuplesto the all model components via teacher forcing.

Original: We use the gumbel-softmax trick [34] [23] whensampling from categorical distributions.
Revised: We use the gumbel-softmax trick [35, 23] whensampling from categorical distributions.

Original: Training was executed on an Nvidia V100 GPU and tookabout 16 hours to complete 300 epochs while comsuming (cid:24) 15 Gb for basketball.
Revised: Training was executed on an Nvidia V100 GPU and tookabout 16 hours to complete 300 epochs while comsuming ∼ 15 Gb for basketball.

Original: All models wereoptimized using Adam [25] with a learning rate of 0 , 001 and gradient clipping using a max norm of10 .
Revised: All models areoptimized using Adam [25] with a learning rate of 0 , 001 and gradient clipping using a max norm of10 .

Revised: Baselines As stated in Section 4.1, we report against the values reported in [40].

Revised: For [16, 33, 37],we adapt the source code from their ofﬁcial repositories to our experimental setting 6 For GVRNN[59], we re-implemented the model according to the descriptions in their paper and designed theoverall architecture such that it is comparable in parameter number to our method.

Original: Soccer Application Since the labels denote ball-centric events, we use the output of the ball nodefor loss computation and evaluation.
Revised: Soccer application Since the labels denote ball-centric events, we use the output of the ball nodefor loss computation and evaluation.

Original: The F 1 score is computed as follows.
Revised: The F 1 score is computed as follows.

Original: We annotate a multiagent segment when the derived probability estimates exceed an externally deﬁned threshold value.
Revised: We annotate a multiagent segment when the derived probability estimates exceed an externally deﬁned threshold value.

Original: We obtain TP values (FP values) when the predicted event coincides (disagrees) with the groundtruth annotation.
Revised: We obtain TP values (FP values) when the predicted event coincides (disagrees) with the groundtruth annotation.

Original: FN values are deﬁned by anotated segments that remain undetected.
Revised: FN values are deﬁned by anotated segments that remain undetected.

Original: We compute F1-scores for 100 distinct threshold values in the range between 0 . 5 and 0 .
Revised: We compute F1-scores for 100 distinct threshold values in the range between 0 . 5 and 0 .

Original: 98 and only reportthe maximum F1-score.
Revised: 98 and only report themaximum F1-score.

Original: However, threshold optimization yields only negligible improvement oversimply using 0 .
Revised: However, threshold optimization yields only negligible improvement over simplyusing 0 .

Revised: C More Details on the MAT Models

Original: Baselines As stated in Section 4.1, we report against the values reported in [39].
Revised: In Section 3.1, we deﬁne y ≤ T as arbitrary discrete (possibly latent) behavioral indicators. To validatethis general formulation (and our proposed architecture), we vary the speciﬁc deﬁnition of the variable across experiments. Accordingly, we propose different framework instantiations that are described inmore detail in this section. S-MAT First, we observe in Section 4.2 that existing SOTA trajectory prediction approaches(Section 4.1) use heuristically generated labels for trajectory prediction that encode agents’ intentsor goals over a discretized position space.

Revised: We then note that our formalization allows to naturallyintegrate such long-term goals into the overall scheme via treating them as discrete semantic concepts.

Revised: Since these labels are generated heuristically based on the trajectory input prior to model training,this model instantiation is fully-supervised and is referred to as S-MAT .

Revised: We refer to [64] for moredetails on how to produce the weak labels used for training S-MAT.

Revised: U-MAT

Original: For [16, 32, 36],we adapt the source code from their ofﬁcial repositories to our experimental setting 6 For GVRNN[58], we re-implemented the model according to the descriptions in their paper and designed theoverall architecture such that it is comparable in parameter number to our method.
Revised: However, we found (although done in [64, 40]) that benchmarking against unsupervisedbaselines is inappropriate for the reasons described in Section 4.1. Since most SOTA approachesare unsupervised generative models, we additionally propose a fully-unsupervised instantiation ofour framework ( U-MAT ) employing no external guidance for discrete latent structuring. From thepredicted (latent) categorical distribution q φ , we sample a label value and exploit separate motionpredictors (different decoder parameterizations) p θ dependent on the realized value. Intuitively, thisencourages the model to learn categories describing fundamentally different movement patterns,which can be interpreted as dynamic “agent roles”.

Revised: Thus, the concept of agent roles here is merelyan intuitive explanation and realized via an inductive bias that increases the “scope” of latentinformation encoded without utilizing any supervision.

Revised: To the best of our knowledge, parameterizingthe generation module based on inferred agent categories is novel and could provide valuable insightsfor practitioners.

Original: C Experiments with Drone Data
Revised: D Experiments with Drone Data

Original:  To showcase applicability to scenarios that exhibt variable numbers of agents, we also report resultson the Stanford Drone Data (SDD) [43].
Revised: [Figure] [Table] To showcase applicability to scenarios that exhibt variable numbers of agents, we also report resultson the Stanford Drone Data (SDD) [44].

Original: SDD is a collection of videos recorded by drones at eightlocations at Stanford.
Revised: SDD is a collection of videos recorded by drones at eightlocations at Stanford.

Original: While pedestrians predominate as interacting agents, cyclists, skateboarders, [Table] cars, buses, and golf carts are also present.
Revised: While pedestrians predominate as interacting agents, cyclists, skateboarders,cars, buses, and golf carts are also present.

Original: We use the TrajNet benchmark [46] of the data, providingsequential two-dimensional real-world coordinates at a frame rate of 2.5 frames per second.
Revised: We use the TrajNet benchmark [47] of the data, providingsequential two-dimensional real-world coordinates at a frame rate of 2.5 frames per second.

Original: Wefollow Monti et al.

Original: [39] regarding the data processing strategy 7 .
Revised: Wefollow [40] regarding the data processing strategy 7 .

Original: We incorporate PECNet [36] forcomparison as it is considered state-of-the-art for SDD [35].
Revised: We incorporate PECNet [37] for comparison asit is considered state-of-the-art for SDD [36].

Original: PECNet models stochasticity in theﬁnal position of the pedestrians conditioned on the past motion history.
Revised: PECNet models stochasticity in the ﬁnal position of thepedestrians conditioned on the past motion history.

Original: [Table] To generate weak labels, we move a time window through each trajectory, with the respective end [Table] cells acting as agent targets y ( a ) t .
Revised: To generate weak labels, we move a time window through each trajectory, with the respective endcells acting as agent targets y ( a ) t .

Original: Static time windows allow us to use the fully-supervised variantfor all comparisons (including fully unsupervised models).
Revised: Static time windows allow us to use the fully-supervised variant forall comparisons (including fully unsupervised models).

Original: The scene at hand is thereby discretizedinto 960 areas and results are shown in Table 4.
Revised: The scene at hand is thereby discretized into960 areas and results are shown in Table 4.

Original: We observe the same general pattern as for thebasketball experiments.
Revised: We observe the same general pattern as for the basketballexperiments. [Table]

Original: D Ablation Study
Revised: E Ablation Study

Original: [Table]
Revised: [Table]

Original: [Table]

Original: Table 5 validates the proposed architecture by showing results of an ablation study.
Revised: Table 5 validates the proposed architecture by showing results of an ablation study.

Original: We test a fullyconnected graph ( S-MAT-Full ), an independent version with diagonal adjacency matrices ( S-MATDiag ), a variant that employs GNNs for variational, generative, and prior distribution parameters,but ignores interactive updates for the hidden states ( S-MAT-GRVNN ) similar to Yeh et al. [58],and a variant that is identical to the latter but additionally includes a GNN for the hidden state ( SMAT-GRVNN-Hidden ).
Revised: We test a fullyconnected graph ( S-MAT-Full ), an independent version with diagonal adjacency matrices ( S-MATDiag ), a variant that employs GNNs for variational, generative, and prior distribution parameters, butignores interactive updates for the hidden states ( S-MAT-GRVNN ) similar to [59], and a variant that isidentical to the latter but additionally includes a GNN for the hidden state ( S-MAT-GRVNN-Hidden ).

Original: The table provides supporting evidence for our design choices: though the GVRNNs are theoretically able to capture intra-timestep dependencies, we observe signiﬁcant dropsin performance compared to the other competitors.
Revised: The table provides supporting evidence for our design choices: though the GVRNNs are theoreticallyable to capture intra-timestep dependencies, we observe signiﬁcant drops in performance compared tothe other competitors.

Original: The S-MAT-Diag and MAT-Full experimentssuggest that our model captures interaction patterns among agents very well; S-MAT denotes avaluable contribution to the large body of research that explicitly addresses modeling multi-agentdata accurately.
Revised: The S-MAT-Diag and MAT-Full experiments suggest that our model capturesinteraction patterns among agents very well; S-MAT denotes a valuable contribution to the large bodyof research that explicitly addresses modeling multi-agent data accurately.