Original: Latent Variable Models for Multi-Agent Trajectories
Revised: Latent Variable Models for Multi-Agent Trajectories

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Analyzing the spatiotemporal behavior of multiple agents is of great interest tomany communities.
Revised: Analyzing the spatiotemporal behavior of multiple agents is of great interest tomany communities.

Original: Existing probabilistic models in this regime employ eitherunsupervised generative settings, in which the latent space is described fully bydiscrete or continuous representations, or, are alternatively formalized in a (fully)supervised framework where weakly preserved labels add explicit information to acontinuous latent representation learned from the data.
Revised: Existing probabilistic models in this regime employ eitherunsupervised generative settings, in which the latent space is described fully bydiscrete or continuous representations, or, are alternatively formalized in a (fully)supervised framework where weakly preserved labels add explicit information to acontinuous latent representation learned from the data.

Original: To overcome the resultinglimitations, we propose a novel objective function for processing multi-agent trajectories based on semi-supervised variational autoencoders, where equivariance andinteraction of agents are modeled via costumized graph networks.
Revised: To overcome the resultinglimitations, we propose a novel objective function for processing multi-agent trajectories based on semi-supervised variational autoencoders, where equivariance andinteraction of agents are modeled via costumized graph networks.

Original: Our formulationdisentangles discrete and continuous effects and allows discrete behavioral indicators in arbitrary quantity and annotation type to guide the generation process.
Revised: Our formulationdisentangles discrete and continuous effects and allows discrete behavioral indicators in arbitrary quantity and annotation type to guide the generation process.

Original: Thislifts applicability to relevant prediction problems beyond the generation of collective movements and provides an effective solution to incoporate expensive domainknowledge into interactive multi-agent systems.
Revised: Thislifts applicability to relevant prediction problems beyond the generation of collective movements and provides an effective solution to incoporate expensive domainknowledge into interactive multi-agent systems.

Original: Empirically, our model outperforms various state-of-the-art baselines in generating future agent movements oninteractive real-world datasets.
Revised: Empirically, our model outperforms various state-of-the-art baselines in generating future agent movements oninteractive real-world datasets.

Original: We also show that our approach effectively learns toleverage unsupervised multi-agent sequences to improve classiﬁcation of long-termlocations as well as manually annotated situations on sports tracking data.
Revised: We also show that our approach effectively learns toleverage unsupervised multi-agent sequences to improve classiﬁcation of long-termlocations as well as manually annotated situations on sports tracking data.

Original:  Introduction
Revised:  Introduction

Original: Analyzing the spatiotemporal behavior of multiple agents bears a great deal of value in many domainssuch as autonomous driving [49, 13], public transportation [30, 58], migration [39, 22] and sportsanalytics [53].
Revised: Analyzing the spatiotemporal behavior of multiple agents bears a great deal of value in many domainssuch as autonomous driving [49, 13], public transportation [30, 58], migration [39, 22] and sportsanalytics [53].

Original: Particularly, the detection of collective patterns across time and space is of interest tomany communities but non-trivial inter-dependencies between agents render estimating the inherentmulti-modal distributions often difﬁcult.
Revised: Particularly, the detection of collective patterns across time and space is of interest tomany communities but non-trivial inter-dependencies between agents render estimating the inherentmulti-modal distributions often difﬁcult.

Original: Existing generative approaches to modeling sequential multi-agent data thus capitalize on variantsfrom graph networks [55, 12] and (sequential) variational autoencoders [26, 7]; being purely unsupervised, their functionality is grounded in uncovering the implicit modular structure in agent-wise latentvariables directly from the observed trajectories via approximate inference [28, 52, 59, 16, 14, 33].
Revised: Existing generative approaches to modeling sequential multi-agent data thus capitalize on variantsfrom graph networks [55, 12] and (sequential) variational autoencoders [26, 7]; being purely unsupervised, their functionality is grounded in uncovering the implicit modular structure in agent-wise latentvariables directly from the observed trajectories via approximate inference [28, 52, 59, 16, 14, 33].

Original: Although a generative view on the problem is very appealing, approaches in this regime either lackinterpretable and controllable latent factors (e.g., [59]), and are thus limited in practical applicability,or put a strong focus on relationship discovery (e.g., [28]), which limits their generative capacity.
Revised: Although a generative view on the problem is very appealing, approaches in this regime either lackinterpretable and controllable latent factors (e.g., [59]), and are thus limited in practical applicability,or put a strong focus on relationship discovery (e.g., [28]), which limits their generative capacity.

Original: An alternative is offered by supervised approaches that aim to counteract these issues by incorporatingdomain-knowledge in form of discrete labels into the generation process [64].
Revised: An alternative is offered by supervised approaches that aim to counteract these issues by incorporatingdomain-knowledge in form of discrete labels into the generation process [64].

Original: Ideally, these labelsencode implicit expert knowledge that would be learned by the model and substantially expand thescope of information captured in latent space.
Revised: Ideally, these labelsencode implicit expert knowledge that would be learned by the model and substantially expand thescope of information captured in latent space.

Original: Instead of arguing in favor of expensive and tediousmanual annotations of spatiotemporal data, however, recent approaches propose to incorporateheuristic surrogates as makeshift labels into the problem [65, 64, 40].
Revised: Instead of arguing in favor of expensive and tediousmanual annotations of spatiotemporal data, however, recent approaches propose to incorporateheuristic surrogates as makeshift labels into the problem [65, 64, 40].

Original: While reporting impressivepredictive accuracies, they require the existence of label sequences for all training instances and cannot include inexpensive and possibly abundant unlabeled data due to their purely supervisednature.
Revised: While reporting impressivepredictive accuracies, they require the existence of label sequences for all training instances and cannot include inexpensive and possibly abundant unlabeled data due to their purely supervisednature.

Original: To close this gap, we propose semi-supervised variational autoencoders for spatiotemporal multi-agentproblems to process (weakly) labeled as well as unlabeled data.
Revised: To close this gap, we propose semi-supervised variational autoencoders for spatiotemporal multi-agentproblems to process (weakly) labeled as well as unlabeled data.

Original: Our contribution generalizes theclass of semi-supervised variational autoencoders [27] to spatiotemporal domains by using ideas fromvariational recurrent neural networks [7].
Revised: Our contribution generalizes theclass of semi-supervised variational autoencoders [27] to spatiotemporal domains by using ideas fromvariational recurrent neural networks [7].

Original: At a high level, the derived objective function subsumesprevious formalizations into a uniﬁed framework via a disentangled latent space that can incorporatesupervision signals in arbitrary quantities for structuring the discrete latent subspace.
Revised: At a high level, the derived objective function subsumesprevious formalizations into a uniﬁed framework via a disentangled latent space that can incorporatesupervision signals in arbitrary quantities for structuring the discrete latent subspace.

Original: Additionaladaptive graph network layers provide order in- and equivariance of agents and render the proposedapproach appropriate for multi-agent scenarios.
Revised: Additionaladaptive graph network layers provide order in- and equivariance of agents and render the proposedapproach appropriate for multi-agent scenarios.

Original: The resulting semi-supervised approach is applicableto a wide range of problems including the generation of collective movements as well as classifyingsituations of interest.
Revised: The resulting semi-supervised approach is applicableto a wide range of problems including the generation of collective movements as well as classifyingsituations of interest.

Original: Empirically, our contribution signiﬁcantly advances the state-of-the art ingeneration and classiﬁcation tasks on interactive real-world data.
Revised: Empirically, our contribution signiﬁcantly advances the state-of-the art ingeneration and classiﬁcation tasks on interactive real-world data.

Original:  Related Work
Revised:  Related Work

Original: Semi-supervised generative models Semi-supervised variational autoencoders have been originally studied by [27] who propose to incorporate additional labels in latent space.
Revised: Semi-supervised generative models Semi-supervised variational autoencoders have been originally studied by [27] who propose to incorporate additional labels in latent space.

Original: Related approachesintroduce auxiliary variables that leave the original model unchanged but increase the ﬂexibility ofthe variational posterior [34].
Revised: Related approachesintroduce auxiliary variables that leave the original model unchanged but increase the ﬂexibility ofthe variational posterior [34].

Original: [24] propose to explicitly capture label characteristics in latent spaceinstead of the label values themselves and [41] study generalizations that allow for learning morecomplex dependency structures.
Revised: [24] propose to explicitly capture label characteristics in latent spaceinstead of the label values themselves and [41] study generalizations that allow for learning morecomplex dependency structures.

Original: However, all the above approaches are introduced only for staticdomains.
Revised: However, all the above approaches are introduced only for staticdomains.

Original: Our contribution constitutes a generalization of the above approaches to spatiotemporaldomains.
Revised: Our contribution constitutes a generalization of the above approaches to spatiotemporaldomains.

Original: Sequential generative models Approaches designed for predicting agent trajectories are frequentlybased on ideas from sequential generative models to encode temporal dependencies and addressthe stochasticity inherent in future predictions.
Revised: Sequential generative models Approaches designed for predicting agent trajectories are frequentlybased on ideas from sequential generative models to encode temporal dependencies and addressthe stochasticity inherent in future predictions.

Original: Sequential extensions of variational autoencoderswith dynamic latent variables include [3, 7, 11, 15].
Revised: Sequential extensions of variational autoencoderswith dynamic latent variables include [3, 7, 11, 15].

Original: Particularly, the VRNN [7] is related to ourcontribution as both deploy a VAE instance per time step conditioned on a recurrent state.
Revised: Particularly, the VRNN [7] is related to ourcontribution as both deploy a VAE instance per time step conditioned on a recurrent state.

Original: However,their work lacks modeling of a social dimension as well as means to integrate (potentially expensive)discrete behavioral indicators.
Revised: However,their work lacks modeling of a social dimension as well as means to integrate (potentially expensive)discrete behavioral indicators.

Original: Other approaches associate a single global latent variable with eachsequence [4, 17, 9, 8].
Revised: Other approaches associate a single global latent variable with eachsequence [4, 17, 9, 8].

Original: Graph neural networks Deep graph-based (GNNs) approaches [50] are a natural methodologicalchoice for applications requiring modeling datasets composed of sets.
Revised: Graph neural networks Deep graph-based (GNNs) approaches [50] are a natural methodologicalchoice for applications requiring modeling datasets composed of sets.

Original: GNNs operate by learning achain of hidden representations for each node through an iterative process that relies on aggregatingmessages along edges in each network layer.
Revised: GNNs operate by learning achain of hidden representations for each node through an iterative process that relies on aggregatingmessages along edges in each network layer.

Original: Prominent instantiations such as GCN [29], GraphSAGE[18], GAT [55], and others [12, 54, 20, 2, 63, 57] mainly differ in their notion of message passing andneighborhood aggregation [60].
Revised: Prominent instantiations such as GCN [29], GraphSAGE[18], GAT [55], and others [12, 54, 20, 2, 63, 57] mainly differ in their notion of message passing andneighborhood aggregation [60].

Original: See [5] for a discussion on the function approximation capabilitiesof this neural network family.
Revised: See [5] for a discussion on the function approximation capabilitiesof this neural network family.

Original: Generative models for sequential multi-agent data Given the potential beneﬁt across variousdomains, a great deal of recent publications focus on movements of pedestrians or self-driving cars[37, 49, 13, 38, 61, 48, 1, 21, 62, 32].
Revised: Generative models for sequential multi-agent data Given the potential beneﬁt across variousdomains, a great deal of recent publications focus on movements of pedestrians or self-driving cars[37, 49, 13, 38, 61, 48, 1, 21, 62, 32].

Original: [36], however, show that these standard benchmark datasetsgenerally exhibit weak social interaction patterns, so in the remainder, we focus on methods usingteam sport data.
Revised: [36], however, show that these standard benchmark datasetsgenerally exhibit weak social interaction patterns, so in the remainder, we focus on methods usingteam sport data.

Original: Related approaches show that learning and computing realistic rollouts signiﬁcantlybeneﬁt from the incorporation of makeshift annotations and heuristic surrogates into the problem[65, 64, 40].
Revised: Related approaches show that learning and computing realistic rollouts signiﬁcantlybeneﬁt from the incorporation of makeshift annotations and heuristic surrogates into the problem[65, 64, 40].

Original: [10, 33] propose an architecture based on conditional VAEs [51] to model distributionsof future movements of basketball players.
Revised: [10, 33] propose an architecture based on conditional VAEs [51] to model distributionsof future movements of basketball players.

Original: [59, 52] combine ideas from graph networks [12] and
Revised: [59, 52] combine ideas from graph networks [12] and

Original: VRNNs [7] into a uniﬁed framework aiming at modeling data from basketball and soccer.
Revised: VRNNs [7] into a uniﬁed framework aiming at modeling data from basketball and soccer.

Original: Otherrelated approaches [31, 28, 16, 14] aim to explicitly infer discrete latent variables to representinteraction types while executing a trajectory prediction task.
Revised: Otherrelated approaches [31, 28, 16, 14] aim to explicitly infer discrete latent variables to representinteraction types while executing a trajectory prediction task.

Original: Hence, existing generative models for spatiotemporal pattern detection focus on multi-agent trajectory prediction.
Revised: Hence, existing generative models for spatiotemporal pattern detection focus on multi-agent trajectory prediction.

Original: Our contribution constitutes the ﬁrst semi-supervised generalization of variationalautoencoders for spatiotemporal domains that gives rise to addressing tasks beyond representationalor generative modeling.
Revised: Our contribution constitutes the ﬁrst semi-supervised generalization of variationalautoencoders for spatiotemporal domains that gives rise to addressing tasks beyond representationalor generative modeling.

Original:  Methods
Revised:  Methods

Original: 3.1 Preliminaries
Revised: 3.1 Preliminaries

Original: Variational autoencoders (VAEs) [26, 43] allow to capture intrinsic conditional dependencies andare often used in structured problems.
Revised: Variational autoencoders (VAEs) [26, 43] allow to capture intrinsic conditional dependencies andare often used in structured problems.

Original: Essentially, they consist of a θ -parameterized generativemodel p θ ( x , z ) =
Revised: Essentially, they consist of a θ -parameterized generativemodel p θ ( x , z ) =

Original: p ( z ) p θ ( x | z ) and an associated φ -parameterized variational distribution q φ ( z | x ) .
Revised: p ( z ) p θ ( x | z ) and an associated φ -parameterized variational distribution q φ ( z | x ) .

Original: Optimal parameters { θ, φ } are obtained via maximizing the variational lower bound on the marginallikelihood, given by
Revised: Optimal parameters { θ, φ } are obtained via maximizing the variational lower bound on the marginallikelihood, given by

Original: [Equation]
Revised: [Equation]

Original: It is often helpful to augment discrete labels y into the model to guide the generating process [27].
Revised: It is often helpful to augment discrete labels y into the model to guide the generating process [27].

Original: The resulting model p ( x , z , y ) =
Revised: The resulting model p ( x , z , y ) =

Original: p ( x | z , y ) p ( y ) p ( z ) then acts on labeled and unlabeled instances.
Revised: p ( x | z , y ) p ( y ) p ( z ) then acts on labeled and unlabeled instances.

Original: Consequentially, the variational distribution q φ ( y, z | x ) =
Revised: Consequentially, the variational distribution q φ ( y, z | x ) =

Original: q φ ( y | x ) q φ ( z | y, x ) needs to be deﬁnedover both quantities ( z and y ).
Revised: q φ ( y | x ) q φ ( z | y, x ) needs to be deﬁnedover both quantities ( z and y ).

Original: Semi-supervised VAEs (SSVAE) are trained by maximizing
Revised: Semi-supervised VAEs (SSVAE) are trained by maximizing

Original: [Equation]
Revised: [Equation]

Original: where L ( x , y )
Revised: where L ( x , y )

Original: =
Revised: =

Original: E q φ ( z | x ,y )
Revised: E q φ ( z | x ,y )

Original: [log p θ ( x | y, z ) + log p ( y ) + log p ( z ) − log q φ ( z | x , y )] and U ( x ) = 102 (cid:80)
Revised: [log p θ ( x | y, z ) + log p ( y ) + log p ( z ) − log q φ ( z | x , y )] and U ( x ) = 102 (cid:80)

Original: y q φ ( y | x )( L ( x , y ))
Revised: y q φ ( y | x )( L ( x , y ))

Original: + H ( q φ ( y | x )) denote (variational) lower bounds on labeled and unlabeledinstances, respectively, and λ is a balancing term.
Revised: + H ( q φ ( y | x )) denote (variational) lower bounds on labeled and unlabeledinstances, respectively, and λ is a balancing term.

Original: In fact, the original motivation for this frameworkwas inspired by semi-supervised classiﬁcation tasks.
Revised: In fact, the original motivation for this frameworkwas inspired by semi-supervised classiﬁcation tasks.

Original: However, their contribution is also regularly
Revised: However, their contribution is also regularly

Original: [Figure]
Revised: [Figure]

Original: used for learning meaningful representations and generating new data, thus rendering it a suitablemethodological foundation for performing tasks with all sorts of data.
Revised: used for learning meaningful representations and generating new data, thus rendering it a suitablemethodological foundation for performing tasks with all sorts of data.

Original: The dependency structures ofq φ and p θ for SSVAEs are displayed visually as part of Figure 1 (right side).
Revised: The dependency structures ofq φ and p θ for SSVAEs are displayed visually as part of Figure 1 (right side).

Original: 3.2 Problem Formulation
Revised: 3.2 Problem Formulation

Original: We are interested in modeling spatiotemporal multi-agent scenarios and focus on positions x ( a ) t ∈ R 2of an agent a ∈
Revised: We are interested in modeling spatiotemporal multi-agent scenarios and focus on positions x ( a ) t ∈ R 2of an agent a ∈

Original: A over time 1 ≤ t ≤ T , denoted by x ( a ) ≤ T .
Revised: A over time 1 ≤ t ≤ T , denoted by x ( a ) ≤ T .

Original: We assume that a non-empty subset ofagents in A is present in any observed segment and collect their trajectories (in random order) to formcollective movements x ≤ T :=
Revised: We assume that a non-empty subset ofagents in A is present in any observed segment and collect their trajectories (in random order) to formcollective movements x ≤ T :=

Original: { x ( a ) ≤ T | a ∈ A} .
Revised: { x ( a ) ≤ T | a ∈ A} .

Original: We will make use of velocities d x /dt and linearizedmotion ∆ x t = x t (cid:48) − x t , for t (cid:48) >
Revised: We will make use of velocities d x /dt and linearizedmotion ∆ x t = x t (cid:48) − x t , for t (cid:48) >

Original: t , of agents in the remainder.
Revised: t , of agents in the remainder.

Original: In addition, we introduce discrete labels y
Revised: In addition, we introduce discrete labels y

Original: ( a ) t ∈ Y associated with agent
Revised: ( a ) t ∈ Y associated with agent

Original: a ∈ A at time t .
Revised: a ∈ A at time t .

Original: Thecollection of sequences y ≤ T :=
Revised: Thecollection of sequences y ≤ T :=

Original: { y ( a ) ≤ T | a ∈ A} may be best thought of as discriminative causesof variation in the corresponding trajectories x ≤ T .
Revised: { y ( a ) ≤ T | a ∈ A} may be best thought of as discriminative causesof variation in the corresponding trajectories x ≤ T .

Original: In practice, y ≤ T may contain anything thataids in generating multi-agent rollouts, including behavioral indicators, long-term goals, or currenttasks.
Revised: In practice, y ≤ T may contain anything thataids in generating multi-agent rollouts, including behavioral indicators, long-term goals, or currenttasks.

Original: Finally, observations x ≤ T accompanied by labels y ≤ T assemble the labeled share of thedata, denoted by D L = { ( x ≤ T , y ≤ T ) n } 1 ≤ n ≤ N , while unlabeled observations are collected in
Revised: Finally, observations x ≤ T accompanied by labels y ≤ T assemble the labeled share of thedata, denoted by D L = { ( x ≤ T , y ≤ T ) n } 1 ≤ n ≤ N , while unlabeled observations are collected in

Original: D U
Revised: D U

Original: = { ( x ≤ T ) m } 1 ≤ m ≤ M .
Revised: = { ( x ≤ T ) m } 1 ≤ m ≤ M .

Original: We refer to all data by D = D L ∪ D U .
Revised: We refer to all data by D = D L ∪ D U .

Original: We aim at estimating the underlying distribution that generated the observations in D .
Revised: We aim at estimating the underlying distribution that generated the observations in D .

Original: A straightforward solution is to maximize the log likelihood given by
Revised: A straightforward solution is to maximize the log likelihood given by

Original: [Equation]
Revised: [Equation]

Original: However, human movement is inherently non-determenistic and multi-modal.
Revised: However, human movement is inherently non-determenistic and multi-modal.

Original: Thus, it is common tomodel the stochasticity inherent in future agent behaviors via latent representations of VAEs.
Revised: Thus, it is common tomodel the stochasticity inherent in future agent behaviors via latent representations of VAEs.

Original: Wepropose to lower-bound Eq.
Revised: Wepropose to lower-bound Eq.

Original: 1 as follows.
Revised: 1 as follows.

Original: 3.3 SSVAEs for Multi-Agent Trajectories
Revised: 3.3 SSVAEs for Multi-Agent Trajectories

Original: We associate disentangled latent realizations { z t , y t } with each time step t of the segment whileinjecting the dependency structures introduced in Section 3.1 for the generative and inference parts.
Revised: We associate disentangled latent realizations { z t , y t } with each time step t of the segment whileinjecting the dependency structures introduced in Section 3.1 for the generative and inference parts.

Original: To model temporal dependencies, we propose to additionally condition the model components on thepast observations x <t and latent variables z <t and
Revised: To model temporal dependencies, we propose to additionally condition the model components on thepast observations x <t and latent variables z <t and

Original: y <t (cmp.
Revised: y <t (cmp.

Original: The joint distribution factorizesinto
Revised: The joint distribution factorizesinto

Original: [Equation]
Revised: [Equation]

Original: with generating distribution p θ ( x t | x <t , z ≤ t , y ≤ t ) and prior distributions p θ ( z t | x <t , z <t , y <t ) andp θ ( y t | x <t , z <t , y <t ) for latents z t and y t .
Revised: with generating distribution p θ ( x t | x <t , z ≤ t , y ≤ t ) and prior distributions p θ ( z t | x <t , z <t , y <t ) andp θ ( y t | x <t , z <t , y <t ) for latents z t and y t .

Original: Note that we differentiate between observed labels y andlatent variables ˜ y to derive the variational distributions.
Revised: Note that we differentiate between observed labels y andlatent variables ˜ y to derive the variational distributions.

Original: Also note that in our sequential setting, the priors are included in the optimization rather than beingrepresented by ﬁxed distributions to account for latent dynamics and effective sampling at inferencetime.
Revised: Also note that in our sequential setting, the priors are included in the optimization rather than beingrepresented by ﬁxed distributions to account for latent dynamics and effective sampling at inferencetime.

Original: Hence, the surrogate posteriors q φ also need to condition on previous observations and randomvariables to allow for accurate approximations of the true posterior and provide a tractable lowerbound on the log-likelihood in Eqn (1).
Revised: Hence, the surrogate posteriors q φ also need to condition on previous observations and randomvariables to allow for accurate approximations of the true posterior and provide a tractable lowerbound on the log-likelihood in Eqn (1).

Original: Using the factorization introduced in Section 3.1 yields
Revised: Using the factorization introduced in Section 3.1 yields

Original: [Equation]
Revised: [Equation]

Original: Labeled instances render reasoning over latents ˜ y ≤ T unnecessary since the true y ≤ T is alreadyknown.
Revised: Labeled instances render reasoning over latents ˜ y ≤ T unnecessary since the true y ≤ T is alreadyknown.

Original: Hence, the right-hand side in Eqn (2) can be ignored when sampling from D L .
Revised: Hence, the right-hand side in Eqn (2) can be ignored when sampling from D L .

Original: We arrive atthe following results and refer to Appendix A for the proofs.
Revised: We arrive atthe following results and refer to Appendix A for the proofs.

Original: Theorem 1.
Revised: Theorem 1.

Original: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by
Revised: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by

Original: [Equation]
Revised: [Equation]

Original: Correspondingly, for unlabeled instances drawn from D U , sequential label information need to beestimated as shown in the following theorem.
Revised: Correspondingly, for unlabeled instances drawn from D U , sequential label information need to beestimated as shown in the following theorem.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let H ( β ) be the entropy of quantity β .
Revised: Let H ( β ) be the entropy of quantity β .

Original: A lower bound on log p θ ( x ≤ T ) in Eqn (1) isgiven by
Revised: A lower bound on log p θ ( x ≤ T ) in Eqn (1) isgiven by

Original: [Equation]
Revised: [Equation]

Original: The full evidence lower bound (ELBO) is obtained by combining the lower bounds for all data D ,
Revised: The full evidence lower bound (ELBO) is obtained by combining the lower bounds for all data D ,

Original: [Equation]
Revised: [Equation]

Original: U
Revised: U

Original: T (cid:88) t = −U ( x t ) .
Revised: T (cid:88) t = −U ( x t ) .

Original: This formalization focuses on either fully labeled or unlabeled observations, however, a more generalformulation can be obtained to also allow for partially labeled observations.
Revised: This formalization focuses on either fully labeled or unlabeled observations, however, a more generalformulation can be obtained to also allow for partially labeled observations.

Original: 3.4 Semi-supervised Sequential Learning
Revised: 3.4 Semi-supervised Sequential Learning

Original: Intrinsic to our contribution is an encoding module that argues over the label space, q φ ( y ≤ T | x ≤ T ) ,and hence can be used as a classiﬁcation network annotating new observations.
Revised: Intrinsic to our contribution is an encoding module that argues over the label space, q φ ( y ≤ T | x ≤ T ) ,and hence can be used as a classiﬁcation network annotating new observations.

Original: However, the ELBOin Eqn (5) is oblivious to classiﬁcation and simply ignores relevant gradient updates for L ( x t , y t )by discarding Eqn (2).
Revised: However, the ELBOin Eqn (5) is oblivious to classiﬁcation and simply ignores relevant gradient updates for L ( x t , y t )by discarding Eqn (2).

Original: This is clearly inappropriate for semi-supervised learning where the overallgoal is to learn an unknown mapping x
Revised: This is clearly inappropriate for semi-supervised learning where the overallgoal is to learn an unknown mapping x

Original: (cid:55)→ y .
Revised: (cid:55)→ y .

Original: Following [27], we circumvent this by heuristicallyincorporating Eqn (2) into the label-dependent objective.
Revised: Following [27], we circumvent this by heuristicallyincorporating Eqn (2) into the label-dependent objective.

Original: Hence, the full training criterion to learn{ θ, φ } is given by
Revised: Hence, the full training criterion to learn{ θ, φ } is given by

Original: [Equation]
Revised: [Equation]

Original: where λ 1 balances generative and discriminative learning and λ 

Revised: where λ 1 balances generative and discriminative learning and λ

Original: the contribution of labeled andunlabeled data, respectively.
Revised: the contribution of labeled andunlabeled data, respectively.

Original: Thus, the semi-supervised variant contains fully (un)supervised modelingtasks as special cases via adjusting λ 1 , λ 2 accordingly.
Revised: Thus, the semi-supervised variant contains fully (un)supervised modelingtasks as special cases via adjusting λ 1 , λ 2 accordingly.

Original: The resulting supervised target differs fromits unsupervised counterpart only in observing y t instead of factorizing over Y , which results incomputing the log-likelihood (auxiliary loss) instead of the entropy H ( q φ ( y t | x ≤ t , z <t , y <t )) .
Revised: The resulting supervised target differs fromits unsupervised counterpart only in observing y t instead of factorizing over Y , which results incomputing the log-likelihood (auxiliary loss) instead of the entropy H ( q φ ( y t | x ≤ t , z <t , y <t )) .

Original: Eqn (4) encodes the behavior of classiﬁer q φ ( y ≤ T | x ≤ T ) when dealing with unsupervised data.
Revised: Eqn (4) encodes the behavior of classiﬁer q φ ( y ≤ T | x ≤ T ) when dealing with unsupervised data.

Original: Sincedata likelihood log p θ ( x t | x <t , , z ≤ t , y ≤ t ) tends to comprise the largest factor in L ( x t , y t ) , takingthe expectation wrt.
Revised: Sincedata likelihood log p θ ( x t | x <t , , z ≤ t , y ≤ t ) tends to comprise the largest factor in L ( x t , y t ) , takingthe expectation wrt.

Original: q φ (˜ y t | x ≤ t , z <t , ˜ y <t ) encourages the model to realize the highest probabilitymass for the latent label valuel y ∈ Y that incurs the smallest reconstruction loss compared to othercandidates in label space.
Revised: q φ (˜ y t | x ≤ t , z <t , ˜ y <t ) encourages the model to realize the highest probabilitymass for the latent label valuel y ∈ Y that incurs the smallest reconstruction loss compared to othercandidates in label space.

Original: This is a desirable property that reﬂects our assumption of data generation.
Revised: This is a desirable property that reﬂects our assumption of data generation.

Original: 3.5 Design Choices
Revised: 3.5 Design Choices

Original: As described in Section 3.2, we aim to learn a distribution over possible sequences of agent sets thatmay differ in share of discrete annotations (and cardinality).
Revised: As described in Section 3.2, we aim to learn a distribution over possible sequences of agent sets thatmay differ in share of discrete annotations (and cardinality).

Original: However, the previously derived objectivefunctions only account for temporal dependencies, so that independence of the agent dimensionis required to realize permutation-invariant models.
Revised: However, the previously derived objectivefunctions only account for temporal dependencies, so that independence of the agent dimensionis required to realize permutation-invariant models.

Original: This assumption is trivially inappropriate forinteractive systems where future agent movements need to be coordinated with other agents.
Revised: This assumption is trivially inappropriate forinteractive systems where future agent movements need to be coordinated with other agents.

Original: Thus, itis common to propose some form of graph encoding strategy [50] to account for both, equivarianceand agent interactions, respectively.
Revised: Thus, itis common to propose some form of graph encoding strategy [50] to account for both, equivarianceand agent interactions, respectively.

Original: Intuitively, optimal interaction modeling allows to adaptively accomodate structural information fromdifferent levels of granularity for each agent.
Revised: Intuitively, optimal interaction modeling allows to adaptively accomodate structural information fromdifferent levels of granularity for each agent.

Original: In team sports, for example, agent nodes should be ableto capture immediate player inﬂuences as well as holistic team strategies dependent on the interactionstructure best suited for the task at-hand.
Revised: In team sports, for example, agent nodes should be ableto capture immediate player inﬂuences as well as holistic team strategies dependent on the interactionstructure best suited for the task at-hand.

Original: Motivated by the previous considerations, we costumize an
Revised: Motivated by the previous considerations, we costumize an

Original: GNN architecture that aims to mimic the descired behavior.
Revised: GNN architecture that aims to mimic the descired behavior.

Original: Speciﬁcally, we stack attention-based
Revised: Speciﬁcally, we stack attention-based

Original: GNN layers [55] with particularly designed skip-connections [57] and construct the assumed graphstructure using the k (spatially) closest agents.
Revised: GNN layers [55] with particularly designed skip-connections [57] and construct the assumed graphstructure using the k (spatially) closest agents.

Original: There are various possibilities to incorporate graph-based approaches into the overall scheme 1 .
Revised: There are various possibilities to incorporate graph-based approaches into the overall scheme 1 .

Original: Anefﬁcient way to model the distributions deﬁning Eqn (6) is via hidden agent states h ( a ) t conditionedon representations o ( a ) t − 1 ,
Revised: Anefﬁcient way to model the distributions deﬁning Eqn (6) is via hidden agent states h ( a ) t conditionedon representations o ( a ) t − 1 ,

Original: [Equation]
Revised: [Equation]

Original: , where f rnn denotes an RNN transition function, GNN is the described GAT-based GNN and o t ={ o ( a ) t | a ∈ A} refers to the set of updated agent representations.
Revised: , where f rnn denotes an RNN transition function, GNN is the described GAT-based GNN and o t ={ o ( a ) t | a ∈ A} refers to the set of updated agent representations.

Original: Since the elements in set o taggregate neighboring information of the RNN outputs h t via graph networks, the inferred featurevectors describe the entire interactive past of individual agents.
Revised: Since the elements in set o taggregate neighboring information of the RNN outputs h t via graph networks, the inferred featurevectors describe the entire interactive past of individual agents.

Original: While it is reasonable to additionallyenforce intra-timestep dependencies on the latent variables using encoding and decoding GNNmodules, we argue that capturing past spatiotemporal patterns is sufﬁciently informative for highfrequency data [46].
Revised: While it is reasonable to additionallyenforce intra-timestep dependencies on the latent variables using encoding and decoding GNNmodules, we argue that capturing past spatiotemporal patterns is sufﬁciently informative for highfrequency data [46].

Original: Hence, assuming conditional independence within t , the joint movement distribution factorizes across the agent dimension and is given by
Revised: Hence, assuming conditional independence within t , the joint movement distribution factorizes across the agent dimension and is given by

Original: [Equation]
Revised: [Equation]

Original: a ∈A p
Revised: a ∈A p

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: where ψ t = { ψ ( a ) t | a ∈ A} are the distribution parameters and ψ ( a ) t = ϕ
Revised: where ψ t = { ψ ( a ) t | a ∈ A} are the distribution parameters and ψ ( a ) t = ϕ

Original: ( z ( a ) t , y ( a ) t , o ( a ) t − 1 ) .
Revised: ( z ( a ) t , y ( a ) t , o ( a ) t − 1 ) .

Original: Thecomputational logic of the full architecture is schematically depicted in Fig.
Revised: Thecomputational logic of the full architecture is schematically depicted in Fig.

Original:  Empirical Evaluation
Revised:  Empirical Evaluation

Original: For evaluation, we focus on team sports as the coordination of players renders these tasks moredifﬁcult than other domains [36].
Revised: For evaluation, we focus on team sports as the coordination of players renders these tasks moredifﬁcult than other domains [36].

Original: Hence, we experiment on STATS SportVU NBA 3 for comparison,and tracking data from elite soccer 4 .
Revised: Hence, we experiment on STATS SportVU NBA 3 for comparison,and tracking data from elite soccer 4 .

Original: As detailed in Section 3.2, we use agent velocities as inputto all models and assume linear motion between consecutive observations.
Revised: As detailed in Section 3.2, we use agent velocities as inputto all models and assume linear motion between consecutive observations.

Original: For NBA, we adopt theexperimental setup and processing strategy from [40] 5 .
Revised: For NBA, we adopt theexperimental setup and processing strategy from [40] 5 .

Original: The STATS SportVU NBA data comprises tracking positions of offensive plays from the 2016 NBAregular season covering more than 1200 different games where game segment are given by sequencesof length 50 and contain two-dimensional positions of all agents ( 10 players and ball) sampled at 5frames per second.
Revised: The STATS SportVU NBA data comprises tracking positions of offensive plays from the 2016 NBAregular season covering more than 1200 different games where game segment are given by sequencesof length 50 and contain two-dimensional positions of all agents ( 10 players and ball) sampled at 5frames per second.

Original: The data is split into 60% training, 20% validation, and 20% test sets.
Revised: The data is split into 60% training, 20% validation, and 20% test sets.

Original: All data istranslated so that the origin of the underlying coordinate system is mapped onto the top-left corner.
Revised: All data istranslated so that the origin of the underlying coordinate system is mapped onto the top-left corner.

Original: The soccer data consists of 12 matches where positions of players and ball are sampled at 25 framesper second.
Revised: The soccer data consists of 12 matches where positions of players and ball are sampled at 25 framesper second.

Original: The tracking data is accompanied by manual event annotations that we will also make useof in the remainder.
Revised: The tracking data is accompanied by manual event annotations that we will also make useof in the remainder.

Original: Models are trained on eight matches, the remaining four are distributed evenlyinto validation and test data (two matches each).
Revised: Models are trained on eight matches, the remaining four are distributed evenlyinto validation and test data (two matches each).

Original: 4.1 Baselines
Revised: 4.1 Baselines

Original: We evaluate versus several baselines on two distinct tasks: trajectory forecasting (Section 4.2 and 4.4)and spatiotemporal classiﬁcation (Section 4.3 and 4.4).
Revised: We evaluate versus several baselines on two distinct tasks: trajectory forecasting (Section 4.2 and 4.4)and spatiotemporal classiﬁcation (Section 4.3 and 4.4).

Original: Since our approach constitutes the ﬁrst semisupervised model for label predictions in multi-agent scenarios, we need to rely on self-constructedsupervised baselines for scenarios where only a few labels are available, which we discuss in moredetail in the relevant sections.
Revised: Since our approach constitutes the ﬁrst semisupervised model for label predictions in multi-agent scenarios, we need to rely on self-constructedsupervised baselines for scenarios where only a few labels are available, which we discuss in moredetail in the relevant sections.

Original: By contrast, trajectory forecasting is an active topic in different ﬁelds (see, e.g., [45] for an overview),which allows comparisons with a wide range of baseline models.
Revised: By contrast, trajectory forecasting is an active topic in different ﬁelds (see, e.g., [45] for an overview),which allows comparisons with a wide range of baseline models.

Original: Fully supervised generativemodels such as DAG-Net [40] and Weak-Sup [64] use weak labels in the form of agent objectives(heuristically inferred prior to model training) to improve trajectory prediction.
Revised: Fully supervised generativemodels such as DAG-Net [40] and Weak-Sup [64] use weak labels in the form of agent objectives(heuristically inferred prior to model training) to improve trajectory prediction.

Original: In these settings, anagent’s objective at each timestep t is to reach the next area where the observed movement speedfalls below a predeﬁned threshold.
Revised: In these settings, anagent’s objective at each timestep t is to reach the next area where the observed movement speedfalls below a predeﬁned threshold.

Original: Adopting the experimental framework of [40] allows us to makefurther comparisons with Social-Ways [1] and STGAT [21] without additional experiments.
Revised: Adopting the experimental framework of [40] allows us to makefurther comparisons with Social-Ways [1] and STGAT [21] without additional experiments.

Original: STGAT[21] augments a standard GAT with an LSTM [19] to capture both spatial and temporal dependenciesof social interactions while Social-Ways uses Info-GAN to learn multimodal predictive distributions.
Revised: STGAT[21] augments a standard GAT with an LSTM [19] to capture both spatial and temporal dependenciesof social interactions while Social-Ways uses Info-GAN to learn multimodal predictive distributions.

Original: It is difﬁcult to determine the current state-of-the-art in generative baselines for sports tracking databecause there is no consistent setting in terms of forecasting horizon or data (see e.g., [33, 40]).
Revised: It is difﬁcult to determine the current state-of-the-art in generative baselines for sports tracking databecause there is no consistent setting in terms of forecasting horizon or data (see e.g., [33, 40]).

Original: Whilesome baselines are restricted to making predictions for a predetermined horizon, our method learnsspatiotemporal representations autoregressively and thus can be used for the full range of predictionscenarios.
Revised: Whilesome baselines are restricted to making predictions for a predetermined horizon, our method learnsspatiotemporal representations autoregressively and thus can be used for the full range of predictionscenarios.

Original: Hence, we conduct two sets of experiments over both short ( 10 timesteps) and long ( 40timesteps) and additionally benchmark against the most recent and methodologically most relevantwork described below.
Revised: Hence, we conduct two sets of experiments over both short ( 10 timesteps) and long ( 40timesteps) and additionally benchmark against the most recent and methodologically most relevantwork described below.

Original: GVRNN [59] is a graph extension of the VRNN for explicitly modeling jointagent movements for basketball and soccer data.
Revised: GVRNN [59] is a graph extension of the VRNN for explicitly modeling jointagent movements for basketball and soccer data.

Original: dNRI [16] dynamically extracts interaction typesthat determine the parametrization of the decoder.
Revised: dNRI [16] dynamically extracts interaction typesthat determine the parametrization of the decoder.

Original: Finally, GRIN [33] recovers (static) interactionsvia attention using a disentangled latent space.
Revised: Finally, GRIN [33] recovers (static) interactionsvia attention using a disentangled latent space.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4.2 Exploring Boundary Cases for Trajectory Prediction
Revised: 4.2 Exploring Boundary Cases for Trajectory Prediction

Original: Methods We study the purely (un)supervised generation of trajectories by maximizing the respective lower bounds on NBA.
Revised: Methods We study the purely (un)supervised generation of trajectories by maximizing the respective lower bounds on NBA.

Original: For the fully-supervised variant, the weak labels described in Section4.1 can be naturally integrated into the overall scheme via treating them as semantic concepts y ( a ) t .
Revised: For the fully-supervised variant, the weak labels described in Section4.1 can be naturally integrated into the overall scheme via treating them as semantic concepts y ( a ) t .

Original: These target areas naturally vary over time so that the sequential label y ≤ T denotes always the nextdesired position for every involved agent at time t .
Revised: These target areas naturally vary over time so that the sequential label y ≤ T denotes always the nextdesired position for every involved agent at time t .

Original: Following related work [40, 64], we obtain 90different areas/class labels.
Revised: Following related work [40, 64], we obtain 90different areas/class labels.

Original: The underlying data distribution is estimated by maximizing Eqn (3) viasetting λ 

Revised: The underlying data distribution is estimated by maximizing Eqn (3) viasetting λ

Original: = λ 1 = 0 in Eqn (6).
Revised: = λ 1 = 0 in Eqn (6).

Original: The supervised variant of our contribution is denoted by S-MAT .
Revised: The supervised variant of our contribution is denoted by S-MAT .

Original: Although done in [40, 64], direct comparison of unsupervised and supervised generative modelsis inappropriate inasmuch as the latter use ground-truth labels over the entire observation period(which include future information) at prediction time.
Revised: Although done in [40, 64], direct comparison of unsupervised and supervised generative modelsis inappropriate inasmuch as the latter use ground-truth labels over the entire observation period(which include future information) at prediction time.

Original: Thus, we additionally benchmark against afully unsupervised instantiation of our proposed framework lacking ground-truth information forstructuring the discrete latent subspace.
Revised: Thus, we additionally benchmark against afully unsupervised instantiation of our proposed framework lacking ground-truth information forstructuring the discrete latent subspace.

Original: To encourage the model to learn semantically meaningfulconcepts describing distinct movement patterns, we parametrize the decoder dependent on thepredicted agent class ˜ y
Revised: To encourage the model to learn semantically meaningfulconcepts describing distinct movement patterns, we parametrize the decoder dependent on thepredicted agent class ˜ y

Original: ( a ) t .
Revised: ( a ) t .

Original: In this context, ˜ y ( a ) t may be best thought of dynamically assigned agentroles that accurately explain the observed trajectories.
Revised: In this context, ˜ y ( a ) t may be best thought of dynamically assigned agentroles that accurately explain the observed trajectories.

Original: We maximize Eqn (4) using 3 agent typesand refer to the unsupervised variant as U-MAT .
Revised: We maximize Eqn (4) using 3 agent typesand refer to the unsupervised variant as U-MAT .

Original: Further details for S-MAT and U-MAT are given in
Revised: Further details for S-MAT and U-MAT are given in

Original: Appendix C.
Revised: Appendix C.

Original: Metrics We measure the quality of the learned multimodal distribution using common standardmetrics: the minimum over 20 generated samples of the average and ﬁnal L 2 distance betweenpredicted ˆ x ( a ) ≤ T and observed positions x ( a ) ≤ T .
Revised: Metrics We measure the quality of the learned multimodal distribution using common standardmetrics: the minimum over 20 generated samples of the average and ﬁnal L 2 distance betweenpredicted ˆ x ( a ) ≤ T and observed positions x ( a ) ≤ T .

Original: Results Tables 1 and 2 show the results.
Revised: Results Tables 1 and 2 show the results.

Original: The proposed approaches clearly outperform their peersand realize the lowest average L 2 and ﬁnal L 2 distances.
Revised: The proposed approaches clearly outperform their peersand realize the lowest average L 2 and ﬁnal L 2 distances.

Original: The MAT variants effectively accommodatenot only mutual inﬂuences across agents, but also discrete generative factors, leading to betterapproximations of the underlying multi-modal distributions.
Revised: The MAT variants effectively accommodatenot only mutual inﬂuences across agents, but also discrete generative factors, leading to betterapproximations of the underlying multi-modal distributions.

Original: Unsurprisingly, Table 1 also shows thatdefending players exhibit more structure and are easier to predict than offensive players.
Revised: Unsurprisingly, Table 1 also shows thatdefending players exhibit more structure and are easier to predict than offensive players.

Original: The resultsof an ablation study are contained in
Revised: The resultsof an ablation study are contained in

Original: Appendix E.
Revised: Appendix E.

Original: The right part of Fig.
Revised: The right part of Fig.

Original: 2 displays exemplary rolloutsthat underline models’ ability to capture highly complex changes in movement directions.
Revised: 2 displays exemplary rolloutsthat underline models’ ability to capture highly complex changes in movement directions.

Original: 4.3 Semi-Supervised Classiﬁcation
Revised: 4.3 Semi-Supervised Classiﬁcation

Original: We now extend the experimental protocol by targeting accurate label discovery in semi-supervisedscenarios.
Revised: We now extend the experimental protocol by targeting accurate label discovery in semi-supervisedscenarios.

Original: We refer to the full model simply as MAT where λ 0 , λ 1 > 0 and incorporate the fully supervised S-MAT ( λ 0 = 0 ) as an additional baseline.
Revised: We refer to the full model simply as MAT where λ 0 , λ 1 > 0 and incorporate the fully supervised S-MAT ( λ 0 = 0 ) as an additional baseline.

Original: We also compare to an RNN-based classiﬁcationnetwork that addresses inter-agent dependencies by inferring hidden states of agents by the GNN architecture described in Section 3.5.
Revised: We also compare to an RNN-based classiﬁcationnetwork that addresses inter-agent dependencies by inferring hidden states of agents by the GNN architecture described in Section 3.5.

Original: A ﬁnal softmax layer is used for classiﬁcation and the model isoptimized by minimizing the negative log-likelihood.
Revised: A ﬁnal softmax layer is used for classiﬁcation and the model isoptimized by minimizing the negative log-likelihood.

Original: We refer to this baseline as MA-RNN .
Revised: We refer to this baseline as MA-RNN .

Original: Figure 3 shows the performance of the models on NBA when using 50% labeled and 50% unlabeledexamples (left side) and when using varying portions of labeled data (right side), while the supervisedbaselines have access only to the labeled part.
Revised: Figure 3 shows the performance of the models on NBA when using 50% labeled and 50% unlabeledexamples (left side) and when using varying portions of labeled data (right side), while the supervisedbaselines have access only to the labeled part.

Original: The semi-supervised MAT clearly emerges as thestrongest classiﬁer as it converges to the lowest generalization error during a training run and theperformance beneﬁt increases via reduction of labeled data.
Revised: The semi-supervised MAT clearly emerges as thestrongest classiﬁer as it converges to the lowest generalization error during a training run and theperformance beneﬁt increases via reduction of labeled data.

Original: Comparing the numbers to its supervisedpeer ( S-MAT ) highlights the value of incorporating unlabeled data that is effectively utilized by MATto inform classiﬁcation decisions.
Revised: Comparing the numbers to its supervisedpeer ( S-MAT ) highlights the value of incorporating unlabeled data that is effectively utilized by MATto inform classiﬁcation decisions.

Original: Comparing the fully supervised approaches S-MAT and MA-RNN ,the importance of an effective regularization mechanism becomes obvious; MA-RNN quickly overﬁtswhile the lower-bound in S-MAT acts as a regularizer and guarantees gradually increasing predictiveaccuracies on validation sets.
Revised: Comparing the fully supervised approaches S-MAT and MA-RNN ,the importance of an effective regularization mechanism becomes obvious; MA-RNN quickly overﬁtswhile the lower-bound in S-MAT acts as a regularizer and guarantees gradually increasing predictiveaccuracies on validation sets.

Original: 4.4 Combining Generation and Classiﬁcation in Semi-Supervised Settings
Revised: 4.4 Combining Generation and Classiﬁcation in Semi-Supervised Settings

Original: In our last set of experiments, we study the combination of trajectory generation and classiﬁcationand resort to tracking data from elite soccer.
Revised: In our last set of experiments, we study the combination of trajectory generation and classiﬁcationand resort to tracking data from elite soccer.

Original: The reason for favoring this data over NBA is thatthe soccer tracking data is manually annotated by experts from the data provider.
Revised: The reason for favoring this data over NBA is thatthe soccer tracking data is manually annotated by experts from the data provider.

Original: Since the datarequires human annotations for further processing, a successful semi-supervised application bearsa great deal of practical relevance by reducing the expensive labeling effort.
Revised: Since the datarequires human annotations for further processing, a successful semi-supervised application bearsa great deal of practical relevance by reducing the expensive labeling effort.

Original: We focus on labels
Revised: We focus on labels

Original: Y = { pass , other ball action , shot , none } , where class none denotes the absence of all other labels ina frame and use 20% annotated data.
Revised: Y = { pass , other ball action , shot , none } , where class none denotes the absence of all other labels ina frame and use 20% annotated data.

Original: Every label is propagated to the previous ﬁve and the subsequent30 frames.
Revised: Every label is propagated to the previous ﬁve and the subsequent30 frames.

Original: We generate a balanced training set where half of the segments carry label none and studythe combined task of trajectory generation and classiﬁcation.
Revised: We generate a balanced training set where half of the segments carry label none and studythe combined task of trajectory generation and classiﬁcation.

Original: This set of experiments mainly serves the purpose to quantitatively validate the beneﬁts of incorporating expensive human annotations as a generation guide or for structuring the discrete posterior.
Revised: This set of experiments mainly serves the purpose to quantitatively validate the beneﬁts of incorporating expensive human annotations as a generation guide or for structuring the discrete posterior.

Original: This
Revised: This

Original: [Table]
Revised: [Table]

Original: approach comes with little additional costs since the method operates semi-supervised and thereforerequires only a small subset of multi-agent segments to be annotated.
Revised: approach comes with little additional costs since the method operates semi-supervised and thereforerequires only a small subset of multi-agent segments to be annotated.

Original: Given the higher sampling rateand less involved plays in soccer dynamics, we also intend to evaluate the extent to which interactionmodules achieve empirical beneﬁts.
Revised: Given the higher sampling rateand less involved plays in soccer dynamics, we also intend to evaluate the extent to which interactionmodules achieve empirical beneﬁts.

Original: We design our baselines accordinly.
Revised: We design our baselines accordinly.

Original: Generation For trajectory prediction, we compare MAT to VRNN [7] that implicitly assumesindependence across agent dimensions, an interactive version of thereof with GNNs on the hiddenstates ( VRNN+GNN ), GVRNN [59], S-MAT with long-term goals (introduced in Section 4.2, anda diagonal MAT version, MAT-Diag .
Revised: Generation For trajectory prediction, we compare MAT to VRNN [7] that implicitly assumesindependence across agent dimensions, an interactive version of thereof with GNNs on the hiddenstates ( VRNN+GNN ), GVRNN [59], S-MAT with long-term goals (introduced in Section 4.2, anda diagonal MAT version, MAT-Diag .

Original: Note that the VRNN constitutes a strong competitor for thetask at-hand, as it effectively corresponds to a diagonal GVRNN, which has been shown to producecompetitive results on soccer data [59].
Revised: Note that the VRNN constitutes a strong competitor for thetask at-hand, as it effectively corresponds to a diagonal GVRNN, which has been shown to producecompetitive results on soccer data [59].

Original: The left part of Table 3 shows the results.
Revised: The left part of Table 3 shows the results.

Original: Although the results are generally more similar comparedto the basketball experiments, the semi-supervised MAT generates trajectories that are closest in bothmetrics to the observed ones.
Revised: Although the results are generally more similar comparedto the basketball experiments, the semi-supervised MAT generates trajectories that are closest in bothmetrics to the observed ones.

Original: We hypothesize that decreasing the frequency of the data and extractingmulti-agent segments that merely consist of highly interactive plays yields an signiﬁcant increase inperformance gap.
Revised: We hypothesize that decreasing the frequency of the data and extractingmulti-agent segments that merely consist of highly interactive plays yields an signiﬁcant increase inperformance gap.

Original: Classiﬁcation MAT learns a model over the label space Y simultaneously to the generative model.
Revised: Classiﬁcation MAT learns a model over the label space Y simultaneously to the generative model.

Original: To evaluate classiﬁcation performance, we turn the output of the corresponding softmax for a givenframe into a class label whenever it exceeded a predeﬁned threshold.
Revised: To evaluate classiﬁcation performance, we turn the output of the corresponding softmax for a givenframe into a class label whenever it exceeded a predeﬁned threshold.

Original: We compare the predictionperformance again to MA-RNN , a diagonal variant MA-RNN-Diag , and S-MAT .
Revised: We compare the predictionperformance again to MA-RNN , a diagonal variant MA-RNN-Diag , and S-MAT .

Original: The resulting accuracies and F1 scores are summarized in Table 3.
Revised: The resulting accuracies and F1 scores are summarized in Table 3.

Original: Once more, MAT clearly beats thebaselines and stands out by the highest accuracy and F1 scores.
Revised: Once more, MAT clearly beats thebaselines and stands out by the highest accuracy and F1 scores.

Original: The result impressively demonstratesthe beneﬁt of including unlabeled data for the task.
Revised: The result impressively demonstratesthe beneﬁt of including unlabeled data for the task.

Original: Surprisingly, MA-RNN performs better than
Revised: Surprisingly, MA-RNN performs better than

Original: S-MAT .
Revised: S-MAT .

Original: This ﬁnding is in contrast to the results on NBA shown in Figure 3.
Revised: This ﬁnding is in contrast to the results on NBA shown in Figure 3.

Original: However, soccer isplayed on a much larger pitch and movements are likely more linear compared to basketball, whichleads to less multimodality of the distributions involved and a consequently decreased beneﬁt ofvariational methods.
Revised: However, soccer isplayed on a much larger pitch and movements are likely more linear compared to basketball, whichleads to less multimodality of the distributions involved and a consequently decreased beneﬁt ofvariational methods.

Original: Together with a signiﬁcantly reduced label space and sufﬁcient amounts of datathis may lead to a simpler learning task where MA-RNN is less prone to overﬁtting.
Revised: Together with a signiﬁcantly reduced label space and sufﬁcient amounts of datathis may lead to a simpler learning task where MA-RNN is less prone to overﬁtting.

Original: The left part of Fig.
Revised: The left part of Fig.

Original: 2 shows exemplary prediction probabilities for two segments and three possibleevents.
Revised: 2 shows exemplary prediction probabilities for two segments and three possibleevents.

Original: In both segments, the algorithm is highly conﬁdent no shot-action will occur in the near future.
Revised: In both segments, the algorithm is highly conﬁdent no shot-action will occur in the near future.

Original: The segment on top contains an event of the class other ball action followed by an event of class pass .
Revised: The segment on top contains an event of the class other ball action followed by an event of class pass .

Original: Both are correctly identiﬁed by MAT .
Revised: Both are correctly identiﬁed by MAT .

Original: The former is clearly indicated by a peak in the correspondingprobability chart which then decreases to give rise to the following pass action.
Revised: The former is clearly indicated by a peak in the correspondingprobability chart which then decreases to give rise to the following pass action.

Original: The segment on thebottom shows solely a single pass event that is also clearly identiﬁable by a peak at the correct frame.
Revised: The segment on thebottom shows solely a single pass event that is also clearly identiﬁable by a peak at the correct frame.

Original:  Conclusion
Revised:  Conclusion

Original: We presented semi-supervised variational autoencoders for spatiotemporal multi-agent scenarios.
Revised: We presented semi-supervised variational autoencoders for spatiotemporal multi-agent scenarios.

Original: The proposed approach effectively combined ideas from semi-supervised variational autoencoders,variational recurrent neural networks, and graph neural networks.
Revised: The proposed approach effectively combined ideas from semi-supervised variational autoencoders,variational recurrent neural networks, and graph neural networks.

Original: Empirically, our approach clearlyoutperformed the state-of-the-art in sequential generation and (semi-supervised) classiﬁcation tasks inall experiments.
Revised: Empirically, our approach clearlyoutperformed the state-of-the-art in sequential generation and (semi-supervised) classiﬁcation tasks inall experiments.

Original: The performance underlines the beneﬁt of including unlabeled data in spatiotemporalproblems where labeled sequences are either scarce or assembled from weak makeshift signals.
Revised: The performance underlines the beneﬁt of including unlabeled data in spatiotemporalproblems where labeled sequences are either scarce or assembled from weak makeshift signals.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Since this work constitutes the
Revised: [Yes] Since this work constitutes the

Original: ﬁrst semi-supervised approach for spatiotemporal multi-agent problems, it is difﬁcult
Revised: ﬁrst semi-supervised approach for spatiotemporal multi-agent problems, it is difﬁcult

Original: to discuss techical limitations at this stage.
Revised: to discuss techical limitations at this stage.

Original: Application-wise, there are clear limitations
Revised: Application-wise, there are clear limitations

Original: e.g., by using weak makeshift labels.
Revised: e.g., by using weak makeshift labels.

Original: We discuss these issues across the paper (e
Revised: We discuss these issues across the paper (e

Original: g.,
Revised: g.,

Original: Section
Revised: Section

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We have
Revised: [No] We have

Original: a clear focus on team sports.
Revised: a clear focus on team sports.

Original: Though dual use of the proposed technique is certainly
Revised: Though dual use of the proposed technique is certainly

Original: possible (e.g., military domains), it may be a bit far fetched as we have no expertise in
Revised: possible (e.g., military domains), it may be a bit far fetched as we have no expertise in

Original: these domains and would only be able to deliver uneducated guesses about whether
Revised: these domains and would only be able to deliver uneducated guesses about whether

Original: this is realistic or not.
Revised: this is realistic or not.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section4.1.
Revised: [Yes] See Section4.1.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] All proofs are
Revised: [Yes] All proofs are

Original: contained in the Appendix.
Revised: contained in the Appendix.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] The NBA data
Revised: [Yes] The NBA data

Original: set is publicly available and we provide a URL in the paper.
Revised: set is publicly available and we provide a URL in the paper.

Original: The soccer data is private
Revised: The soccer data is private

Original: and must not be shared with third parties.
Revised: and must not be shared with third parties.

Original: We will release the code and include a URL
Revised: We will release the code and include a URL

Original: in the paper as well after the paper has been accepted.
Revised: in the paper as well after the paper has been accepted.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: See Section more details are provided in the Appendixl.
Revised: See Section more details are provided in the Appendixl.

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [No]
Revised: [No]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix4.
Revised: [Yes] See Appendix4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: [Table]
Revised: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: [Table]
Revised: [Table]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See supplemental material
Revised: [Yes] See supplemental material

Original: [Table]
Revised: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: [Table]
Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]
Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [Yes] The NBA data is publicly available and free, for soccer we have
Revised: [Yes] The NBA data is publicly available and free, for soccer we have

Original: [Table]
Revised: [Table]

Original: the permission of the data provider.
Revised: the permission of the data provider.

Original: [Table]
Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] The soccer and basketball players are probably
Revised: [Yes] The soccer and basketball players are probably

Original: [Table]
Revised: [Table]

Original: identiﬁable in the data, but they/their contract/their clubs agreed in recording the data.
Revised: identiﬁable in the data, but they/their contract/their clubs agreed in recording the data.

Original: [Table]
Revised: [Table]

Original: The soccer data is private and cannot be shared.
Revised: The soccer data is private and cannot be shared.

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]
Revised: [Table]

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: A Proofs of the Theorems
Revised: A Proofs of the Theorems

Original: We begin with Theorem 2 for simplicity and sketch the proof of Theorem 1 afterwards.
Revised: We begin with Theorem 2 for simplicity and sketch the proof of Theorem 1 afterwards.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let H ( β ) be the entropy of quantity β .
Revised: Let H ( β ) be the entropy of quantity β .

Original: A lower bound on log p θ ( x ≤ T ) in Eqn (1) isgiven by
Revised: A lower bound on log p θ ( x ≤ T ) in Eqn (1) isgiven by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For unlabeled training instances, deﬁning the marginal likelihood according to the generativestructure introduced in Section 3.3 yields
Revised: For unlabeled training instances, deﬁning the marginal likelihood according to the generativestructure introduced in Section 3.3 yields

Original: [Equation]
Revised: [Equation]

Original: To derive the lower bound on the log likelihood
Revised: To derive the lower bound on the log likelihood

Original: [Equation]
Revised: [Equation]

Original: we incorporate the variational information into theabove deﬁnition and apply Jensen’s inequality:
Revised: we incorporate the variational information into theabove deﬁnition and apply Jensen’s inequality:

Original: [Equation]
Revised: [Equation]

Original: , y <t
Revised: , y <t

Original: [Equation]
Revised: [Equation]

Original: Thus, we can write
Revised: Thus, we can write

Original: [Equation]
Revised: [Equation]

Original: which is identical to Eqn 3.
Revised: which is identical to Eqn 3.

Original: Theorem 1.
Revised: Theorem 1.

Original: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by
Revised: A lower bound on log p θ ( x ≤ T , y ≤ T ) in Eqn (1) is given by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤ T
Revised: ≤ T

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: B Implementation Details
Revised: B Implementation Details

Original: All models are implemented using
Revised: All models are implemented using

Original: PyTorch [42].
Revised: PyTorch [42].

Original: For our experiments, wedeﬁne the model components as follows.
Revised: For our experiments, wedeﬁne the model components as follows.

Original: The encoders are modeled asq φ (˜ y ( a ) | x ≤ t , z <t , y <t ) =
Revised: The encoders are modeled asq φ (˜ y ( a ) | x ≤ t , z <t , y <t ) =

Original: Cat ( y ( a ) t | f y enc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t )]) and q φ ( z ( a ) t | x ≤ t , z <t , y ≤ t ) =
Revised: Cat ( y ( a ) t | f y enc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t )]) and q φ ( z ( a ) t | x ≤ t , z <t , y ≤ t ) =

Original: N ( z ( a ) t | f
Revised: N ( z ( a ) t | f

Original: z enc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t ) , ϕ y ( y ( a ) t )]) for uncovering discrete and continuous latent information,respectively.
Revised: z enc ([ o ( a ) t − 1 , ϕ x ( x ( a ) t ) , ϕ y ( y ( a ) t )]) for uncovering discrete and continuous latent information,respectively.

Original: The prior distributions are computed analogously omitting input x t .
Revised: The prior distributions are computed analogously omitting input x t .

Original: Agent movementsare represented as p θ ( x ( a ) t | x <t , z ≤ t , y ≤ t ) =
Revised: Agent movementsare represented as p θ ( x ( a ) t | x <t , z ≤ t , y ≤ t ) =

Original: N ( z ( a ) t | f dec ([ o ( a ) t − 1 , ϕ x ( x ( a ) t ) , ϕ y ( y ( a ) t )]) .
Revised: N ( z ( a ) t | f dec ([ o ( a ) t − 1 , ϕ x ( x ( a ) t ) , ϕ y ( y ( a ) t )]) .

Original: All functions f and feature extractors ϕ are two-layer MLPs with LeakyRelu [56] activations.
Revised: All functions f and feature extractors ϕ are two-layer MLPs with LeakyRelu [56] activations.

Original: We use2 -layer
Revised: We use2 -layer

Original: GRU networks [6] for recurrence.
Revised: GRU networks [6] for recurrence.

Original: Moreover, we deﬁne the graph network using 3 GAT layersimplemented according to [55] and the graph structure wrt.the k
Revised: Moreover, we deﬁne the graph network using 3 GAT layersimplemented according to [55] and the graph structure wrt.the k

Original: = 2 and k = 5 spatially closestagents for basketball and soccer, respectively.
Revised: = 2 and k = 5 spatially closestagents for basketball and soccer, respectively.

Original: We implement skip-connections as described in [57]and use concatenation followed by a linear transformation to aggregate intermediate layer embeddingsto the GNN model output.
Revised: We implement skip-connections as described in [57]and use concatenation followed by a linear transformation to aggregate intermediate layer embeddingsto the GNN model output.

Original: For generative tasks, we use λ 1 = 0 , however, including the auxiliary lossyields negligible deterioration.
Revised: For generative tasks, we use λ 1 = 0 , however, including the auxiliary lossyields negligible deterioration.

Original: For classiﬁcation, we use λ 0 = 1 and α = 10 ∗ (1 / % of labeled data ) .
Revised: For classiﬁcation, we use λ 0 = 1 and α = 10 ∗ (1 / % of labeled data ) .

Original: The model operates solely on agent velocities (input and output).
Revised: The model operates solely on agent velocities (input and output).

Original: However, we inject position tuplesto the all model components via teacher forcing.
Revised: However, we inject position tuplesto the all model components via teacher forcing.

Original: We use the gumbel-softmax trick [35, 23] whensampling from categorical distributions.
Revised: We use the gumbel-softmax trick [35, 23] whensampling from categorical distributions.

Original: Training was executed on an Nvidia V100 GPU and tookabout 16 hours to complete 300 epochs while comsuming ∼ 15 Gb for basketball.
Revised: Training was executed on an Nvidia V100 GPU and tookabout 16 hours to complete 300 epochs while comsuming ∼ 15 Gb for basketball.

Original: All models areoptimized using Adam [25] with a learning rate of 0 , 001 and gradient clipping using a max norm of10 .
Revised: All models areoptimized using Adam [25] with a learning rate of 0 , 001 and gradient clipping using a max norm of10 .

Original: Baselines As stated in Section 4.1, we report against the values reported in [40].
Revised: Baselines As stated in Section 4.1, we report against the values reported in [40].

Original: For [16, 33, 37],we adapt the source code from their ofﬁcial repositories to our experimental setting 6 For GVRNN[59], we re-implemented the model according to the descriptions in their paper and designed theoverall architecture such that it is comparable in parameter number to our method.
Revised: For [16, 33, 37],we adapt the source code from their ofﬁcial repositories to our experimental setting 6 For GVRNN[59], we re-implemented the model according to the descriptions in their paper and designed theoverall architecture such that it is comparable in parameter number to our method.

Original: Soccer application Since the labels denote ball-centric events, we use the output of the ball nodefor loss computation and evaluation.
Revised: Soccer application Since the labels denote ball-centric events, we use the output of the ball nodefor loss computation and evaluation.

Original: The F 1 score is computed as follows.
Revised: The F 1 score is computed as follows.

Original: We annotate a multiagent segment when the derived probability estimates exceed an externally deﬁned threshold value.
Revised: We annotate a multiagent segment when the derived probability estimates exceed an externally deﬁned threshold value.

Original: We obtain TP values (FP values) when the predicted event coincides (disagrees) with the groundtruth annotation.
Revised: We obtain TP values (FP values) when the predicted event coincides (disagrees) with the groundtruth annotation.

Original: FN values are deﬁned by anotated segments that remain undetected.
Revised: FN values are deﬁned by anotated segments that remain undetected.

Original: We compute
Revised: We compute

Original: F1-scores for 100 distinct threshold values in the range between 0 . 5 and 0 .
Revised: F1-scores for 100 distinct threshold values in the range between 0 . 5 and 0 .

Original: 98 and only report themaximum F1-score.
Revised: 98 and only report themaximum F1-score.

Original: However, threshold optimization yields only negligible improvement over simplyusing 0 .
Revised: However, threshold optimization yields only negligible improvement over simplyusing 0 .

Original: CMore Details on the MAT Models
Revised: C MAT Model Details

Original: In Section 3.1, we deﬁne y ≤ T as arbitrary discrete (possibly latent) behavioral indicators.
Revised: In Section 3.1, we deﬁne y ≤ T as arbitrary discrete (possibly latent) behavioral indicators.

Original: To validatethis general formulation (and our proposed architecture), we vary the speciﬁc deﬁnition of the variable across experiments.
Revised: To validatethis general formulation (and our proposed architecture), we vary the speciﬁc deﬁnition of the variable across experiments.

Original: Accordingly, we propose different framework instantiations that are described inmore detail in this section.
Revised: Accordingly, we propose different framework instantiations that are described inmore detail in this section.

Original: S-MAT First, we observe in Section 4.2 that existing SOTA trajectory prediction approaches(Section 4.1) use heuristically generated labels for trajectory prediction that encode agents’ intentsor goals over a discretized position space.
Revised: S-MAT First, we observe in Section 4.2 that existing SOTA trajectory prediction approaches(Section 4.1) use heuristically generated labels for trajectory prediction that encode agents’ intentsor goals over a discretized position space.

Original: We then note that our formalization allows to naturallyintegrate such long-term goals into the overall scheme via treating them as discrete semantic concepts.
Revised: We then note that our formalization allows to naturallyintegrate such long-term goals into the overall scheme via treating them as discrete semantic concepts.

Original: Since these labels are generated heuristically based on the trajectory input prior to model training,this model instantiation is fully-supervised and is referred to as S-MAT .
Revised: Since these labels are generated heuristically based on the trajectory input prior to model training,this model instantiation is fully-supervised and is referred to as S-MAT .

Original: We refer to [64] for moredetails on how to produce the weak labels used for training S-MAT.
Revised: We refer to [64] for moredetails on how to produce the weak labels used for training S-MAT.

Original: U-MAT
Revised: U-MAT

Original: However, we found (although done in [64, 40]) that benchmarking against unsupervisedbaselines is inappropriate for the reasons described in Section 4.1.
Revised: However, we found (although done in [64, 40]) that benchmarking against unsupervisedbaselines is inappropriate for the reasons described in Section 4.1.

Original: Since most SOTA approachesare unsupervised generative models, we additionally propose a fully-unsupervised instantiation ofour framework ( U-MAT ) employing no external guidance for discrete latent structuring.
Revised: Since most SOTA approachesare unsupervised generative models, we additionally propose a fully-unsupervised instantiation ofour framework ( U-MAT ) employing no external guidance for discrete latent structuring.

Original: From thepredicted (latent) categorical distribution q φ , we sample a label value and exploit separate motionpredictors (different decoder parameterizations) p θ dependent on the realized value.
Revised: From thepredicted (latent) categorical distribution q φ , we sample a label value and exploit separate motionpredictors (different decoder parameterizations) p θ dependent on the realized value.

Original: Intuitively, thisencourages the model to learn categories describing fundamentally different movement patterns,which can be interpreted as dynamic “agent roles”.
Revised: Intuitively, thisencourages the model to learn categories describing fundamentally different movement patterns,which can be interpreted as dynamic “agent roles”.

Original: Thus, the concept of agent roles here is merelyan intuitive explanation and realized via an inductive bias that increases the “scope” of latentinformation encoded without utilizing any supervision.
Revised: Thus, the concept of agent roles here is merelyan intuitive explanation and realized via an inductive bias that increases the “scope” of latentinformation encoded without utilizing any supervision.

Original: To the best of our knowledge, parameterizingthe generation module based on inferred agent categories is novel and could provide valuable insightsfor practitioners.
Revised: To the best of our knowledge, parameterizingthe generation module based on inferred agent categories is novel and could provide valuable insightsfor practitioners.

Original: D Experiments with Drone Data
Revised: D Experiments with Drone Data

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: To showcase applicability to scenarios that exhibt variable numbers of agents, we also report resultson the Stanford Drone Data (SDD) [44].
Revised: To showcase applicability to scenarios that exhibt variable numbers of agents, we also report resultson the Stanford Drone Data (SDD) [44].

Original: SDD is a collection of videos recorded by drones at eightlocations at Stanford.
Revised: SDD is a collection of videos recorded by drones at eightlocations at Stanford.

Original: While pedestrians predominate as interacting agents, cyclists, skateboarders,cars, buses, and golf carts are also present.
Revised: While pedestrians predominate as interacting agents, cyclists, skateboarders,cars, buses, and golf carts are also present.

Original: We use the TrajNet benchmark [47] of the data, providingsequential two-dimensional real-world coordinates at a frame rate of 2.5 frames per second.
Revised: We use the TrajNet benchmark [47] of the data, providingsequential two-dimensional real-world coordinates at a frame rate of 2.5 frames per second.

Original: Wefollow [40] regarding the data processing strategy 7 .
Revised: Wefollow [40] regarding the data processing strategy 7 .

Original: We incorporate PECNet [37] for comparison asit is considered state-of-the-art for SDD [36].
Revised: We incorporate PECNet [37] for comparison asit is considered state-of-the-art for SDD [36].

Original: PECNet models stochasticity in the ﬁnal position of thepedestrians conditioned on the past motion history.
Revised: PECNet models stochasticity in the ﬁnal position of thepedestrians conditioned on the past motion history.

Original: To generate weak labels, we move a time window through each trajectory, with the respective endcells acting as agent targets y ( a ) t .
Revised: To generate weak labels, we move a time window through each trajectory, with the respective endcells acting as agent targets y ( a ) t .

Original: Static time windows allow us to use the fully-supervised variant forall comparisons (including fully unsupervised models).
Revised: Static time windows allow us to use the fully-supervised variant forall comparisons (including fully unsupervised models).

Original: The scene at hand is thereby discretized into960 areas and results are shown in Table 4.
Revised: The scene at hand is thereby discretized into960 areas and results are shown in Table 4.

Original: We observe the same general pattern as for the basketballexperiments.
Revised: We observe the same general pattern as for the basketballexperiments.

Original: [Table]
Revised: [Table]

Original: E Ablation Study
Revised: E Ablation Study

Original: [Table]
Revised: [Table]

Original: Table 5 validates the proposed architecture by showing results of an ablation study.
Revised: Table 5 validates the proposed architecture by showing results of an ablation study.

Original: We test a fullyconnected graph ( S-MAT-Full ), an independent version with diagonal adjacency matrices ( S-MATDiag ), a variant that employs GNNs for variational, generative, and prior distribution parameters, butignores interactive updates for the hidden states ( S-MAT-GRVNN ) similar to [59], and a variant that isidentical to the latter but additionally includes a GNN for the hidden state ( S-MAT-GRVNN-Hidden ).
Revised: We test a fullyconnected graph ( S-MAT-Full ), an independent version with diagonal adjacency matrices ( S-MATDiag ), a variant that employs GNNs for variational, generative, and prior distribution parameters, butignores interactive updates for the hidden states ( S-MAT-GRVNN ) similar to [59], and a variant that isidentical to the latter but additionally includes a GNN for the hidden state ( S-MAT-GRVNN-Hidden ).

Original: The table provides supporting evidence for our design choices: though the GVRNNs are theoreticallyable to capture intra-timestep dependencies, we observe signiﬁcant drops in performance compared tothe other competitors.
Revised: The table provides supporting evidence for our design choices: though the GVRNNs are theoreticallyable to capture intra-timestep dependencies, we observe signiﬁcant drops in performance compared tothe other competitors.

Original: The S-MAT-Diag and MAT-Full experiments suggest that our model capturesinteraction patterns among agents very well; S-MAT denotes a valuable contribution to the large bodyof research that explicitly addresses modeling multi-agent data accurately.
Revised: The S-MAT-Diag and MAT-Full experiments suggest that our model capturesinteraction patterns among agents very well; S-MAT denotes a valuable contribution to the large bodyof research that explicitly addresses modeling multi-agent data accurately.