Original: G ENERATING M ULTI -A GENT T RAJECTORIES USING P ROGRAMMATIC W EAK S UPERVISION
Revised: G ENERATING M ULTI -A GENT T RAJECTORIES USING P ROGRAMMATIC W EAK S UPERVISION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as offensive basketball gameplay.
Revised: We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as offensive basketball gameplay.

Original: When modeling such settings, it is often beneﬁcial to design hierarchical models that can capture long-term coordination using intermediate variables.
Revised: When modeling such settings, it is often beneﬁcial to design hierarchical models that can capture long-term coordination using intermediate variables.

Original: Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulatable way.
Revised: Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulatable way.

Original: We present a hierarchical framework that can effectively learn such sequential generative models.
Revised: We present a hierarchical framework that can effectively learn such sequential generative models.

Original: Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime.
Revised: Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime.

Original: In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods.
Revised: In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods.

Original: We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts
Revised: We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ongoing explosion of recorded tracking data is enabling the study of ﬁne-grained behavior in many domains: sports (Miller et al., 2014;
Revised: The ongoing explosion of recorded tracking data is enabling the study of ﬁne-grained behavior in many domains: sports (Miller et al., 2014;

Original: Yue et al., 2014; Zheng et al., 2016; Le et al., 2017), video games (Ross et al., 2011), video & motion capture (Suwajanakorn et al., 2017; Taylor et al., 2017;
Revised: Yue et al., 2014; Zheng et al., 2016; Le et al., 2017), video games (Ross et al., 2011), video & motion capture (Suwajanakorn et al., 2017; Taylor et al., 2017;

Original: Xue et al., 2016), navigation & driving (Ziebart et al., 2009; Zhang & Cho, 2017; Li et al., 2017), laboratory animal behaviors (Johnson et al., 2016; Eyjolfsdottir et al., 2017), and tele-operated robotics (Abbeel & Ng, 2004; Lin et al., 2006).
Revised: Xue et al., 2016), navigation & driving (Ziebart et al., 2009; Zhang & Cho, 2017; Li et al., 2017), laboratory animal behaviors (Johnson et al., 2016; Eyjolfsdottir et al., 2017), and tele-operated robotics (Abbeel & Ng, 2004; Lin et al., 2006).

Original: However, it is an open challenge to develop sequential generative models leveraging such data, for instance, to capture the complex behavior of multiple cooperating agents.
Revised: However, it is an open challenge to develop sequential generative models leveraging such data, for instance, to capture the complex behavior of multiple cooperating agents.

Original: Figure 1a shows an example of offensive players in basketball moving unpredictably and with multimodal distributions over possible trajectories.
Revised: Figure 1a shows an example of offensive players in basketball moving unpredictably and with multimodal distributions over possible trajectories.

Original: Figure 1b depicts a simpliﬁed
Revised: Figure 1b depicts a simpliﬁed

Original: Boids model from (Reynolds, 1987) for modeling animal schooling behavior in which the agents can be friendly or unfriendly.
Revised: Boids model from (Reynolds, 1987) for modeling animal schooling behavior in which the agents can be friendly or unfriendly.

Original: In both cases, agent behavior is highly coordinated and non-deterministic , and the space of all multi-agent trajectories is naively exponentially large.
Revised: In both cases, agent behavior is highly coordinated and non-deterministic , and the space of all multi-agent trajectories is naively exponentially large.

Original: When modeling such sequential data, it is often beneﬁcial to design hierarchical models that can capture long-term coordination using intermediate variables or representations (Li et al., 2015; Zheng et al., 2016).
Revised: When modeling such sequential data, it is often beneﬁcial to design hierarchical models that can capture long-term coordination using intermediate variables or representations (Li et al., 2015; Zheng et al., 2016).

Original: An attractive use-case for these intermediate variables is to capture interesting highlevel behavioral semantics in an interpretable and manipulable way .
Revised: An attractive use-case for these intermediate variables is to capture interesting highlevel behavioral semantics in an interpretable and manipulable way .

Original: For instance, in the basketball setting, intermediate variables can encode long-term strategies and team formations.
Revised: For instance, in the basketball setting, intermediate variables can encode long-term strategies and team formations.

Original: Conventional approaches to learning interpretable intermediate variables typically focus on learning disentangled latent representations in an unsupervised way (e.g., (Li et al., 2017; Wang et al., 2017)), but it is challenging for such approaches to handle complex sequential settings (Chen et al., 2016).
Revised: Conventional approaches to learning interpretable intermediate variables typically focus on learning disentangled latent representations in an unsupervised way (e.g., (Li et al., 2017; Wang et al., 2017)), but it is challenging for such approaches to handle complex sequential settings (Chen et al., 2017). [Figure]

Original:  To address this challenge, we present a hierarchical framework that can effectively learn such sequential generative models, while using programmatic weak supervision (PWS).
Revised: [Figure] To address this challenge, we present a hierarchical framework that can effectively learn such sequential generative models, while using programmatic weak supervision.

Original: Our approach uses a labeling function to programmatically produce useful weak labels for supervised learning of interpretable intermediate representations.
Revised: Our approach uses a labeling function to programmatically produce useful weak labels for supervised learning of interpretable intermediate representations.

Original: This approach is inspired by recent work on data programming (Ratner et al., 2016), which uses cheap and noisy labeling functions to signiﬁcantly speed up learning.
Revised: This approach is inspired by recent work on data programming (Ratner et al., 2016), which uses cheap and noisy labeling functions to signiﬁcantly speed up learning.

Original: In this work, we extend this approach to the spatiotemporal regime.
Revised: In this work, we extend this approach to the spatiotemporal regime.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] [Figure] [Figure] • We propose a hierarchical framework for sequential generative modeling.
Revised: • We propose a hierarchical framework for sequential generative modeling.

Original: Our approach is compatible with many existing deep generative models.
Revised: Our approach is compatible with many existing deep generative models.

Original: • We show how to programmatically produce weak labels of macro-intents to train the intermediate representation in a supervised fashion.
Revised: • We show how to programmatically produce weak labels of macro-intents to train the intermediate representation in a supervised fashion.

Original: Our approach is easy to implement and results in highly interpretable intermediate variables, which allows for conditional inference by grounding macro-intents to manipulate behaviors.
Revised: Our approach is easy to implement and results in highly interpretable intermediate variables, which allows for conditional inference by grounding macro-intents to manipulate behaviors.

Original: • Focusing on multi-agent tracking data, we show that our approach can generate highquality trajectories and effectively encode long-term coordination between multiple agents.
Revised: • Focusing on multi-agent tracking data, we show that our approach can generate highquality trajectories and effectively encode long-term coordination between multiple agents.

Original: In addition to synthetic settings, we showcase our approach in an application on modeling team offense in basketball.
Revised: In addition to synthetic settings, we showcase our approach in an application on modeling team offense in basketball.

Original: We validate our approach both quantitatively and qualitatively, including a user study comparison with professional sports analysts, and show signiﬁcant improvements over standard baselines.
Revised: We validate our approach both quantitatively and qualitatively, including a user study comparison with professional sports analysts, and show signiﬁcant improvements over standard baselines.

Original: An interactive demo is available at http://basketball-ai.com/ .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Deep generative models.
Revised: Deep generative models.

Original: The study of deep generative models is an increasingly popular research area, due to their ability to inherit both the ﬂexibility of deep learning and the probabilistic semantics of generative models.
Revised: The study of deep generative models is an increasingly popular research area, due to their ability to inherit both the ﬂexibility of deep learning and the probabilistic semantics of generative models.

Original: In general, there are two ways that one can incorporate stochastics into deep models.
Revised: In general, there are two ways that one can incorporate stochastics into deep models.

Original: The ﬁrst approach models an explicit distribution over actions in the output layer, e.g., via logistic regression (Chen et al., 2015; Oord et al., 2016a;b; Zheng et al., 2016; Eyjolfsdottir et al., 2017).
Revised: The ﬁrst approach models an explicit distribution over actions in the output layer, e.g., via logistic regression (Chen et al., 2015; Oord et al., 2016a;b; Zheng et al., 2016; Eyjolfsdottir et al., 2017).

Original: The second approach uses deep neural nets to deﬁne a transformation from a simple distribution to one of interest (Goodfellow et al., 2014; Kingma & Welling, 2014; Rezende et al., 2014) and can more readily be extended to incorporate additional structure, such as a hierarchy of random variables (Ranganath et al., 2016) or dynamics (Johnson et al., 2016; Chung et al., 2015;
Revised: The second approach uses deep neural nets to deﬁne a transformation from a simple distribution to one of interest (Goodfellow et al., 2014; Kingma & Welling, 2014; Rezende et al., 2014) and can more readily be extended to incorporate additional structure, such as a hierarchy of random variables (Ranganath et al., 2016) or dynamics (Johnson et al., 2016; Chung et al., 2015;

Original: Krishnan et al., 2017; Fraccaro et al., 2016).
Revised: Krishnan et al., 2017; Fraccaro et al., 2016).

Original: Our framework can incorporate both variants.
Revised: Our framework can incorporate both variants.

Original: Structured probabilistic models.
Revised: Structured probabilistic models.

Original: Recently, there has been increasing interest in probabilistic modeling with additional structure or side information.
Revised: Recently, there has been increasing interest in probabilistic modeling with additional structure or side information.

Original: Existing work includes approaches that enforce logic constraints (Akkaya et al., 2016), specify generative models as programs (Tran et al., 2016), or automatically produce weak supervision via data programming (Ratner et al., 2016).
Revised: Existing work includes approaches that enforce logic constraints (Akkaya et al., 2016), specify generative models as programs (Tran et al., 2016), or automatically produce weak supervision via data programming (Ratner et al., 2016).

Original: Our framework is inspired by the latter, which we extend to the spatiotemporal regime.
Revised: Our framework is inspired by the latter, which we extend to the spatiotemporal regime.

Original: Imitation Learning.
Revised: Imitation Learning.

Original: Our work is also related to imitation learning, which aims to learn a policy that can mimic demonstrated behavior (Syed & Schapire, 2008; Abbeel & Ng, 2004; Ziebart et al., 2008; Ho & Ermon, 2016).
Revised: Our work is also related to imitation learning, which aims to learn a policy that can mimic demonstrated behavior (Syed & Schapire, 2008; Abbeel & Ng, 2004; Ziebart et al., 2008; Ho & Ermon, 2016).

Original: There has been some prior work in multi-agent imitation learning (Le et al., 2017; Song et al., 2018) and learning stochastic policies (Ho & Ermon, 2016; Li et al., 2017), but no previous work has focused on learning generative polices while simultaneously addressing generative and multi-agent imitation learning.
Revised: There has been some prior work in multi-agent imitation learning (Le et al., 2017; Song et al., 2018) and learning stochastic policies (Ho & Ermon, 2016; Li et al., 2017), but no previous work has focused on learning generative polices while simultaneously addressing generative and multi-agent imitation learning.

Original: For instance, experiments in (Ho & Ermon, 2016) all lead to highly peaked distributions, while (Li et al., 2017) captures multimodal distributions by learning unimodal policies for a ﬁxed number of experts.
Revised: For instance, experiments in (Ho & Ermon, 2016) all lead to highly peaked distributions, while (Li et al., 2017) captures multimodal distributions by learning unimodal policies for a ﬁxed number of experts.

Original: (Hrolenok et al., 2017)also raise the issue of learning stochastic multi-agent behavior, but their solution relies on signiﬁcant feature engineering.
Revised: (Hrolenok et al., 2017) raise the issue of learning stochastic multi-agent behavior, but their solution involves signiﬁcant feature engineering.

Original:  B ACKGROUND : S EQUENTIAL G ENERATIVE M ODELING
Revised:  B ACKGROUND : S EQUENTIAL G ENERATIVE M ODELING

Original: Let x t ∈ R d denote the state at time t and x ≤ T = { x 1 , . . .
Revised: Let x t ∈ R d denote the state at time t and x ≤ T = { x 1 , . . .

Original: , x T } denote a sequence of states of length T .
Revised: , x T } denote a sequence of states of length T .

Original: Suppose we have a collection of N demonstrations D = { x ≤ T } .
Revised: Suppose we have a collection of N demonstrations D = { x ≤ T } .

Original: In our experiments, all sequences have the same length T , but in general this does not need to be the case.
Revised: In our experiments, all sequences have the same length T , but in general this does not need to be the case.

Original: The goal of sequential generative modeling is to learn the distribution over sequential data D .
Revised: The goal of sequential generative modeling is to learn the distribution over sequential data D .

Original: A common approach is to factorize the joint distribution and then maximize the log-likelihood:
Revised: A common approach is to factorize the joint distribution and then maximize the log-likelihood:

Original: [Equation]
Revised: [Equation]

Original: where θ are the learn-able parameters of the model, such as a recurrent neural network (RNN).
Revised: where θ are the learn-able parameters of the model, such as a recurrent neural network (RNN).

Original: Stochastic latent variable models.
Revised: Stochastic latent variable models.

Original: However, RNNs with simple output distributions that optimize
Revised: However, RNNs with simple output distributions that optimize

Original: Eq.
Revised: Eq.

Original: (1) often struggle to capture highly variable and structured sequential data.
Revised: (1) often struggle to capture highly variable and structured sequential data.

Original: For example, an
Revised: For example, an

Original: RNN with Gaussian output distribution has difﬁculty learning the multimodal behavior of the green player moving to the top-left/bottom-left in Figure 1a.
Revised: RNN with Gaussian output distribution has difﬁculty learning the multimodal behavior of the green player moving to the top-left/bottom-left in Figure 1a.

Original: Recent work in sequential generative models address this issue by injecting stochastic latent variables into the model and optimizing using amortized variational inference to learn the latent variables (Fraccaro et al., 2016; Goyal et al., 2017).
Revised: Recent work in sequential generative models address this issue by injecting stochastic latent variables into the model and optimizing using amortized variational inference to learn the latent variables (Fraccaro et al., 2016; Goyal et al., 2017).

Original: In particular, we use a variational RNN (VRNN, (Chung et al., 2015)) as our base model (Figure 3a), but we emphasize that our approach is compatible with other sequential generative models as well.
Revised: In particular, we use a variational RNN (VRNN (Chung et al., 2015)) as our base model (Figure 3a), but we emphasize that our approach is compatible with other sequential generative models as well.

Original: A VRNN is essentially a variational autoencoder (VAE) conditioned on the hidden state of an RNN and is trained by maximizing the (sequential) evidence lower-bound (ELBO):
Revised: A VRNN is essentially a variational autoencoder (VAE) conditioned on the hidden state of an RNN and is trained by maximizing the (sequential) evidence lower-bound (ELBO):

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (2) is a lower-bound of the log-likelihood in Eq.
Revised: (2) is a lower-bound of the log-likelihood in Eq.

Original: (1) and can be interpreted as the VAE ELBO summed over each timestep t .
Revised: (1) and can be interpreted as the VAE ELBO summed over each timestep t .

Original: We refer to appendix A for more details of VAEs and VRNNs.
Revised: We refer to appendix A for more details of VAEs and VRNNs.

Original:  H IERARCHICAL F RAMEWORK USING M ACRO - INTENTS
Revised:  H IERARCHICAL F RAMEWORK USING M ACRO - INTENTS

Original: In our problem setting, we assume that each sequence x ≤ T consists of the trajectories of K coordinating agents.
Revised: In our problem setting, we assume that each sequence x ≤ T consists of the trajectories of K coordinating agents.

Original: That is, we can decompose each x ≤ T into K trajectories: x ≤ T = { x 1 ≤ T , . . .
Revised: That is, we can decompose each x ≤ T into K trajectories: x ≤ T = { x 1 ≤ T , . . .

Original: , x K ≤ T } .
Revised: , x K ≤ T } .

Original: For example, the sequence in Figure 1a can be decomposed into the trajectories of K = 5 basketball players.
Revised: For example, the sequence in Figure 1a can be decomposed into the trajectories of K = 5 basketball players.

Original: Assuming conditional independence between the agent states x kt given state history x <t , we can factorize the maximum log-likelihood objective in Eq.
Revised: Assuming conditional independence between the agent states x kt given state history x <t , we can factorize the maximum log-likelihood objective in Eq.

Original: (1) even further:
Revised: (1) even further:

Original: [Equation]
Revised: [Equation]

Original: Naturally, there are two baseline approaches in this setting:
Revised: Naturally, there are two baseline approaches in this setting:

Original: Treat the data as a single-agent trajectory and train a single model: θ = θ 1 = · · · = θ K . 2.
Revised: Treat the data as a single-agent trajectory and train a single model: θ = θ 1 = · · · = θ K . 2.

Original: Train independent models for each agent: θ = { θ 1 , . .
Revised: Train independent models for each agent: θ = { θ 1 , . .

Original: , θ K } .
Revised: , θ K } .

Original: As we empirically verify in Section 5,RNN and VRNN models using these two approaches have difﬁculty learning representations of the data that generalize well over long time horizons and capturing the coordinationthat is inherent within the multi-agent trajectories.
Revised: As we empirically verify in Section 5, VRNN models using these two approaches have difﬁculty learning representations of the data that generalize well over long time horizons, and capturing the coordination inherent in multi-agent trajectories.

Original: Our solution introduces a hierarchical structure of macro-intents  to effectively learn low-dimensional (distributional) representations of the data that extend in both time and space for multiple coordinating agents.
Revised: Our solution introduces a hierarchical structure of macro-intents obtained via labeling functions to effectively learn low-dimensional (distributional) representations of the data that extend in both time and space for multiple coordinating agents.

Original: Deﬁning macro-intents.
Revised: Deﬁning macro-intents.

Original: We assume there exists shared latent variables called macro-intents that: 1) provide a tractable way to capture coordination between agents; 2) encode long-term intents of agents and enable long-term planning at a higher-level timescale; and 3) compactly represent some low-dimensional structure in an exponentially large multi-agent state space.
Revised: We assume there exists shared latent variables called macro-intents that: 1) provide a tractable way to capture coordination between agents; 2) encode long-term intents of agents and enable long-term planning at a higher-level timescale; and 3) compactly represent some low-dimensional structure in an exponentially large multi-agent state space.

Original: For example, Figure 2 illustrates macro-intents for two basketball players as speciﬁc areas on the court (boxes).
Revised: For example, Figure 2 illustrates macro-intents for two basketball players as speciﬁc areas on the court (boxes).

Original: Upon reaching its macro-intent in the top-right, the blue player moves towards its next macro-intent in the bottom-left.
Revised: Upon reaching its macro-intent in the top-right, the blue player moves towards its next macro-intent in the bottom-left.

Original: Similarly, the green player moves towards its macro-intents from bottom-right to middle-left.
Revised: Similarly, the green player moves towards its macro-intents from bottom-right to middle-left.

Original: These macro-intents are visible to both players and capture the coordination as they describe how the players plan to position themselves on the court.
Revised: These macro-intents are visible to both players and capture the coordination as they describe how the players plan to position themselves on the court.

Original: Macro-intents provide a compact summary of the players’ trajectories over a long time.
Revised: Macro-intents provide a compact summary of the players’ trajectories over a long time.

Original: Macro-intents do not need to have a geometric interpretation.
Revised: Macro-intents do not need to have a geometric interpretation.

Original: For example, macro-intents in the Boids model in Figure 1b can be a binary label indicating friendly vs. unfriendly behavior.
Revised: For example, macro-intents in the Boids model in Figure 1b can be a binary label indicating friendly vs. unfriendly behavior.

Original: The goal is for macro-intents to encode long-term intent and ensure that agents behave more cohesively.
Revised: The goal is for macro-intents to encode long-term intent and ensure that agents behave more cohesively.

Original: Our modeling assumptions for macro-intents are:
Revised: Our modeling assumptions for macro-intents are:

Original: • agent states { x kt } in an episode [ t 1 , t 2 ] are conditioned on some shared macro-intent g t , • the start and end times [ t 1 , t 2 ] of episodes can vary between trajectories, • macro-intents change slowly over time relative to the agent states: d g t /dt (cid:28) 1 , • and due to their reduced dimensionality, we can model (near-)arbitrary dependencies between macro-intents (e
Revised: • agent states { x kt } in an episode [ t 1 , t 2 ] are conditioned on some shared macro-intent g t , • the start and end times [ t 1 , t 2 ] of episodes can vary between trajectories, • macro-intents change slowly over time relative to the agent states: d g t /dt (cid:28) 1 , • and due to their reduced dimensionality, we can model (near-)arbitrary dependencies between macro-intents (e.g., coordination) via black box learning.

Original: g., coordination) via black box learning.
Revised: Labeling functions for macro-intents. Obtaining macro-intent labels from experts for training is ideal, but often too expensive. Instead, our work is inspired by recent advances in weak supervision settings known as data programming , in which multiple weak and noisy label sources called labeling functions can be leveraged to learn the underlying structure of large unlabeled datasets (Ratner et al., 2018; Bach et al., 2017). These labeling functions often compute heuristics that allow users to incorporate domain knowledge into the model.

Original: [Figure]
Revised: For instance, the labeling function we use to obtain macro-intents for basketball trajectories computes the regions on the court in which players remain stationary; this integrates the idea that players aim to set up speciﬁc formations on the court. In general, labeling functions are simple scripts/programs that can parse and label data very quickly, hence the name programmatic weak supervision . Other approaches that try to learn macro-intents in a fully unsupervised learning setting can encounter difﬁculties that have been previously noted, such as the importance of choosing the correct prior and approximate posterior (Rezende & Mohamed, 2015) and the interpretability of learned latent variables (Chen et al., 2017). We ﬁnd our approach using labeling functions to be much more attractive, as it outperforms other baselines by generating samples of higher quality, while also avoiding the engineering required to address the aforementioned difﬁculties.

Original: Hierarchical model with macro-intents Our hierarchical model uses an intermediate layer to model macro-intent, so our agent VRNN-models becomes:
Revised: Hierarchical model with macro-intents Our hierarchical model uses an intermediate layer to model macro-intent, so our agent VRNN-models becomes:

Original: [Equation]
Revised: [Equation]

Original: where ϕ k maps to a distribution over states, z kt is the VRNN latent variable, h kt is the hidden state of an RNN that summarizes the trajectory up to time t , and g t is the shared macro-intent at time t .
Revised: where ϕ k maps to a distribution over states, z kt is the VRNN latent variable, h kt is the hidden state of an RNN that summarizes the trajectory up to time t , and g t is the shared macro-intent at time t .

Original: Figure 3b shows our hierarchical model, which samples macro-intents during generation rather than using only ground-truth macro-intents.
Revised: Figure 3b shows our hierarchical model, which samples macro-intents during generation rather than using only ground-truth macro-intents.

Original: Here, we train an RNN-model to sample macro-intents:
Revised: Here, we train an RNN-model to sample macro-intents:

Original: [Equation]
Revised: [Equation]

Original:  where ϕ g maps to a distribution over macro-intents and h g,t − 1 summarizes the history of macro-intents up to time t .
Revised: 5) where ϕ g maps to a distribution over macro-intents and h g,t − 1 summarizes the history of macrointents up to time t .

Original: We condition the macro-intent model on previous states x t − 1 in Eq.
Revised: We condition the macro-intent model on previous states x t − 1 in Eq.

Original: (5) and generate next states by ﬁrst sampling a macro-intent g t , and then sampling x kt conditioned on g t (see Figure 3b).
Revised: (5) and generate next states by ﬁrst sampling a macro-intent g t , and then sampling x k t conditioned on g t (see Figure 3b).

Original: Note that all agent-models for generating x kt share the same macro-intent variable g t .
Revised: Note that all agent-models for generating x kt share the same macro-intent variable g t .

Original: This is core to our approach as it induces coordination betweengenerated agent trajectories (see Section 5).
Revised: This is core to our approach as it induces coordination between agent trajectories (see Section 5).

Original: We learn our agent-models by maximizing the VRNN objective from Eq (2) conditioned on the shared g t variables while independently learning the macro-intent model via supervised learning by maximizing the log-likelihood of macro-intent labels. Obtaining macro-intents labels.
Revised: We learn our agent-models by maximizing the VRNN objective from Eq (2) conditioned on the shared g t variables while independently learning the macro-intent model via supervised learning by maximizing the log-likelihood of macro-intent labels obtained programmatically.

Original:  One can obtain macro-intent labels for training in a variety of ways. Learning macro-intents via unsupervised learning is an end-to-end solution, but our experiments show that it rarely results in meaningful and interpretable macro-intent variables (see experiments with mutual information). Obtaining expert labels is also ideal, but often too expensive. Instead, our work in inspired by recent advances in weak supervision settings known as data programming , in which multiple weak and noisy label sources called “labeling functions” can be leveraged to learn the underlying structure of large unlabeled datasets (Ratner et al., 2018; Bach et al., 2017).
Revised:  E XPERIMENTS

Original: These labeling functions are simple heuristics that can be computed quickly and allow users to incorporate domain knowledge into the model. For instance, the labeling function we use to obtain macro-intents for basketball takes a trajectory and computes the areas on the court in which the players are stationary. This integrates the idea that players aim to set up speciﬁc formations.
Revised: [Figure]

Original: Other approaches that operate in a fully unsupervised learning setting can encounter difﬁculties that have been previously noted, such as the importance of choosing the correct prior and approximate posterior (Rezende & Mohamed, 2015) and the interpretability of learned latent variables (Chen et al., 2016). We ﬁnd our approach using programmatic weak supervision with labeling functions to bemuch more attractive, as it outperforms other baselines by generating samples of higher quality, while also avoiding the engineering required to address the aforementioned difﬁculties.  E XPERIMENTS
Revised: be

Original: We ﬁrst apply our approach on generating offensive team basketball gameplay (team with possession of the ball), and then on a synthetic Boids model dataset.
Revised: We ﬁrst apply our approach on generating offensive team basketball gameplay (team with possession of the ball), and then on a synthetic Boids model dataset.

Original: We present both quantitative and qualitative experimental results.
Revised: We present both quantitative and qualitative experimental results.

Original: Our quantitative results include a user study comparison with professional sports analysts, who signiﬁcantly preferred basketball rollouts generated from our approach to standard baselines.
Revised: Our quantitative results include a user study comparison with professional sports analysts, who signiﬁcantly preferred basketball rollouts generated from our approach to standard baselines.

Original: Examples from the user study and videos of  rollouts are in the supplementary material.
Revised: Examples from the user study and videos of generated rollouts can be seen in our demo video.

Original: 1 Our qualitative results demonstrate the ability of our approach to generate high-quality rollouts under various conditions.
Revised: Our qualitative results demonstrate the ability of our approach to generate high-quality rollouts under various conditions.

Original: An interactive demo is available at http://basketball-ai.com/ .

Original:  E XPERIMENTAL S ETUP FOR B ASKETBALL
Revised: 5. E XPERIMENTAL S ETUP FOR B ASKETBALL

Original: Training data.
Revised: Training data.

Original: Each demonstration in our data contains trajectories of K = 5 players on the left half-court, recorded for T = 50 timesteps at 6 Hz.
Revised: Each demonstration in our data contains trajectories of K = 5 players on the left half-court, recorded for T = 50 timesteps at 6 Hz.

Original: The offensive team has possession of the ball for the entire sequence.
Revised: The offensive team has possession of the ball for the entire sequence.

Original: x kt are the coordinates of player k
Revised: x kt are the coordinates of player k

Original: at time t on the court ( 50 × 94 feet).
Revised: at time t on the court ( 50 × 94 feet).

Original: We normalize and mean-shift the data.
Revised: We normalize and mean-shift the data.

Original: Players are ordered based on their relative positions, similar to the role assignment in (Lucey et al., 2013).
Revised: Players are ordered based on their relative positions, similar to the role assignment in (Lucey et al., 2013).

Original: There are 107,146 training and 13,845 test examples.
Revised: There are 107,146 training and 13,845 test examples.

Original: We ignore the defensive players and the ball to focus on capturing the coordination and multimodality of the offensive team.
Revised: We ignore the defensive players and the ball to focus on capturing the coordination and multimodality of the offensive team.

Original: In principle, we can provide the defensive positions as conditional input for our model and update the defensive positions using methods such as (Le et al., 2017).
Revised: In principle, we can provide the defensive positions as conditional input for our model and update the defensive positions using methods such as (Le et al., 2017).

Original: We leave the task of modeling the ball and defense for future work.
Revised: We leave the task of modeling the ball and defense for future work.

Original: Macro-intent labeling function.
Revised: Macro-intent labeling function.

Original: We extract weak macro-intent labels ˆ g kt for each player k as done in (Zheng et al., 2016).
Revised: We extract weak macro-intent labels ˆ g kt for each player k as done in (Zheng et al., 2016).

Original: We segment the left half-court into a 10 × 9 grid of 5 ft × 5 ft boxes.
Revised: We segment the left half-court into a 10 × 9 grid of 5 ft × 5 ft boxes.

Original: The weak macro-intent ˆ g kt at time t is a 1-hot encoding of dimension 90 of the next box in which player k is stationary (speed (cid:107) x
Revised: The weak macro-intent ˆ g kt at time t is a 1-hot encoding of dimension 90 of the next box in which player k is stationary (speed (cid:107) x

Original: kt +1 − x kt (cid:107) 2 below a threshold).
Revised: kt +1 − x kt (cid:107) 2 below a threshold).

Original: The shared global macro-intent g t is the concatenation of individual macro-intents.
Revised: The shared global macro-intent g t is the concatenation of individual macro-intents.

Original: Figure 4 shows the distribution ofextracted macro-intents for each player and pseudocode can be found in appendix C.
Revised: Figure 4 shows the distribution of macro-intents for each player. We refer to this labeling function as LF-stationary (pseudocode in appendix D).

Original: Model details.
Revised: Model details.

Original: We model each latent variable z kt as a multivariate Gaussian with diagonal covariance of dimension 16.
Revised: We model each latent variable z kt as a multivariate Gaussian with diagonal covariance of dimension 16.

Original: All output models are implemented with memory-less 2-layer fully-connected neural networks with a hidden layer of size 200.
Revised: All output models are implemented with memory-less 2-layer fully-connected neural networks with a hidden layer of size 200.

Original: Our agent-models sample from a multivariate Gaussian with diagonal covariance while our macro-intent models sample from a multinomial distribution over the macro-intents.
Revised: Our agent-models sample from a multivariate Gaussian with diagonal covariance while our macro-intent models sample from a multinomial distribution over the macro-intents.

Original: All hidden states ( h g,t , h 1 t , . . .
Revised: All hidden states ( h g,t , h 1 t , . . .

Original: h K t ) are modeled with 200 2-layer GRU memory cells each.
Revised: h Kt ) are modeled with 200 2-layer GRU memory cells each.

Original: We maximize the log-likelihood/ELBO with stochastic gradient descent using the
Revised: We maximize the log-likelihood/ELBO with stochastic gradient descent using the

Original: Adam optimizer (Kingma & Ba, 2014) and a learning rate of 0.0001. 
Revised: Adam optimizer (Kingma & Ba, 2015) and a learning rate of 0.0001. Baselines.

Revised: We compare with 5 baselines that do not use macro-intents from labeling functions:

Original: [Table]
Revised: [Table]

Original: Our hierarchical model achieves higher log
Revised: Our hierarchical model achieves higher loglikelihoods than baselines for both

Original: [Table]
Revised: [Table]

Original: Win/Tie/Loss indicates how often our model is preferred over baselines (25 comparisons per baseline).
Revised: Win/Tie/Loss indicates how often our model is preferred over baselines (25 comparisons per baseline).

Original: Gain is computed by scoring  our model is preferred and 
Revised: Gain is computed by scoring +1 when our model is preferred and -1 otherwise.

Original: Results are 98% signiﬁcant using a one-sample t-test
Revised: Results are 98% signiﬁcant using a one-sample t-test.

Original:  which we concatenate all player positions together ( K = 1 ) with 900 2-layer GRU cells for the hidden state and a 80-dimensional latent variable.
Revised: VRNN-single: VRNN in which we concatenate all player positions together ( K = 1 ) with 900 2-layer GRU cells for the hidden state and a 80-dimensional latent variable.

Original: VRNN-indep: VRNN for each agent with 250 2-layer GRUs and 16-dim latent variables.
Revised: VRNN-indep: VRNN for each agent with 250 2-layer GRUs and 16-dim latent variables.

Revised: VRNN-mixed: Combination of VRNN-single and VRNN-indep.

Revised: Shared hidden state of 600 2-layer GRUs is fed into decoders with 16-dim latent variables for each agent.

Original: VRAE-mi: VRAE-style architecturefrom (Fabius & van Amersfoort) and we maximize the mutual information between x ≤ T and macro-intent.
Revised: VRAE-mi: VRAE-style architecture (Fabius & van Amersfoort, 2014) that maximizes the mutual information between x ≤ T and macro-intent.

Original: We refer to appendix B for details.
Revised: We refer to appendix C for details.

Original:  Q UANTITATIVE E VALUATION FOR B ASKETBALL
Revised: 5. Q UANTITATIVE E VALUATION FOR B ASKETBALL

Original: Log-likelihood.
Revised: Log-likelihood.

Original: Table 1 reports the average log-likelihoods on the test data.
Revised: Table 1 reports the average log-likelihoods on the test data.

Original: Our approach outperforms RNN-gauss andVRNN-single and is comparable with VRNN-indep.
Revised: Our approach outperforms RNN-gauss and is comparable with other baselines.

Original: However, higher log-likelihoods do not necessarily indicate higher quality of generated samples (Theis et al., 2015).
Revised: However, higher log-likelihoods do not necessarily indicate higher quality of generated samples (Theis et al., 2015).

Original: As such, we also assess using other means, such as human preference studies and auxiliary statistics.
Revised: As such, we also assess using other means, such as human preference studies and auxiliary statistics.

Original: Human preference study.
Revised: Human preference study.

Original: We recruited 14 professional sports analysts as judges to compare the quality of rollouts.
Revised: We recruited 14 professional sports analysts as judges to compare the quality of rollouts.

Original: Each comparison animates two rollouts, one from our model and another from a baseline.
Revised: Each comparison animates two rollouts, one from our model and another from a baseline.

Original: Both rollouts are burned-in for 10 timesteps with the same ground-truth states from the test set, and then generated for the next 40 timesteps.
Revised: Both rollouts are burned-in for 10 timesteps with the same ground-truth states from the test set, and then generated for the next 40 timesteps.

Original: Judges decide which of the two rollouts looks more realistic.
Revised: Judges decide which of the two rollouts looks more realistic.

Original: Table 2 shows the results from the preference study.
Revised: Table 2 shows the results from the preference study.

Original: We tested our model against two baselines, VRNN-single and VRNN-indep, with 25 comparisons for each.
Revised: We tested our model against two baselines, VRNN-single and VRNN-indep, with 25 comparisons for each.

Original: All judges preferred our model over the baselines with 98% statistical signiﬁcance.
Revised: All judges preferred our model over the baselines with 98% statistical signiﬁcance.

Original: These results suggest that our model generates rollouts of signiﬁcantly higher quality than the baselines.
Revised: These results suggest that our model generates rollouts of signiﬁcantly higher quality than the baselines.

Original: Domain statistics.
Revised: Domain statistics.

Original: Finally, we compute several basketball statistics (average speed, average total distance traveled, % of frames with players out-of-bounds) and summarize them in Table 3.
Revised: Finally, we compute several basketball statistics (average speed, average total distance traveled, % of frames with players out-of-bounds) and summarize them in Table 3.

Original: Ourmodel generates trajectories that are most similar to ground-truth trajectories with respect to these statistics, indicating that our model generates signiﬁcantly more realistic behavior than all baselines.
Revised: Our

Original: [Table]
Revised: [Table]

Revised: Choice of labeling function.

Revised: In addition to LF-stationary , we also assess the quality of our approach using macro-intents obtained from different labeling functions.

Revised: LF-window25 and

Original:  Q UALITATIVE E VALUATION OF G ENERATED R OLLOUTS FOR B ASKETBALL
Revised: LF-window50 labels macro-intents as the last region a player resides in every window of 25 and 50 timesteps respectively (pseudocode in appendix D). Table 3 shows that domain statistics from our models using programmatic weak supervision match closer to the ground-truth with more informative labeling functions ( LF-stationary > LF-window25 > LF-window50 ). This is expected, since LF-stationary provides the most information about the structure of the data. 5. Q UALITATIVE E VALUATION OF G ENERATED R OLLOUTS FOR B ASKETBALL

Original: We next conduct a qualitative visual inspection of rollouts.
Revised: We next conduct a qualitative visual inspection of rollouts.

Original: Figure 5 shows rollouts generated from
Revised: Figure 5 shows rollouts generated from

Original: VRNN-single, VRNN-indep, and our model by sampling states for 40 timesteps after an initial burnin period of 10 timesteps with ground-truth states from the test set.
Revised: VRNN-single, VRNN-indep, and our model by sampling states for 40 timesteps after an initial burnin period of 10 timesteps with ground-truth states from the test set.

Original: An interactive demo to generate more rollouts from our hierarchical model can be found at: http://basketball-ai.com/ .
Revised: An interactive demo to generate more rollouts from our hierarchical model can be found at: http://basketball-ai.com/ .

Original: Common problems in baseline rollouts include players moving out of bounds or in the wrong direction (Figure 5a).
Revised: Common problems in baseline rollouts include players moving out of bounds or in the wrong direction (Figure 5a).

Original: These issues tend to occur at later timesteps, suggesting that the baselines do not perform well over long horizons.
Revised: These issues tend to occur at later timesteps, suggesting that the baselines do not perform well over long horizons.

Original: One possible explanation is due to compounding errors (Ross et al., 2011): if the model makes a mistake and deviates from the states seen during training, it is likely to make more mistakes in the future and generalize poorly.
Revised: One possible explanation is due to compounding errors (Ross et al., 2011): if the model makes a mistake and deviates from the states seen during training, it is likely to make more mistakes in the future and generalize poorly.

Original: On the other hand, generated rollouts from our model are more robust to the types of errors made by the baselines (Figure 5b).
Revised: On the other hand, generated rollouts from our model are more robust to the types of errors made by the baselines (Figure 5b).

Original: Macro-intents induce multimodal and interpretable rollouts.
Revised: Macro-intents induce multimodal and interpretable rollouts.

Original: Generated macro-intents allow us to intepret the intent of each individual player as well as a global team strategy (e.g. setting up a speciﬁc formation on the court).
Revised: Generated macro-intents allow us to intepret the intent of each individual player as well as a global team strategy (e.g. setting up a speciﬁc formation on the court).

Original: We highlight that our model learns a multimodal generating distribution, as repeated rollouts with the same burn-in result in a dynamic range of generated trajectories, as seen in Figure 6a Left.
Revised: We highlight that our model learns a multimodal generating distribution, as repeated rollouts with the same burn-in result in a dynamic range of generated trajectories, as seen in Figure 6a Left.

Original: Furthermore, Figure 6a Right demonstrates that grounding macro-intents during generation instead of sampling them allows us to control agent behavior.
Revised: Furthermore, Figure 6a Right demonstrates that grounding macro-intents during generation instead of sampling them allows us to control agent behavior.

Original: Macro-intents induce coordination.
Revised: Macro-intents induce coordination.

Original: Figure 6b illustrates how the macro-intents encode coordination between players that results in realistic rollouts of players moving cohesively.
Revised: Figure 6b illustrates how the macro-intents encode coordination between players that results in realistic rollouts of players moving cohesively.

Original: As we change the trajectory and macro-intent of the red player, the distribution of macro-intents generated from our model for the green player changes such that the two players occupy different areas of the court.
Revised: As we change the trajectory and macro-intent of the red player, the distribution of macro-intents generated from our model for the green player changes such that the two players occupy different areas of the court.

Original: 5. S YNTHETIC E XPERIMENTS : B OIDS M ODEL OF S CHOOLING B EHAVIOR
Revised: 5. S YNTHETIC E XPERIMENTS : B OIDS M ODEL OF S CHOOLING B EHAVIOR

Original: To illustrate the generality of our approach, we apply our model to a simpliﬁed version of the Boids model (Reynolds, 1987) that produces realistic trajectories of schooling behavior.
Revised: To illustrate the generality of our approach, we apply our model to a simpliﬁed version of the Boids model (Reynolds, 1987) that produces realistic trajectories of schooling behavior.

Original: We generate trajectories for 8 agents for 50 frames.
Revised: We generate trajectories for 8 agents for 50 frames.

Original: The agents start in ﬁxed positions around the origin with initial velocities sampled from a unit Gaussian.
Revised: The agents start in ﬁxed positions around the origin with initial velocities sampled from a unit Gaussian.

Original: Each agent’s velocity is then updated at each timestep:
Revised: Each agent’s velocity is then updated at each timestep:

Original: [Equation]
Revised: [Equation]

Original: average distance to an agent’s closest neighbor in

Original: Our model more closely captures the two distinct modes for friendly (small distances, left peak) vs. unfriendly (large distances, right peak) behavior compared to baselines, which do not learn to distinguish them.

Original: Full details of the model can be found in Appendix D. We randomly sample the sign of c 1 for each trajectory, which produces two distinct types of behaviors: friendly agents ( c 1 > 0 ) that like to group together, and unfriendly agents ( c 1 < 0 ) that like to stay apart (see Figure 1b).
Revised: Full details of the model can be found in Appendix B. We randomly sample the sign of c 1 for each trajectory, which produces two distinct types of behaviors: friendly agents ( c 1 > 0 ) that like to group together, and unfriendly agents ( c 1 < 0 ) that like to stay apart (see Figure 1b).

Original: We also introduce more stochasticity into the model by periodically updating β randomly.
Revised: We also introduce more stochasticity into the model by periodically updating β randomly.

Original: Our labeling function thresholds the average distance to an agent’s closest neighbor (see last plot in
Revised: Our labeling function thresholds the average distance to an agent’s closest neighbor (see last plot in

Original: Figure 7).
Revised: Figure 7).

Original: This is equivalent to using the sign of c 1 as our macro-intents, which indicates the type of behavior.
Revised: This is equivalent to using the sign of c 1 as our macro-intents, which indicates the type of behavior.

Original: Note that unlike our macro-intents for the basketball dataset, these macro-intents are simpler and have no geometric interpretation.
Revised: Note that unlike our macro-intents for the basketball dataset, these macro-intents are simpler and have no geometric interpretation.

Original: All models have similar average log-likelihoods on the test set in Table 1, but our hierarchical model can capture the true generating distribution much better than the baselines.
Revised: All models have similar average log-likelihoods on the test set in Table 1, but our hierarchical model can capture the true generating distribution much better than the baselines.

Original: For example, Figure 7 depicts the histograms of average distances to an agent’s closest neighbor in trajectories generated from all models and the ground-truth.
Revised: For example, Figure 7 depicts the histograms of average distances to an agent’s closest neighbor in trajectories generated from all models and the ground-truth.

Original: Our model more closely captures the two distinct modes in the ground-truth (friendly, small distances, left peak vs. unfriendly, large distances, right peak) whereas the baselines fail to distinguish them.
Revised: Our model more closely captures the two distinct modes in the ground-truth (friendly, small distances, left peak vs. unfriendly, large distances, right peak) whereas the baselines fail to distinguish them.

Original: 5. I NSPECTING THE H IERARCHICAL M ODEL C LASS
Revised: 5. I NSPECTING THE H IERARCHICAL M ODEL C LASS

Original: Output distribution for states.
Revised: Output distribution for states.

Original: The outputs of all models (including baselines) sample from a multivariate Gaussian with diagonal covariance.
Revised: The outputs of all models (including baselines) sample from a multivariate Gaussian with diagonal covariance.

Original: We also experimented with sampling from a mixture of 2 , 3 , 4 , and 8 Gaussian components, but discovered that the models would always learn to assign all the weight on a single component and ignore the others.
Revised: We also experimented with sampling from a mixture of 2 , 3 , 4 , and 8 Gaussian components, but discovered that the models would always learn to assign all the weight on a single component and ignore the others.

Original: The variance of the active component is also very small.
Revised: The variance of the active component is also very small.

Original: This is intuitive because sampling with a large variance at every timestep would result in noisy trajectories and not the smooth ones that we see in Figures 5, 6a.
Revised: This is intuitive because sampling with a large variance at every timestep would result in noisy trajectories and not the smooth ones that we see in Figures 5, 6a.

Original: Choice of macro-intent model.
Revised: Choice of macro-intent model.

Original: In principle, we can use more expressive generative models, like a VRNN, to model macro-intents over richer macro-intent spaces in Eq.
Revised: In principle, we can use more expressive generative models, like a VRNN, to model macro-intents over richer macro-intent spaces in Eq.

Original: In our case, we found that an RNN was sufﬁcient in capturing the distribution of macro-intents shown in Figure 4.
Revised: In our case, we found that an RNN was sufﬁcient in capturing the distribution of macro-intents shown in Figure 4.

Original: The
Revised: The

Original: RNN learns multinomial distributions over macro-intents that are peaked at a single macro-intent and relatively static through time, which is consistent with the macro-intent labels that we extracted from data.
Revised: RNN learns multinomial distributions over macro-intents that are peaked at a single macro-intent and relatively static through time, which is consistent with the macro-intent labels that we extracted from data.

Original: Latent variables in a VRNN had minimal effect on the multinomial distribution.
Revised: Latent variables in a VRNN had minimal effect on the multinomial distribution.

Original: Maximizing mutual information isn’t effective.
Revised: Maximizing mutual information isn’t effective.

Original: The learned macro-intents in our fully unsupervised VRAE-mi model do not encode anything useful and are essentially ignored by the model.
Revised: The learned macro-intents in our fully unsupervised VRAE-mi model do not encode anything useful and are essentially ignored by the model.

Original: In particular, the model learns to match the approximate posterior of macro-intents from the encoder with the discriminator from the mutual information lower-bound.
Revised: In particular, the model learns to match the approximate posterior of macro-intents from the encoder with the discriminator from the mutual information lower-bound.

Original: This results in a lack of diversityin rollouts as we vary the macro-intents during generation. We refer to appendix B for examples.
Revised: This results in a lack of diversity

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: The macro-intents labeling functions used in our experiments are relatively simple.
Revised: The macro-intents labeling functions used in our experiments are relatively simple.

Original: For instance, rather than simply using location-based macro-intents, we can also incorporate complex interactions such as “pick and roll”.
Revised: For instance, rather than simply using location-based macro-intents, we can also incorporate complex interactions such as “pick and roll”.

Original: Another future direction is to explore how to adapt our method to different domains, e.g., deﬁning a macro-intent representing “argument” for a dialogue between two agents, or a macro-intent representing “refrain” for music generation for “coordinating instruments” (Thickstun et al., 2017).
Revised: Another future direction is to explore how to adapt our method to different domains, e g., deﬁning a macro-intent representing “argument” for a dialogue between two agents, or a macro-intent representing “refrain” for music generation for “coordinating instruments” (Thickstun et al., 2017).

Original: We have shown that weak macro-intent labels extracted using simple domain-speciﬁc heuristics can be effectively used to generate high-quality coordinated multi-agent trajectories.
Revised: We have shown that weak macro-intent labels extracted using simple domain-speciﬁc heuristics can be effectively used to generate high-quality coordinated multi-agent trajectories.

Original: An interesting direction is to incorporate multiple labeling functions, each viewed as noisy realizations of true macro-intents, similar to (Ratner et al., 2016; 2018; Bach et al., 2017). 
Revised: An interesting direction is to incorporate multiple labeling functions, each viewed as noisy realizations of true macro-intents, similar to (Ratner et al., 2016; 2018; Bach et al., 2017). A CKNOWLEDGMENTS

Revised: This research is supported in part by NSF #1564330, NSF # and gifts from Bloomberg,

Revised: Activision/Blizzard and Northrop Grumman.

Revised: Dataset was provided by STATS:

Revised: https://www.

Original:  A S EQUENTIAL G ENERATIVE M ODELS
Revised: stats.com/data-science/ . A S EQUENTIAL G ENERATIVE M ODELS

Original: Recurrent neural networks.
Revised: Recurrent neural networks.

Original: A RNN models the conditional probabilities in Eq.
Revised: A RNN models the conditional probabilities in Eq.

Original: (1) with a hidden state h t that summarizes the information in the ﬁrst t − 1 timesteps:
Revised: (1) with a hidden state h t that summarizes the information in the ﬁrst t − 1 timesteps:

Original: [Equation]
Revised: [Equation]

Original: where ϕ maps the hidden state to a probability distribution over states and f is a deterministic function such as LSTMs (Hochreiter & Schmidhuber, 1997) or GRUs (Cho et al., 2014).
Revised: where ϕ maps the hidden state to a probability distribution over states and f is a deterministic function such as LSTMs (Hochreiter & Schmidhuber, 1997) or GRUs (Cho et al., 2014).

Original: RNNs with simple output distributions often struggle to capture highly variable and structured sequential data.
Revised: RNNs with simple output distributions often struggle to capture highly variable and structured sequential data.

Original: Recent work in sequential generative models address this issue by injecting stochastic latent variables into the model and using amortized variational inference to infer latent variables from data.
Revised: Recent work in sequential generative models address this issue by injecting stochastic latent variables into the model and using amortized variational inference to infer latent variables from data.

Original: Variational Autoencoders.
Revised: Variational Autoencoders.

Original: A variational autoencoder (VAE) (Kingma & Welling, 2014) is a generative model for non-sequential data that injects latent variables z into the joint distribution p θ ( x , z ) and introduces an inference network parametrized by φ to approximate the posterior q φ ( z | x ) .
Revised: A variational autoencoder (VAE) (Kingma & Welling, 2014) is a generative model for non-sequential data that injects latent variables z into the joint distribution p θ ( x , z ) and introduces an inference network parametrized by φ to approximate the posterior q φ ( z | x ) .

Original: The learning objective is to maximize the evidence lower-bound (ELBO) of the log-likelihood with respect to the model parameters θ and φ : 
Revised: The learning objective is to maximize the evidence lower-bound (ELBO) of the log-likelihood with respect to the model parameters θ and φ : [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is known as the reconstruction term and can be approximated with Monte Carlo sampling.
Revised: The ﬁrst term is known as the reconstruction term and can be approximated with Monte Carlo sampling.

Original: The second term is the Kullback-Leibler divergence between the approximate posterior and the prior, and can be evaluated analytically (i.e. if both distributions are Gaussian with diagonal covariance).
Revised: The second term is the Kullback-Leibler divergence between the approximate posterior and the prior, and can be evaluated analytically (i.e. if both distributions are Gaussian with diagonal covariance).

Original: The inference model q φ ( z | x ) , generative model p θ ( x | z ) , and prior p θ ( z ) are often implemented with neural networks.
Revised: The inference model q φ ( z | x ) , generative model p θ ( x | z ) , and prior p θ ( z ) are often implemented with neural networks.

Original: Variational RNNs.
Revised: Variational RNNs.

Original: VRNNs combine VAEs and RNNs by conditioning the VAE on a hidden state h t (see Figure 3a):
Revised: VRNNs combine VAEs and RNNs by conditioning the VAE on a hidden state h t (see Figure 3a):

Original: [Equation]
Revised: [Equation]

Original: VRNNs are also trained by maximizing the ELBO, which in this case can be interpreted as the sum of the ELBOs over each timestep of the sequence:
Revised: VRNNs are also trained by maximizing the ELBO, which in this case can be interpreted as the sum of VAE ELBOs over each timestep of the sequence:

Original: [Equation]
Revised: [Equation]

Original: Note that the prior distribution of latent variable z t depends on the history of states and latent variables (Eq.
Revised: Note that the prior distribution of latent variable z t depends on the history of states and latent variables (Eq.

Original: This temporal dependency of the prior allows VRNNs to model complex sequential data like speech and handwriting (Chung et al., 2015).
Revised: This temporal dependency of the prior allows VRNNs to model complex sequential data like speech and handwriting (Chung et al., 2015).

Original: B  M AXIMIZING M UTUAL I NFORMATION
Revised: B B OIDS M ODEL D ETAILS

Revised: We generate 32,768 training and 8,192 test trajectories. Each agent’s velocity is updated as: [Equation]

Revised: • v coh is the normalized cohesion vector towards an agent’s local neighborhood (radius 0.9) • v sep is the normalized vector away from an agent’s close neighborhood (radius 0.2) • v ali is the average velocity of other agents in a local neighborhood • v ori is the normalized vector towards the origin • ( c 1 , c 2 , c 3 , c 4 )

Revised: = ( ± 1 , 0 . 1 , 0 . 2 , 1) • β is sampled uniformly at random every 10 frames in range [0 . 8 , 1 . 4]

Revised: C M AXIMIZING M UTUAL I NFORMATION

Original: We ran experiments to see if we can learn meaningful macro-intents in a fully unsupervised fashion by maximizing the mutual information between macro-intent variables and trajectories x ≤ T .
Revised: We ran experiments to see if we can learn meaningful macro-intents in a fully unsupervised fashion by maximizing the mutual information between macro-intent variables and trajectories x ≤ T .

Original: We use a VAE-style model  in which we encode an entire trajectory into a latent macro-intent variable z , with the idea that z should encode global properties of the sequence.
Revised: We use a VRAE-style model from (Fabius & van Amersfoort, 2014) in which we encode an entire trajectory into a latent macro-intent variable z , with the idea that z should encode global properties of the sequence.

Original: The corresponding ELBO is:
Revised: The corresponding ELBO is:

Original: [Equation]
Revised: [Equation]

Original: where p θ ( z ) is the prior, q φ ( z | x ≤ T ) is the encoder, and p θ k ( x k t | x <t , z ) are the decoders.
Revised: where p θ ( z ) is the prior, q φ ( z | x ≤ T ) is the encoder, and p θ k ( x kt | x <t , z ) are decoders per agent.

Original: It is intractable to compute the mutual information between z and x ≤ T exactly, so we introduce a discriminator q ψ ( z | x ≤ T ) and use the following variational lower-bound of mutual information: 
Revised: It is intractable to compute the mutual information between z and x ≤ T exactly, so we introduce a discriminator q ψ ( z | x ≤ T ) and use the following variational lower-bound of mutual information: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We jointly maximize L 1 + λ L 2 wrt model parameters ( θ, φ, ψ ) , with λ = 1 in our experiments.
Revised: We jointly maximize L 1 + λ L 2 wrt. model parameters ( θ, φ, ψ ) , with λ = 1 in our experiments.

Original: Categorical vs. real-valued macro-intent z.
Revised: Categorical vs. real-valued macro-intent z.

Original: When we train an 8-dimensional categorical macrointent variable with a uniform prior (using gumbel-softmax trick (Jang et al., 2017)), the average distribution from the encoder matches the discriminator but not the prior (Figure 8) we train a 2-dimensional real-valued macro-intent variable with a standard Gaussian prior, the learned model generates trajectories with limited variability as we vary the macro-intent variable (Figure 9)
Revised: When we train an 8-dimensional categorical macrointent variable with a uniform prior (using gumbel-softmax trick (Jang et al., 2017)), the average distribution from the encoder matches the discriminator but not the prior (Figure 8). When we train a 2-dimensional real-valued macro-intent variable with a standard Gaussian prior, the learned model generates trajectories with limited variability as we vary the macro-intent variable (Figure 9).

Original: C M ACRO - INTENT L ABELING F UNCTION FOR  B ASKETBALL
Revised: D L ABELING F UNCTIONS FOR M ACRO - INTENTS IN B ASKETBALL

Original: Algorithm 1 outlines the macro-intent labeling function that we used to obtain weak macro-intent labels for basketball. We segment the left half-court into a 10 × 9 grid of 5 ft × 5 ft boxes (Figure 2). 
Revised: We deﬁne macro-intents in basketball by segmenting the left half-court into a 10 × 9 grid of 5 ft × 5 ft boxes (Figure 2). Algorithm 1 describes LF-window25 , which computes macro-intents based on last positions in 25-timestep windows ( LF-window50 is similar). Algorithm 2 describes LF-stationary , which computes macro-intents based on stationary positions.

Revised: For both,

Original: Court-Region computes the 1-hot encoding of the box that contains the position x kt .
Revised: Label-macro-intent ( x kt ) returns the 1-hot encoding of the box that contains the position x kt .

Original: Algorithm
Revised: 1: procedure LF- WINDOW 25( x ≤ T ) (cid:46) Trajectory x ≤ T of K players 2: macro-intents g ← initialize array of size ( K, T, 90) 3: for k = 1 . . . K do 4: g [ k, T ] ← L ABEL - MACRO - INTENT ( x kT ) (cid:46) Last timestep 5: for t = T − 1 . . .

Revised: 1 do 6: if (t+1)

Revised: mod 25 =

Original: 1: procedure COMPUTE{+= 
 then (cid:46) End of 25-timestep window 7: g [ k, t ] ← L ABEL+} - MACRO - INTENTS ( x ≤ T ) (cid:46) Trajectory x ≤ T of K players 2: macro-intents g ← initialize array of size ( K, T, 90) 3: for k = 1 . . .
Revised: {+= 
 then (cid:46) End of 25-timestep window 7: g [ k, t ] ← L ABEL+} - MACRO - INTENT ( x kt ) 8: else 9: g [ k, t ] ← g [ k, t + 1] 10: return g 1: procedure LF- STATIONARY ( x ≤ T ) (cid:46) Trajectory x ≤ T of K players 2: macro-intents g ← initialize array of size ( K, T, 90) 3: for k = 1 . . .

Original: K do 4: speed ← compute speeds of player k in x k ≤ T 5: stationary ← speed < threshold 6: g [ k, T ] ← COURT - REGION ( x kT ) (cid:46)
Revised: K do 4: speed ← compute speeds of player k in x k ≤ T 5: stationary ← speed < threshold 6: g [ k, T ] ← L ABEL - MACRO - INTENT ( x kT ) (cid:46)

Original: Last timestep 7: for t =
Revised: Last timestep 7: for t =

Original: T − 1 . . .
Revised: T − 1 . . .

Original: 1 do 8: if stationary[t] and not stationary[t+1] then (cid:46) Player k starts moving 9: g [ k, t ] ← COURT - REGION ( x kt ) 10: else (cid:46) Player  remains stationary 11: g [ k, t ] ← g [ k, t + 1] 12: return gD B OIDS M ODELS D ETAILS We generate 32,768 training and 8,192 test trajectories.
Revised: 1 do 8: if stationary[t] and not stationary[t+1] then (cid:46) Player k starts moving 9: g [ k, t ] ← L ABEL - MACRO - INTENT ( x kt ) 10: else (cid:46) Player k remains stationary 11: g [ k, t ] ← g [ k, t + 1] 12: return g

Original: Each agent’s velocity is updated as:

Original: [Equation]

Original: • v coh is the normalized cohesion vector towards an agent’s local neighborhood (radius 0.9) • v sep is the normalized vector away from an agent’s close neighborhood (radius 0.2) • v ali is the average velocity of other agents in a local neighborhood • v ori is the normalized vector towards the origin • ( c 1 , c 2 , c 3 , c 4 )

Original: = ( ± 1 , 0 . 1 , 0 . 2 , 1) • β is sampled uniformly at random every 10 frames from [0 . 8 , 1 . 4]