Paper:
GENERATING MULTI-AGENT TRAJECTORIES USING PROGRAMMATIC WEAK SUPERVISION
We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as offensive basketball gameplay. When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables. Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulatable way. We present a hierarchical framework that can effectively learn such sequential generative models. Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime. In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods. We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.

Review:
This paper proposes training multiple generative models that share a common latent variable, which is learned in a weakly supervised fashion, to achieve high level coordination between multiple agents. Each agent has a separate VRNN model which is conditioned on the agent’s own trajectory history as well as the shared latent variable. The model is trained to maximize the ELBO objective and log-likelihood over macro-intent labels. Experimental results are conducted over a basketball gameplay dataset (to model the trajectories of the offensive team members) and a synthetic dataset. The results show that the proposed model is on-par with the baseline models in terms of ELBO while showing that it can model multi-modality better and is preferred more by humans. 
In general, the paper is well written and the overall framework captures the essence of the problem that the authors are trying to solve.
Furthermore, incorporating an auxiliary latent variable to model the coordination between multiple agents is interesting.
I have several comments related to the strength of the baselines and contribution of individual components in the proposed model.
Major Comments
- It seems that VRNN-single and VRNN-indep are two models on the far two ends of a spectrum. To understand the contribution of the shared macro-intent, how would an intermediate baseline model where a set of parameters are shared between agents and each agent also has an independent set of parameters perform? This could be accomplished by sharing the parameters of the first layer of GRU networks and learning the second layer parameters independently.
- How is the threshold for macro-intent generation selected? How does this parameter affect the overall performance? Since the smoothness of the segments between two macro-intents depend on this parameter, I am wondering its effect on the learned posterior distribution.
- Rather than using the prediction of the macro-intent RNN as a single global vector (\hat{g}_t), could using separate vectors for each agent (corresponding blocks of \hat{g}_t) as inputs to VRNN give the same results? Since the macro-intent RNN is already aware of all the macro-intents, it would be interesting to see if individual macro-intents are sufficient for VRNN to generate corresponding trajectories.
Minor Comments
- Do results in Table (1) come from sampling or using mode of the distributions? How peaked are the learned posterior distributions?
- What is the performance of the macro-intent RNN model?
- In Eq (2), “<=T” should be “<=t” (as in Eq (11) in Chung 2015).
- In Page 6, bullet point 4: it should be “except we maximize the mutual information…”