Paper:
STOCHASTIC PREDICTION OF MULTI-AGENT INTERACTIONS FROM PARTIAL OBSERVATIONS
We present a method that learns to integrate temporal information, from a learned dynamics model, with ambiguous visual information, from a learned vision model, in the context of interacting agents. Our method is based on a graph-structured variational recurrent neural network (Graph-VRNN), which is trained end-to-end to infer the current state of the (partially observed) world, as well as to forecast future states. We show that our method outperforms various baselines on two sports datasets, one based on real basketball trajectories, and one generated by a soccer game engine.

Review:
1) Summary
This paper presents a graph neural network based architecture that is trained to locate and model the interactions of agents in an environment directly from pixels. They propose an architecture that is a composition of recurrent neural networks where each models a single object independently and communicate with other for the overall environment modeling. The model is trained with a variational recurrent neural network objective that allows for stochasticity in the predictions while at the same time allows to model the current and future steps simultaneously. In experiments, they show the advantage of using the proposed model for tasks of tracking as well as forecasting of agents locations.
2) Pros:
+ Novel recurrent neural network architecture to model structured dynamics of agents in an environment.
+ Outperforms baseline methods.
+ New dataset for partially observable prediction research.
3) Cons:
Forecasting task:
- The authors argue that a discretization needs to be performed because of the many possible futures given the past, and also provide an error measure based on likelihood. However, if trajectories are actually generated from these distributions, I suspect the many possible futures generated will be very shaky. Can the authors provide trajectories sampled from this? If sampling trajectories does not make sense somehow, can the authors comment on how we can sample multiple trajectories?
Lack of baselines:
- The authors mention social LSTM and social GAN in the related work, however, no comparison is provided. From a quick glance, the authors of these papers work on trajectories. However, the “social” principle in those papers is general since it’s done from the computed feature vector. Could it have not been used on top of one of the baselines? If not, could the authors provide a reason why this is not the case?
Additional comments:
As the authors mention, it would be nice to extend this paper to an unsupervised or semi-supervised task. Here are a couple of papers that may interest you:
https://arxiv.org/abs/1804.04412
https://arxiv.org/abs/1705.02193
https://arxiv.org/abs/1806.07823
4) Conclusion
Overall, the paper is well written, easy to understand, and seems to be simple enough to quickly reproduce. Additionally, the proposed dataset may be of use for the community. If the authors are able to successfully address the issues mentioned, I am willing to improve my score.