From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper9/Authors
title: authors' reply
comment: Thank you for the detailed evaluation and the positive assessment. We are happy to provide some clarifying answers:

* "multiple moving objects, moving closer to each other" 

Indeed, as we describe the architecture it is meant to model individual objects. It can easily be extend to multiple objects, though: the trajectories of all objects serve as additional input channels and for each object an output probability distribution is generated. Inbetween layers are shared. Because of the single time-step parametrization, the network is aware which objects are close to each other at any given time and can, e.g., prevent collisions. 
We indeed had already performed experiments on this setting, as the SDD dataset contains trajectories of multiple objects. Results were improved compared to the single object setting, but there was no space to describe the setting and include the results because of the page limit. 

* "interesting for the majority of the applications is the probability of trajectories - some trajectories are more probable than others"
    
This is exactly what the model does: it allows evaluating the probability of arbitrary trajectories of any length with a single network evaluation using the chain rule of probability (1). We consider this the main advantage over other approaches, e.g. VAE- or GAN-based, that allow sampling trajectories but not evaluating the likelihood function.


* "the probabilities should gradually drop as we try to predict more time steps into the future."

This is indeed the case: the probabilities of shorter trajectories and longer ones are connected via the marginalization rule of probability. In contrast to latent variable models this relation emerges automatically, no numeric integration is required. The result is that a short trajectory always has a higher probability than if it were extended, and by how much depends on how "deterministic" the continuation is. 

* "rigorous experiments and comparisons"

We agree, and we would like to emphasize that as one of our main contributions we show that commonly used evaluation measures (oracle accuracy, mean square distance) are unsuited for multi-modal trajectory forecasting and can easily be fooled. We propose the log-likelihood as a more principled alternative. Unfortunately, this means that results from the literature are largely not trustworthy and the methods need to be reimplemented and reevaluated.
We plan this for future work, but it would not have fit into a 4-page manuscript. 

* "architecture named spatiotemporal CNNs"

Indeed, there are multiple variants of spatio-temporal CNNs. The purpose of our work is not to claim novelty for spatio-temporal CNNs, but to show that already in their simplest form, if equipped with probabilistic outputs, they can perform trajectory estimation (at least) on par with much more complex models that had been proposed in the recent literature. 

* "the title"

We'll be happy to make it more descriptive. It was meant to emphasize the point of the previous answer: multi-modal trajectory estimation is possible with simpler models than what can be found in the literature. 

* "supplementary video showing predictions"

A very good idea. We will prepare such a video.

* recommended literature

Thank you for these. We'll include these.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper9/AnonReviewer2
evaluation: The paper presents a method for trajectory estimation. However, the paper is interesting because it provides uncertainties of predictions and can estimate multiple possible trajectories. To this end, a fully convolutional neural network (FCNN) is modified to take into account several consecutive frames, allowing the proposed architecture to capture both spatial and temporal relationships. Interestingly, the final layer is designed in a way to output probabilities. 

I highly appreciate the attempt to try novel methods to solve the problem of probabilistic multimodal trajectory estimation which is extremely important in a variety of application domains such as self-driving cars, cyclone prediction, etc. However, I am dubious if the proposed architecture would work well in the presence of multiple moving objects, moving closer to each other.  In general, what would be more interesting for the majority of the applications is the probability of trajectories-some trajectories are more probable than others, and the probabilities should gradually drop as we try to predict more time steps into the future.

There is numerous other work on video prediction, and hence more rigorous experiments and comparisons are expected. Also, already there is an architecture named spatiotemporal CNNs: "Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation," though less relevant to trajectory estimation

Personally, I don't think the title indicates much about the content of the paper. Also, a supplementary video showing predictions would undoubtedly strengthen this paper. 

    1. What did you learn from reading this paper that you did not know before?
A new spatiotemporal FCNN architecture.
    2. What was the most exciting part of the work?
Multimodal trajectory estimation.
    3. What in your opinion makes this work stand apart from other papers in this area?
Multimodal probabilistic trajectory estimation.
    4. Does the paper fit into the scope of the workshop?
Yes
    5. Are you aware of relevant literature that you may want to make the authors aware of?
- Context-Aware Pedestrian Motion Prediction In Urban Intersections
- Kernel Embeddings of Longitudinal Data
- Multi-Modal  Estimation  with  Kernel  Embeddings  for Learning  Motion  Models
- Directional grid maps: modeling multimodal angular uncertainty in dynamic environments
- Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper9/AnonReviewer2
relevance: 5: very high
impact: 3: borderline
novelty: 3: borderline