Original: Back to square one: probabilistic trajectory forecasting without bells and whistles
Revised: Back to square one: probabilistic trajectory forecasting without bells and whistles

Original: Abstract
Revised: Abstract

Original: We introduce a spatio-temporal convolutional neural network model for trajectory forecasting from visual sources.
Revised: We introduce a spatio-temporal convolutional neural network model for trajectory forecasting from visual sources.

Original: Applied in an auto-regressive way it provides an explicit probability distribution over continuations of a given initial trajectory segment.
Revised: Applied in an auto-regressive way it provides an explicit probability distribution over continuations of a given initial trajectory segment.

Original: We discuss it in relation to (more complicated) existing work and report on experiments on two standard datasets for trajectory forecasting: MNISTseq and
Revised: We discuss it in relation to (more complicated) existing work and report on experiments on two standard datasets for trajectory forecasting: MNISTseq and

Original: Stanford Drones , achieving results on-par with or better than previous methods.
Revised: Stanford Drones , achieving results on-par with or better than previous methods.

Original:  Introduction
Revised:  Introduction

Original: A crucial task for an autonomous system, such as a drone or a delivery robot, is to react to moving objects in their environment, either to avoid them or to interact with them.
Revised: A crucial task for an autonomous system, such as a drone or a delivery robot, is to react to moving objects in their environment, either to avoid them or to interact with them.

Original: In order to be able to plan such actions, the system needs the skill to observe other objects, e.g. with cameras, and to predict how they are likely to move, i.e. forecast their trajectories.
Revised: In order to be able to plan such actions, the system needs the skill to observe other objects, e.g. with cameras, and to predict how they are likely to move, i.e. forecast their trajectories.

Original: In this work, we study the corresponding spatio-temporal learning problem: trajectory forecasting from visual sources .
Revised: In this work, we study the corresponding spatio-temporal learning problem: trajectory forecasting from visual sources .

Original: Speciﬁcally, we aim at learning not just individual trajectories but probability distributions that reﬂect the intrinsic uncertainty of the task.
Revised: Speciﬁcally, we aim at learning not just individual trajectories but probability distributions that reﬂect the intrinsic uncertainty of the task.

Original: Existing models for this task typically make use of latent variables to achieve the necessary multi-modal output distributions.
Revised: Existing models for this task typically make use of latent variables to achieve the necessary multi-modal output distributions.

Original: Instead, we propose using a spatio-temporal convolutional network with a probabilistic output layer.
Revised: Instead, we propose using a spatio-temporal convolutional network with a probabilistic output layer.

Original: It is simpler and easier to train than previous models, yet our experiments show that it produces as good or even better forecasts.
Revised: It is simpler and easier to train than previous models, yet our experiments show that it produces as good or even better forecasts.

Original: We also discuss previously used evaluation measures for trajectory prediction, arguing that they are unsatisfactory, because they either penalize multi-modal outputs instead of encouraging them, or encourage random guessing.
Revised: We also discuss previously used evaluation measures for trajectory prediction, arguing that they are unsatisfactory, because they either penalize multi-modal outputs instead of encouraging them, or encourage random guessing.

Original: Instead, we propose to use a simple cross-entropy measure that is mathematically well-understood and well-deﬁned for all models that output discrete probability distributions.
Revised: Instead, we propose to use a simple cross-entropy measure that is mathematically well-understood and well-deﬁned for all models that output discrete probability distributions.

Original:  Probabilistic Trajectory Forecasting
Revised:  Probabilistic Trajectory Forecasting

Original: We formalize the task of probabilistic trajectory forecasting as follows: for any given initial trajectory segment of points, x 1: s = ( x 1 , . . .
Revised: We formalize the task of probabilistic trajectory forecasting as follows: for any given initial trajectory segment of points, x 1: s = ( x 1 , . . .

Original: , x s ) with x i ∈ R 2 for i = 1 , . . .
Revised: , x s ) with x i ∈ R 2 for i = 1 , . . .

Original: , s , predict a continuation of the trajectory for T more steps, x s +1:
Revised: , s , predict a continuation of the trajectory for T more steps, x s +1:

Original: s + T
Revised: s + T

Original: =
Revised: =

Original: ( x s +1 , . . .
Revised: ( x s +1 , . . .

Original: , x s + T ) , where T is a user-deﬁned horizon.
Revised: , x s + T ) , where T is a user-deﬁned horizon.

Original: ∗ Work done while at IST Austria.

Original: Reﬂecting that future trajectories are uncertain, we do not seek just a deterministic prediction but rather a probability distribution, p ( x s +1: s + T | x 1: s ) , i.e. the probability of any forecast, conditioned on the observed segment.
Revised: Reﬂecting that future trajectories are uncertain, we do not seek just a deterministic prediction but rather a probability distribution, p ( x s +1: s + T | x 1: s ) , i.e. the probability of any forecast, conditioned on the observed segment.

Original: Speciﬁcally for forecasts from visual sources, we assume that the positions correspond to pixel locations in a ﬁxed-size image domain.
Revised: Speciﬁcally for forecasts from visual sources, we assume that the positions correspond to pixel locations in a ﬁxed-size image domain.

Original: Optionally, a reference image, y , of the same resolution can be given to serve as additional input (conditioning) to the prediction task.
Revised: Optionally, a reference image, y , of the same resolution can be given to serve as additional input (conditioning) to the prediction task.

Original: As training data, a set of example trajectories is given, x (1) , . . .
Revised: As training data, a set of example trajectories is given, x (1) , . . .

Original: , x ( n ) , of potentially different lengths, i.e. x ( i ) =
Revised: , x ( n ) , of potentially different lengths, i.e. x ( i ) =

Original: x ( i ) 1: T i =
Revised: x ( i ) 1: T i =

Original: ( x ( i ) 1 , . . .
Revised: ( x ( i ) 1 , . . .

Original: , x ( i ) T i ) with x ( i ) j ∈ R 2 for any i = 1 , . . . , n and j = 1 , . . .
Revised: , x ( i ) T i ) with x ( i ) j ∈ R 2 for any i = 1 , . . . , n and j = 1 , . . .

Original: , T i .
Revised: , T i .

Original: When reference images are available, we are given one, y ( i ) , for each training trajectory, x ( i ) .
Revised: When reference images are available, we are given one, y ( i ) , for each training trajectory, x ( i ) .

Original: Different trajectories can have the same reference images, e.g. if there are multiple moving objects in a scene.
Revised: Different trajectories can have the same reference images, e.g. if there are multiple moving objects in a scene.

Original: A fully convolution spatio-temporal model for probabilitic trajectory forecasting.
Revised: A fully convolution spatio-temporal model for probabilitic trajectory forecasting.

Original: We propose a new model, called STCNN (for spatio-temporal convolutional neural network ), that combines simplicity, efﬁciency and expressiveness.
Revised: We propose a new model, called STCNN (for spatio-temporal convolutional neural network ), that combines simplicity, efﬁciency and expressiveness.

Original: It has a classical encoder–bottleneck–decoder architecture, see Figure 1, with only spatio-temporal convolutions and upconvolutions (i.e. transpose convolutions).
Revised: It has a classical encoder–bottleneck–decoder architecture, see Figure 1, with only spatio-temporal convolutions and upconvolutions (i.e. transpose convolutions).

Original: An initial trajectory segment is encoded visually as a sequence of images, which are processed using two-dimensional convolutional layers.
Revised: An initial trajectory segment is encoded visually as a sequence of images, which are processed using two-dimensional convolutional layers.

Original: The information of s time steps is aggregated through a three-dimensional convolution layer.
Revised: The information of s time steps is aggregated through a three-dimensional convolution layer.

Original: This is followed by further two-dimensional convolutional layers, the output of which we consider a latent representation of the input segment.
Revised: This is followed by further two-dimensional convolutional layers, the output of which we consider a latent representation of the input segment.

Original: From the representation, a discrete probability distribution is constructed by a sequence of upconvolutions and a softmax output layer.
Revised: From the representation, a discrete probability distribution is constructed by a sequence of upconvolutions and a softmax output layer.

Original: Details will be reported in a technical report, and we will also make our source code publicy available.
Revised: Details will be reported in a technical report, and we will also make our source code publicy available.

Original: For any trajectory segment ¯ x , the model output g (¯ x ) is a discrete probability distribution with as many entries as the image has pixels.
Revised: For any trajectory segment ¯ x , the model output g (¯ x ) is a discrete probability distribution with as many entries as the image has pixels.

Original: The probability that the next step of the trajectory is at a location x , we denote as g (¯ x )[ x ] .
Revised: The probability that the next step of the trajectory is at a location x , we denote as g (¯ x )[ x ] .

Original: From this one-step forecast model we obtain a closed-form expression for the probability of any trajectory given an initial segment by applying the chain rule of probability:
Revised: From this one-step forecast model we obtain a closed-form expression for the probability of any trajectory given an initial segment by applying the chain rule of probability:

Original: [Equation]
Revised: [Equation]

Original: A major advantage of modeling the probability distribution in the output layer, compared to, e.g., a latent variable model, is that one can train the model using ordinary maximum likelihood training:
Revised: A major advantage of modeling the probability distribution in the output layer, compared to, e.g., a latent variable model, is that one can train the model using ordinary maximum likelihood training:

Original: [Equation]
Revised: [Equation]

Original: where θ denotes all parameters of the network model g .
Revised: where θ denotes all parameters of the network model g .

Original: Note that at training time, all steps of the trajectories are observed, so all terms in (2) can be evaluated explicitly.
Revised: Note that at training time, all steps of the trajectories are observed, so all terms in (2) can be evaluated explicitly.

Original: In fact, by implementing the model in a fully-convolutional way with respect to the time dimension, all terms for one training example can be evaluated in parallel with a single network evaluation.
Revised: In fact, by implementing the model in a fully-convolutional way with respect to the time dimension, all terms for one training example can be evaluated in parallel with a single network evaluation.

Original: From the trained model, we can sample trajectory forecasts of arbitrary length in an auto-regressive way: for each t = 1 , . . .
Revised: From the trained model, we can sample trajectory forecasts of arbitrary length in an auto-regressive way: for each t = 1 , . . .

Original: , T , we sample a next location x
Revised: , T , we sample a next location x

Original: s + t ∼ g
Revised: s + t ∼ g

Original: ( x t : t + s − 1 ) , append it to the existing segment, and repeat the procedure at the next location.
Revised: ( x t : t + s − 1 ) , append it to the existing segment, and repeat the procedure at the next location.

Original: The encoder-decoder architecture also allows an easy integration of additional information, in particular about potential reference images.
Revised: The encoder-decoder architecture also allows an easy integration of additional information, in particular about potential reference images.

Original: Similar to [9], we use a standard convolutional network to extract a feature representation from the reference image.
Revised: Similar to [9], we use a standard convolutional network to extract a feature representation from the reference image.

Original: This is concatenated with the latent trajectory representation, such that the upconvolution operations can make use of the encoded information both from the trajectory and from the reference images.
Revised: This is concatenated with the latent trajectory representation, such that the upconvolution operations can make use of the encoded information both from the trajectory and from the reference images.

Original: Again, all terms are observed during training, so both components can be trained jointly and efﬁciently using likelihood maximization.
Revised: Again, all terms are observed during training, so both components can be trained jointly and efﬁciently using likelihood maximization.

Original: Comparison to models from the literature.
Revised: Comparison to models from the literature.

Original: Trajectory prediction is a classic computer vision topic, see [6] for a survey.
Revised: Trajectory prediction is a classic computer vision topic, see [6] for a survey.

Original: However, only recent methods are able to produce high-quality multi-modal probabilistic outputs.
Revised: However, only recent methods are able to produce high-quality multi-modal probabilistic outputs.

Original: Most recent works [1, 5] combine recurrent neural networks, in particular
Revised: Most recent works [1, 5] combine recurrent neural networks, in particular

Original: LSTMs [4], with conditional variational autoencoders (CVAE) [8].
Revised: LSTMs [4], with conditional variational autoencoders (CVAE) [8].

Original: The LSTMs act as deterministic encoder and decoder of trajectories into and from a latent Euclidean space Z .
Revised: The LSTMs act as deterministic encoder and decoder of trajectories into and from a latent Euclidean space Z .

Original: The CVAE models a distribution q ( z (cid:48) | z ) , where z, z ∈ Z are latent space encodings of the initial and the forecasted trajectory, respectively.
Revised: The CVAE models a distribution q ( z (cid:48) | z ) , where z, z ∈ Z are latent space encodings of the initial and the forecasted trajectory, respectively.

Original: The CVAEs distribution, q , itself is Gaussian with respect to z , with a neural network providing the mean and covariance as functions of z .
Revised: The CVAEs distribution, q , itself is Gaussian with respect to z , with a neural network providing the mean and covariance as functions of z .

Original: From the resulting model one can sample trajectory forecasts: one encodes the initial segment, samples latent states according to the CVAE distribution and decoding them.
Revised: From the resulting model one can sample trajectory forecasts: one encodes the initial segment, samples latent states according to the CVAE distribution and decoding them.

Original: One does not have, however, explicit access to the trajectory distribution, as this would require an intractable integration over a continuous latent space.
Revised: One does not have, however, explicit access to the trajectory distribution, as this would require an intractable integration over a continuous latent space.

Original: Therefore, one also cannot train the models using actual likelihood maximization.
Revised: Therefore, one also cannot train the models using actual likelihood maximization.

Original: Instead, [5] minimizes the least squares difference between sampled trajectories and the ones in the training set plus typical VAE-regularization terms.
Revised: Instead, [5] minimizes the least squares difference between sampled trajectories and the ones in the training set plus typical VAE-regularization terms.

Original: [1] ﬁrst constructs a sampling-based surrogate to the log-likelihood, which, however, is not an unbiased estimator, see [2].
Revised: [1] ﬁrst constructs a sampling-based surrogate to the log-likelihood, which, however, is not an unbiased estimator, see [2].

Original: The actual training objective is another approximation of this with an averaging step replaced by a maximization.
Revised: The actual training objective is another approximation of this with an averaging step replaced by a maximization.

Original: Evaluation measure.
Revised: Evaluation measure.

Original: A priori, it is not clear what is the best evaluation measure for probabilistic forecasting, where the goal is to predict a –potentially multi-modal– distribution over possible trajectories, yet a test set typically contains only individual realizations.
Revised: A priori, it is not clear what is the best evaluation measure for probabilistic forecasting, where the goal is to predict a –potentially multi-modal– distribution over possible trajectories, yet a test set typically contains only individual realizations.

Original: In this work, we argue in favor of a probabilistically justiﬁed approach also to the evaluation problem.
Revised: In this work, we argue in favor of a probabilistically justiﬁed approach also to the evaluation problem.

Original: One should use a measure that assigns small values if and only if exactly the (unknown) target distribution is predicted, and that allows a fair comparison between different methods and different parametrizations.
Revised: One should use a measure that assigns small values if and only if exactly the (unknown) target distribution is predicted, and that allows a fair comparison between different methods and different parametrizations.

Original: A measure that fulﬁlls these properties is the negative cross-entropy, D ( P ; Q ) =
Revised: A measure that fulﬁlls these properties is the negative cross-entropy, D ( P ; Q ) =

Original: − E x ∼ P log Q ( x ) , between the modeled distribution Q and the target distribution P of the test data.
Revised: − E x ∼ P log Q ( x ) , between the modeled distribution Q and the target distribution P of the test data.

Original: It can be estimated in an unbiased way from samples of the target distribution, as they are available to us in form of test data trajectories.
Revised: It can be estimated in an unbiased way from samples of the target distribution, as they are available to us in form of test data trajectories.

Original: Note that for this expression to be well-deﬁned, in particular non-negative and minimal for P = Q , the value Q ( x ) should be an actual probability value, not the evaluation of a continuous density functional.
Revised: Note that for this expression to be well-deﬁned, in particular non-negative and minimal for P = Q , the value Q ( x ) should be an actual probability value, not the evaluation of a continuous density functional.

Original: This is indeed the case for the discrete model we propose, while for
Revised: This is indeed the case for the discrete model we propose, while for

Original: CVAE-based models, an additional discretization or binning step might be required.
Revised: CVAE-based models, an additional discretization or binning step might be required.

Original: Evaluation measures in the literature.
Revised: Evaluation measures in the literature.

Original: Unfortunately, prior work relied on other evaluation measures: [5] reports several different measures, based either on the L 2 distance between trajectories at different time-steps, or using a prediction oracle, such as measuring the error of only those 10% of predicted trajectories that are most similar to the test set trajectories.
Revised: Unfortunately, prior work relied on other evaluation measures: [5] reports several different measures, based either on the L 2 distance between trajectories at different time-steps, or using a prediction oracle, such as measuring the error of only those 10% of predicted trajectories that are most similar to the test set trajectories.

Original: [1] also reports different values: the average L 2 distance between full trajectories or trajectories at certain time-steps, or the sampling-based surrogate to the log-likelihood, which has the form of a softmin over L 2 distances, and therefore improves the more sampled trajectories are used to evaluate it.
Revised: [1] also reports different values: the average L 2 distance between full trajectories or trajectories at certain time-steps, or the sampling-based surrogate to the log-likelihood, which has the form of a softmin over L 2 distances, and therefore improves the more sampled trajectories are used to evaluate it.

Original: A shortcoming of these measures is they do not measure if the predicted probability reﬂects the true one, not even approximately, but each has a bias that allows constructing simple baselines that beat the complex learning-based models.
Revised: A shortcoming of these measures is they do not measure if the predicted probability reﬂects the true one, not even approximately, but each has a bias that allows constructing simple baselines that beat the complex learning-based models.

Original: For example, any L 2 -based distance discourages multi-modal output distributions: in the case that multiple future paths are possible, the L 2 distance is smaller when always predicting the mean location instead of different samples hitting different modes of the distribution.
Revised: For example, any L 2 -based distance discourages multi-modal output distributions: in the case that multiple future paths are possible, the L 2 distance is smaller when always predicting the mean location instead of different samples hitting different modes of the distribution.

Original: The oracle measure does encourage diversity, but it does so by simply ignoring all predictions that are not close to the ground truth.
Revised: The oracle measure does encourage diversity, but it does so by simply ignoring all predictions that are not close to the ground truth.

Original: This allows achieving very good scores by a shot gun type of prediction, where trajectories are simply spread out over different directions in the image, see our discussion below.
Revised: This allows achieving very good scores by a shot gun type of prediction, where trajectories are simply spread out over different directions in the image, see our discussion below.

Original:  Experiments
Revised:  Experiments

Original: Datasets.
Revised: Datasets.

Original: We report on experiments on two datasets that were also used in previous work: the
Revised: We report on experiments on two datasets that were also used in previous work: the

Original: MNIST sequence dataset (MNISTseq) [3] consists of the original MNIST digits converted to pixel-bypixel sequential trajectories.
Revised: MNIST sequence dataset (MNISTseq) [3] consists of the original MNIST digits converted to pixel-bypixel sequential trajectories.

Original: While usually smooth, trajectories can also have long-distance jumps, reﬂecting the non-trivial topology of the digits.
Revised: While usually smooth, trajectories can also have long-distance jumps, reﬂecting the non-trivial topology of the digits.

Original: The Stanford Drone Dataset (SDD) [7] contains aerial images of street scenes, with trajectories corresponding to the movement of different objects in the scene.
Revised: The Stanford Drone Dataset (SDD) [7] contains aerial images of street scenes, with trajectories corresponding to the movement of different objects in the scene.

Original: Because objects move at different speeds, locations at adjacent time points can have different distances from each other.
Revised: Because objects move at different speeds, locations at adjacent time points can have different distances from each other.

Original: Long-range jumps are not possible, though.
Revised: Long-range jumps are not possible, though.

Original: For SDD, the ﬁrst image of any sequence is used as a reference image.
Revised: For SDD, the ﬁrst image of any sequence is used as a reference image.

Original: For MNISTseq, no reference images are used.
Revised: For MNISTseq, no reference images are used.

Original: Results.
Revised: Results.

Original: Qualitative results for our prediction method are depicted in Figure 2.
Revised: Qualitative results for our prediction method are depicted in Figure 2.

Original: One can see that the proposed model has learned diverse distributions that also reﬂect speciﬁc characteristics of the data, such as the presence or absence of jumps.
Revised: One can see that the proposed model has learned diverse distributions that also reﬂect speciﬁc characteristics of the data, such as the presence or absence of jumps.

Original: We infer from this that complex latent variable models are not required to learn probabilistic and multi-modal distribution.
Revised: We infer from this that complex latent variable models are not required to learn probabilistic and multi-modal distribution.

Original: A spatio-temporal convolutional structure that encodes probability in the output layer is sufﬁcient.
Revised: A spatio-temporal convolutional structure that encodes probability in the output layer is sufﬁcient.

Original: The quantitative results are promising, too.
Revised: The quantitative results are promising, too.

Original: On MNIST, the proposed model achieves a cross-entropy score of 27 .
Revised: On MNIST, the proposed model achieves a cross-entropy score of 27 .

Original: 7 , far better than the score of 73 .
Revised: 7 , far better than the score of 73 .

Original: 4 achieved by the commonly used baseline of learning a
Revised: 4 achieved by the commonly used baseline of learning a

Original: Gaussian distribution with mean provided by a LSTM that is trained on the sequence coordinates, and with Laplacian smoothing.
Revised: Gaussian distribution with mean provided by a LSTM that is trained on the sequence coordinates, and with Laplacian smoothing.

Original: Even though [1] also reported result on MNISTseq, the values are, unfortunately, not comparable, because that model was trained and evaluated on sequences that were
Revised: Even though [1] also reported result on MNISTseq, the values are, unfortunately, not comparable, because that model was trained and evaluated on sequences that were

Original: [Table]
Revised: [Table]

Original: padded with constant entries to a ﬁxed length.
Revised: padded with constant entries to a ﬁxed length.

Original: Quantitative results for SDD are provided in Table(left).
Revised: Quantitative results for SDD are provided in Table(left).

Original: Again, STCNN achieves better results than an baseline.
Revised: Again, STCNN achieves better results than an baseline.

Original: For reference, we also report some measures as they have been used previously in the literature.
Revised: For reference, we also report some measures as they have been used previously in the literature.

Original: Measured by average L 2 -error, our proposed models achieve a value 43 . 0 , while the (uni-modal)
Revised: Measured by average L 2 -error, our proposed models achieve a value 43 . 0 , while the (uni-modal)

Original: LSTM baseline achieves a better value of 35 . 6 .
Revised: LSTM baseline achieves a better value of 35 . 6 .

Original: This is consistent with the concern that the L 2 -error penalizes multi-modality rather than encouraging it.
Revised: This is consistent with the concern that the L 2 -error penalizes multi-modality rather than encouraging it.

Original: For SDD, Table 1 (right) shows the top-10% oracle error at different time points in the forecasted trajectory, as in [1, 5].
Revised: For SDD, Table 1 (right) shows the top-10% oracle error at different time points in the forecasted trajectory, as in [1, 5].

Original: Even though our method yields results comparable to the literature, we argue that the oracle measure should not used to judge model quality, because it encourages random guessing.
Revised: Even though our method yields results comparable to the literature, we argue that the oracle measure should not used to judge model quality, because it encourages random guessing.

Original: To show this, we performed a simple sanity check: for any initial segment we estimate the average speed of the object and extrapolate from the last position and orientation in 10 different ways using either slightly different orientations (0 ◦ , ± 8 ◦ , ± 15 ◦ ) or different speeds (no movement, or exponentially weighted average with coefﬁcients 0 , 0 . 3 , 0 . 7 , 1 . 0 ).
Revised: To show this, we performed a simple sanity check: for any initial segment we estimate the average speed of the object and extrapolate from the last position and orientation in 10 different ways using either slightly different orientations (0 ◦ , ± 8 ◦ , ± 15 ◦ ) or different speeds (no movement, or exponentially weighted average with coefﬁcients 0 , 0 . 3 , 0 . 7 , 1 . 0 ).

Original: Table 1 (right) shows that this "shot gun" baseline achieves oracle scores at least as good as all other methods, despite not even making use of the reference image.
Revised: Table 1 (right) shows that this "shot gun" baseline achieves oracle scores at least as good as all other methods, despite not even making use of the reference image.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed a simple probabilistic model for probabilistic trajectory forecasting from image data, called STCNN.
Revised: We proposed a simple probabilistic model for probabilistic trajectory forecasting from image data, called STCNN.

Original: It consists of a spatio-temporal convolutional neural networks that is applied in an auto-regression way.
Revised: It consists of a spatio-temporal convolutional neural networks that is applied in an auto-regression way.

Original: Probabilistic behavior emerges by parametrizing the output layer to represent a (discrete) probability distribution.
Revised: Probabilistic behavior emerges by parametrizing the output layer to represent a (discrete) probability distribution.

Original: STCNN is easy and efﬁcient to train using a maximum likelihood objective and yields promising results.
Revised: STCNN is easy and efﬁcient to train using a maximum likelihood objective and yields promising results.

Original: Furthermore, we identiﬁed shortcomings in the evaluation protocol of earlier work, which either discourages or overencourages diversity and can be fooled by simple baselines.
Revised: Furthermore, we identiﬁed shortcomings in the evaluation protocol of earlier work, which either discourages or overencourages diversity and can be fooled by simple baselines.

Original: To encourage reproducibility, we will also release our code.
Revised: To encourage reproducibility, we will also release our code.

Original: In future work, we plan to reimplement previous models and to reevaluate them with the proposed cross-entropy measure to allow fair comparisons.
Revised: In future work, we plan to reimplement previous models and to reevaluate them with the proposed cross-entropy measure to allow fair comparisons.

Original: We also plan to study the integration of more contextual knowledge, such as trajectories of multiple simultaneously moving objects.
Revised: We also plan to study the integration of more contextual knowledge, such as trajectories of multiple simultaneously moving objects.

Original: Acknowledgements.
Revised: Acknowledgements.

Original: This work was in parts funded by the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no 308036.
Revised: This work was in parts funded by the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no 308036.