Paper:
The ability to forecast a set of likely yet diverse possible future behaviors of an agent (e.g., future trajectories of a pedestrian) is essential for safety-critical perception systems (e.g., autonomous vehicles). In particular, a set of possible future behaviors generated by the system must be diverse to account for all possible outcomes in order to take necessary safety precautions. It is not sufficient to maintain a set of the most likely future outcomes because the set may only contain perturbations of a dominating single outcome (major mode). While generative models such as variational autoencoders (VAEs) have been shown to be a powerful tool for learning a distribution over future trajectories, randomly drawn samples from the learned implicit likelihood model may not be diverse -the likelihood model is derived from the training data distribution and the samples will concentrate around the major mode of the data. In this work, we propose to learn a diversity sampling function (DSF) that generates a diverse yet likely set of future trajectories. The DSF maps forecasting context features to a set of latent codes which can be decoded by a generative model (e.g., VAE) into a set of diverse trajectory samples. Concretely, the process of identifying the diverse set of samples is posed as DSF parameter estimation. To learn the parameters of the DSF, the diversity of the trajectory samples is evaluated by a diversity loss based on a determinantal point process (DPP). Gradient descent is performed over the DSF parameters, which in turn moves the latent codes of the sample set to find an optimal set of diverse yet likely trajectories. Our method is a novel application of DPPs to optimize a set of items (forecasted trajectories) in continuous space. We demonstrate the diversity of the trajectories produced by our approach on both low-dimensional 2D trajectory data and high-dimensional human motion data. (Video)

Review:
This paper presents a novel approach for forecasting object trajectories (e.g., predicted paths of vehicles) forcing diversity of outputs. The authors adopt determinantal point processes (DPPs) to capture the diversity and propose a diversity sampling function (DSF) which consists of a neural network. Its trainable parameter (i.e., \gamma) maps the past trajectory into a set of latent codes and they are decoded to feasible trajectories by a pre-trained conditional variational autoencoder (cVAE). The DSF is trained by maximizing the diversity of the output trajectory. But, since the standard log-likelihood function can be singular, they additionally present an objective function for diversity by maximizing the expected cardinality, which admits replicated outputs. In experiments, the proposed method finds more diverse paths than other competitors under both for synthetic 2D objects and human motions.
This paper is well-written and easy to understand. The contribution can be important as it performs better than other generative networks that are not forcing diversity. Unlike them, the proposed method can capture the diversity trajectories, which requires for safety-critical applications. 
Main concerns:
1. The DPP kernel consists of a similarity (equation (8)) and a quality score (equation (9)). However, it is unnatural that the similarity is defined in the data space (x) and the quality is defined in latent space (z). A more naive approach is to define the DPP kernel as a function of x or z. Is there any specific reason to define those scores are defined in different space?
2. To maximize the expected cardinality, it is enough that the eigenvalues of L(\gamma) become large. Does the network find the trivial solution? E.g., all eigenvalues are the same as a very large value. In addition, the proposed diverse loss is not a new approach. The maximum induced cardinality of a DPP was proposed by [1] and its various properties were also studied therein. 
3. In a work of [1], a method to optimize the induced cardinality (a similar to diverse loss in this paper) is proposed. And it would be great to compare the maximum induced cardinality (also can be approximated by the greedy algorithm) to the MAP for inference of diverse trajectory.
4. It is also possible to apply DPP MAP inference to a set of latent codes generated from the encoder of cVAE. Then, the decoder can map the diverse latent variables to feasible data trajectories. Are these outputs also diverse? or does the diversity of latent space reflect the diversity in the data trajectory space? 
5. Computing the gradient of the proposed diverse loss is expensive since it is the trace of an inverse of a parameterized matrix. How long does it take to learn the proposed DSF compared to other methods? 
6. To avoid that the determinants become zero, a popular choice is to shift all eigenvalues with a small amount (this can be done by adding eps * identity matrix to the kernel matrix). Did the authors investigate other practical diverse losses?
7. It seems to be possible to train the parameters of the kernel matrix, i.e., k in equation (8) and \omega in equation (9). Did the author try to learn those parameters?
Overall, this work proposes an approach combined cVAE with a DPP for forecasting diverse trajectories and the empirical results are promising as it outperforms other methods. But, its novelty is incremental and competitors are not actually the models capturing diversity. I vote for a weak acceptance but depending on clarifications on the above concerns in an author response, I would be willing to increase the score.
Minor comments:
1 . Please specify the network architecture of DSF and details on the parameter \gamma.
[1] Gillenwater, Jennifer A., et al. "Maximizing induced cardinality under a determinantal point process." Advances in Neural Information Processing Systems. 2018.