Paper:
Variational Memory Encoder-Decoder
Introducing variability while maintaining coherence is a core task in learning to generate utterances in conversation. Standard neural encoder-decoder models and their extensions using conditional variational autoencoder often result in either trivial or digressive responses. To overcome this, we explore a novel approach that injects variability into neural encoder-decoder via the use of external memory as a mixture model, namely Variational Memory Encoder-Decoder (VMED). By associating each memory read with a mode in the latent mixture distribution at each timestep, our model can capture the variability observed in sequential data such as natural conversations. We empirically compare the proposed model against other recent approaches on various conversational datasets. The results show that VMED consistently achieves significant improvement over others in both metricbased and qualitative evaluations.

Review:
Summary This paper introduces a new model for natural conversation modelling, the Variational Memory Encoder-Decoder (VMED). The purpose of the model is to learn a distribution over latent variables that captures time dependence and has different modes corresponding to for instance different moods and intentions during the conversation. In order to ensure coherence over time and variability of the generated conversations, an external memory is used as a mixture of Gaussians prior over the latent space, which is updated during the conversation. VMED outperforms all baselines in four conversational datasets.   Originality As far as I understand the building blocks for this model include: 1. A Gaussian mixture model for the prior in the VAE. 2. The use of an external memory  [5,35] to produce the parameters of the Gaussian mixture prior, which is updated at each time step, especially useful for coherence and multimodality in language tasks. 3. A probabilistic framework for sequence generation. [8], in the form of a conditional generative model [17, 38]. To my knowledge, all of these ideas are joined for the first time for sequence generation.  Quality Besides being well motivated, the paper appears technically sound. The evaluation of the model as a whole is strong. The baselines are outperformed by the proposed model.  However, it is hard to understand the influence of the different components on the increased performance over the baselines: i) the use for a multimodal prior distribution for a conditional VAE, ii) an external memory that adapts this distribution over time. It would provide useful insights to show experiments on this, for instance by comparing against a CVAE without an external memory but with a Gaussian mixture model prior.  The details of the architectures used are limited.  Clarity Overall this paper is written in an excellent manner. The formation of ideas is clear and the embedding in relevant literature is good. The figures are helpful in understanding the method, although I would encourage the authors to extend the captions to be more standalone.                          Significance The approach seems novel and relevant for language modelling to me. Since the main application in this paper is conversation modeling, I wonder what modeling choices need to be adapted in order to perform well on other sequence generation tasks, such as translation. Can the authors comment on this?  Detailed comments/questions  1. In Eq. 2, There is a p missing inside the logarithm corresponding to the reconstruction term of the lower bound.  2. Should there be a solid arrow from x to z in figure 1? 3. Can the authors motivate why you choose to use a Gaussian mixture model as a prior instead of using for instance flows to create a multimodal distribution? 4. In Eq. 5, the minus sign in front of the sum should only be in front of the KL term.  5. In Eq. 6, the sum over i=1 to K is outside the log, whereas in the definition above for Dvar it is inside the log. Furthermore, it seems that z^l_t should be z^l_{<=t} in the last term. It would be good to explicitly mention that the last term is an MC estimate with samples from the posterior. Does r_{t-1} depends on z_