Paper:
Variational Memory Addressing in Generative Models
Aiming to augment generative models with external memory, we interpret the output of a memory module with stochastic addressing as a conditional mixture distribution, where a read operation corresponds to sampling a discrete memory address and retrieving the corresponding content from memory. This perspective allows us to apply variational inference to memory addressing, which enables effective training of the memory module by using the target information to guide memory lookups. Stochastic addressing is particularly well-suited for generative models as it naturally encourages multimodality which is a prominent aspect of most high-dimensional datasets. Treating the chosen address as a latent variable also allows us to quantify the amount of information gained with a memory lookup and measure the contribution of the memory module to the generative process. To illustrate the advantages of this approach we incorporate it into a variational autoencoder and apply the resulting model to the task of generative few-shot learning. The intuition behind this architecture is that the memory module can pick a relevant template from memory and the continuous part of the model can concentrate on modeling remaining variations. We demonstrate empirically that our model is able to identify and access the relevant memory contents even with hundreds of unseen Omniglot characters in memory.
The KL(q||p) cost to identify a specific memory entry might be substantial, even though the cost of accessing a memory entry should be in the order of log |M|. Middle & Right: We combine a top-level categorical distribution p(a) and a conditional variational autoencoder with a Gaussian p(z|m).
when retrieving contents from memory during training. We compute the sampling distribution over the addresses based on a learned similarity measure between the memory contents at each address and the target. The memory contents m a at the selected address a serve as a context for a continuous latent variable z, which together with m a is used to generate the target observation. We therefore interpret memory as a non-parametric conditional mixture distribution. It is non-parametric in the sense that we can change the content and the size of the memory from one evaluation of the model to another without having to relearn the model parameters. And since the retrieved content m a is dependent on the stochastic variable a, which is part of the generative model, we can directly use it downstream to generate the observation x. These two properties set our model apart from other work on VAEs with mixture priors [9,10] aimed at unconditional density modelling. Another distinguishing feature of our approach is that we perform sampling-based variational inference on the mixing variable instead of integrating it out as is done in prior work, which is essential for scaling to a large number of memory addresses.
Most existing memory-augmented generative models use soft attention with the weights dependent on the continuous latent variable to access the memory. This does not provide clean separation between inferring the address to access in memory and the latent factors of variation that account for the variability of the observation relative to the memory contents (see Figure 1). Or, alternatively, when the attention weights depend deterministically on the encoder, the retrieved memory content can not be directly used in the decoder.
Our contributions in this paper are threefold: a) We interpret memory-read operations as conditional mixture distribution and use amortized variational inference for training; b) demonstrate that we can combine discrete memory addressing variables with continuous latent variables to build powerful models for generative few-shot learning that scale gracefully with the number of items in memory; and c) demonstrate that the KL divergence over the discrete variable a serves as a useful measure to monitor memory usage during inference and training.

Review:
This paper proposes a memory-augmented generative model that performs stochastic, discrete memory addressing, by interpreting the memory as a non-parametric conditional mixture distribution. This variational memory addressing model can combine discrete memory addressing variables with continuous latent variables, to generate samples only with few samples in the memory, which is useful for few-shot learning. The authors implement a VAE version of their model and validate it for the few-shot recognition tasks on the Omniglot dataset, on which it significantly outperforms the Generative Matching Networks, which is an existing memory-augmented network model. Further analysis shows that the proposed model accesses relevant part of the memory even with hundreds of unseen instances in the memory. 
Pros:
- Performing discrete, stochastic memory addressing for memory-augmented generative model is a novel idea which makes sense. Also, the authors have done a good job in motivating why this model is superior to soft attention approach.
- The proposed variational addressing scheme is shown to work well in case of few-shot learning, even in case where existing soft-attention model fails to work.   
- The proposed scheme of interpreting the memory usage with KL divergence seems useful.
Cons
-Not much, except that the experimental study only considers character data, although they are standard datasets. It would be better if the paper provides experimental results on other types of data, such as images, and compared against (or coupled with) recent generative models (such as GANs)
Overall, this is a good paper that presents a novel, working idea. It has effectively solved the problem with existing soft-attention memory addressing model, and is shown to work well for few-shot learning. Thus I vote for accepting the paper. 
- Some typos:
Line 104: unconditioneal -> unconditional
Line 135: "of" is missing between context and supervised.
Line 185: eachieving -> achieving
Table 1 is missing a label "number of shots" for the numbers 1,2,3,4,5,10 and 19.