Image Modeling with Deep Convolutional Gaussian Mixture Models
Authors: Alexander Gepperth, Benedikt Pfülb
Keywords: Gaussian Mixture Model, Deep Learning, Unsupervised Representation Learning, Sampling
Abstract: In this conceptual work, we present DCGMM, a deep hierarchical Gaussian Mixture Model (GMM) that is particularly suited for describing and generating images.
Vanilla (i.e., "flat") GMMs require a very large number of components to well describe images, leading to long training times and memory issues. 
DCGMMs avoid this by a stacked architecture of multiple GMM layers, linked by convolution and pooling operations. 
This allows to exploit the compositionality of images in a similar way as deep CNNs do.
This sets them apart from vanilla GMMs which are trained by EM, requiring a prior k-means initialization which is infeasible in a layered structure.
For generating sharp images with DCGMM, we introduce a new gradient-based technique for sampling through non-invertible operations like convolution and pooling.
Based on the MNIST and FashionMNIST datasets, we validate the DCGMM model by demonstrating its superiority over "flat" GMMs for clustering, sampling and outlier detection.
We additionally demonstrate the applicability of DCGMM to variant generation, in-painting and class-conditional sampling.