Paper:
Generative Adversarial Networks (GANs) have achieved remarkable results in the task of generating realistic natural images. In most applications, GAN models share two aspects in common. On the one hand, GANs training involves solving a challenging saddle point optimization problem, interpreted as an adversarial game between a generator and a discriminator functions. On the other hand, the generator and the discriminator are parametrized in terms of deep convolutional neural networks. The goal of this paper is to disentangle the contribution of these two factors to the success of GANs. In particular, we introduce Generative Latent Optimization (GLO), a framework to train deep convolutional generators without using discriminators, thus avoiding the instability of adversarial optimization problems. Throughout a variety of experiments, we show that GLO enjoys many of the desirable properties of GANs: learning from large data, synthesizing visually-appealing samples, interpolating meaningfully between samples, and performing linear arithmetic with noise vectors.

Review:
Summary: The authors observe that the success of GANs can be attributed to two factors; leveraging the inductive bias of deep CNNs and the adversarial training protocol. In order to disentangle the factors of success, and they propose to eliminate the adversarial training protocol while maintaining the first factor. The proposed Generative Latent Optimization (GLO) model maps a learnable noise vector to the real images of the dataset by minimizing a reconstruction loss. The experiments are conducted on CelebA and LSUN-Bedroom datasets. 
Strengths: 
The paper is well written and the topic is relevant for the community.
The notations are clear, as far as I can tell, there are no technical errors.
The design choices are well motivated in Chapter 2 which makes the main idea easy to grasp. 
The image reconstruction results are good. 
The experiments are conducted on two challenging datasets, i.e. CelebA and LSUN-Bedroom.
Weaknesses:
A relevant model is Generative Moment Matching Network (GMMN) which can also be thought of as a “discriminator-less GAN”. However, the paper does not contrast GLO with GMMN either in the conceptual level or experimentally. 
Another relevant model is Variational Autoencoders (VAE) which also learns the data distribution through a learnable latent representation by minimizing a reconstruction loss. The paper would be more convincing if it provided a comparison with VAE.
In general, having no comparisons with other models proposed in the literature as improvements over GAN such as ACGAN, InfoGAN, WGAN weakens the experimental section.
The evaluation protocol is quite weak: CelebA images are 128x128 while LSUN images are 64x64. Especially since it is a common practice nowadays to generate much higher dimensional images, i.e. 256x256, the results presented in this paper appear weak. 
Although the reconstruction examples (Figure 2 and 3) are good, the image generation results (Figure 4 and 5) are worse than GAN, i.e. the 3rd images in the 2nd row in Figure 4 for instance has unrealistic artifacts, the entire Figure 5 results are quite boxy and unrealistic. The authors mention in Section 3.3.2 that they leave the careful modeling of Z to future work, however the paper is quite incomplete without this.
In Section 3.3.4, the authors claim that the latent space that GLO learns is interpretable. For example, smiling seems correlated with the hair color in Figure 6. This is a strong claim based on one example, moreover the evidence of this claim is not as obvious (based on the figure) to the reader. Moreover, in Figure 8, the authors claim that the principal components of the GLO latent space is interpretable. However it is not clear from this figure what each eigenvector generates. The authors’ observations on Figure 8 and 9 are not clearly visible through manual inspection.  
Finally, as a minor note, the paper has some vague statements such as
“A linear interpolation in the noise space will generate a smooth interpolation of visually-appealing images”
“Several works attempt at recovering the latent representation of an image with respect to a generator.”
Therefore, a careful proofreading would improve the exposition.