Paper:
We propose a novel generative model architecture designed to learn representations for images that factor out a single attribute from the rest of the representation. A single object may have many attributes which when altered do not change the identity of the object itself. Consider the human face; the identity of a particular person is independent of whether or not they happen to be wearing glasses. The attribute of wearing glasses can be changed without changing the identity of the person. However, the ability to manipulate and alter image attributes without altering the object identity is not a trivial task. Here, we are interested in learning a representation of the image that separates the identity of an object (such as a human face) from an attribute (such as 'wearing glasses'). We demonstrate the success of our factorization approach by using the learned representation to synthesize the same face with and without a chosen attribute. We refer to this specific synthesis process as image attribute manipulation. We further demonstrate that our model achieves competitive scores, with state of the art, on a facial attribute classification task.

Review:
In this paper, the authors introduce a neural network architecture that has three components.
First a VAE is used to encode images in to two latent states \hat{y} and \hat{z}, with \hat{z}
intended to be class (e.g. face attribute) agnostic. The decoder reconstructs images from \hat{y}
and \hat{z} concatenated together. A GAN style discriminator attempts to distinguish the 
decoded image from the original input image as real or fake, allowing the decoder to produce 
higher quality decoded images. An auxiliary network A attempts to classify the face attribute y
from the class agnostic features \hat{z}, with the idea being that the encoder should try to produce 
\hat{z} vectors from which the class cannot be predicted. An additional classifier is trained
using a classification loss \hat{L}_{class} on the encoded reconstructed image, the use of which 
I don't understand.
I think additional work on section 2.5 through section 3 would be helpful to improve clarity.
As one example, "y" is unnecessarily overloaded: y denotes a specific attribute, \hat{y}
denotes a latent vector that is intended to not be class agnostic, \tilde{y} denotes the
prediction of an auxiliary network on an intended class-agnostic latent vector \hat{z} of
the presence of the original attribute y, and \hat{\hat{y}} denotes the non agnostic latent
vector achieved by passing the decoded image back through the encoder.
This notational complexity is compounded by the fact that a number of steps in the method are
not well motivated in the text, and left to the reader to understand their purpose. For example,
the authors state that "we incorporate a classification model into the encoder so that our model may
easily be used to perform classification tasks." What does this mean? In the diagram (Figure 1),
where is this classification model? Why in the GAN loss is there a term that compares the
fake loss with the result of classifying a decoded z vector? Is this z \hat{z}, or a latent vector
drawn from a distribution p(z)? If it is the former, how does this term differ from the second
term in the GAN loss. If it is the latter, then shouldn't it be concatenated with some y in order to
be used as input to the decoder D_{\theta}?
Why is it important to extract \hat{\hat{y}} from \hat{x}? In the paper you state that the loss
"provides a gradient containing label information to the decoder," but why can't we use the known label y
of the original input x to ensure that the encoder and decoder preserve this information if it is used as \hat{y}?
Later in the paper, you explicitly state that \hat{\mathcal{L}_{class}} "does not provide any clear benefit."
If that is the case, then you should ideally include it neither in the model nor in the paper. If it was
included primarily because previous models included it, then I would recommend you introduce its use
in a background section on Bao et al., 2017 rather than including it in your model description with an
explanation like "so that our model may easily be used to perform classification tasks."
Ultimately, this last point brings us to a good summary of my concerns with the model: the inclusion
of too many moving parts, some of which the authors explicitly say later on provide no benefit.
Moving on to experimental results, I think this is another area where I have a few concerns. First, in
Figure 2, the authors argue that your model is "better for 6 out of 10 attributes" and comparable results for most others. The authors include a gap of 0.1 in the "Gray_hair" category as "better" but label a gap of 0.5
in the Black hair category as "comparable." I think results in several of the categories are sufficiently close
that error bars would be necessary to draw actual conclusions. If "better" were to mean "better by 0.5" for example,
then the authors method is better on 4 tasks (smiling, blonde hair, heavy makeup, mustache) and worse on 3 (black hair, brown hair, wavy hair).
With respect to the actual attribute editing, my main concern here is a lack of comparison to models other than Bao et al., despite the fact that face attribute changing is an exhaustively studied task. A number of papers like Perarnau et al., 2016, Upchurch et al., 2017, Lample et al., 2017 and others study this task from machine learning perspectives, and in some cases can perform photorealistic image attribute editing without complicated machinery on megapixel face
images. At least the images in Figure 3 and 4 are substantially downsampled from the typical resolution found in the Celeba dataset, suggesting that there was some failure mode on full resolution images.
----
Edit: I've reviewed the authors' addressing my concerns in their paper and am happy to increase my rating as a result.