Paper:
Generative adversarial networks (GANs) have achieved great success at generating realistic samples. However, achieving disentangled and controllable generation still remains challenging for GANs, especially in the high-resolution image domain. Motivated by this, we introduce AC-StyleGAN, a combination of AC-GAN and StyleGAN, for demonstrating that very limited supervision is sufficient to benefit the generic disentanglement learning of high-resolution images. Specifically, by only using 1% of the labelled data, the disentanglement quality is very close to the fully supervised case and significantly outperforms the unsupervised alternative. Inspired by the observed separation of fine and coarse styles in Style-GAN, we then extend AC-StyleGAN to a new image-to-image model called FCStyleGAN for semantic manipulation of fine-grained factors in a high-resolution image. In experiments, we show that FC-StyleGAN performs well in only controlling fine-grained factors, with the use of instance normalization, and also demonstrate its good generalization ability to unseen images. Finally, we create two new datasets -Falcor3D and Isaac3D with higher resolution, more photorealism, and richer variation, as compared to existing disentanglement datasets.

Review:
Summary:
The paper is reasonably clear (though see some of my detailed comments on writing below).
The idea seems well-motivated and somewhat new though not revolutionary,
and the new data sets are nice (though see comments below about how I'm not qualified to evaluate them),
but I don't understand why the proposed algorithm wasn't evaluated on existing data sets as well,
and I don't understand why it wasn't compared against other algorithms that purport to do the same thing.
Barring certain exceptions (again see detailed comments), it seems like those two things are things
we ask of essentially all machine learning paper submissions, and I don't see why this paper is different?
I am open to being persuaded, but as of now I can't recommend acceptance.
Detailed Comments:
> More importantly, only using 5% of the labelled data significantly improves the disentanglement quality.
Hard to parse.
Does this mean using only 5% of data is better for disentangling than using 100%?
>  Generative adversarial networks (GANs) (Goodfellow et al., 2014) have achieved great success at generating realistic images, such as StyleGAN
Strange sentence - styleGAN is not itself a realistic image.
>  the controllable generation of high-resolution images is possible
Nit: i would cut the 'the'
> which characterizes how significant that the model can change a factor.
What does this mean?
> present a state-ofthe-art challenge
What does it really mean for a challenge to be state of the art?
> that enables conditional generation of high-fidelity images
Pretty minor nit, but I don't think it makes sense to refer to samples themselves as high fidelity.
High fidelity to what?
It makes a little more sense to refer to a trained generator as having high fidelity to the training data set,
but TBH I still don't even like the phrase in that context. 
Fig 1 is good. 
I felt like I could understand what's going on mostly from looking at the figure, which is nice.
The section surrounding eqs 1 and 2 is a little hard to read for me. 
Lots of single letter variable names and it's hard to keep them all in my head when I read the equation.
I'd replace e.g. c_r with \text{code}_r and so forth.
> Note that AC-StyleGAN reduces to an InfoGAN variant in the special case
This is helpful.
> AC-StyelGAN
>  FCStyleAGN
> by symmetry
I don't follow this part.
> while its high-resolution blocks accounts for fine styles
I feel like it's worth making a distinction between high frequency bits of an image and non-essential (or nuisance variables or whatever) bits.
They often are the same, but not always (it really just depends how close up your photo is, right?) but this technique is really separating high-freq from low-freq, IIUC.
>  handling complex high-fidelity images
Again, I just don't feel like this phrase makes any sense.
Re eq 3:
I guess you can call this the 'interpolation variance'
if you want, but it's not really a new thing.
You're just getting discretized measurements of the Jacobian of the mapping from code to predicted code, and I don't really think in a way that fits with the
intuition you describe.
There are a few things I can think of that are weird about this measurement, but here's just one:
I might have a dimension of the code that causes one really peaky change at one point, and then the change quickly reverts.
If i measure the variance of the output, I won't see much (in fact, depending on how you discretize, maybe I'll miss it altogether),
even though the fact that the change is peaky doesn't say anything about the 'importance' of the change.
I think you want something more like a line integral.
It's possible I'm misunderstanding something about the description of this, however.
Re: the experiments:
I like the new data sets, although I'm not familiar enough w/ the robotics literature to know whether they add something really new.
What I don't understand is why there's not really any attempt to do either of:
a) compare the models from the paper to existing models 
b) evaluate the models from the paper on existing data-sets?
I understand that sometimes one comes up with a thing that does something totally new,
and then reviewers will (maybe unjustifiably) still insist that the some kind of comparison be made even though it doesn't make sense,
but that doesn't seem to be what has happened here?
This paper proposes a new algorithm for synthesizing images and controlling various attributes of the images, but this has certainly been done in the past,
so why can't your technique be compared to those prior techniques?
Similarly, there are lots of data sets that people are already familiar with that you could have evaluated these models on, right?