Paper:
Encoding images as a series of high-level constructs, such as brush strokes or discrete shapes, can often be key to both human and machine understanding. In many cases, however, data is only available in pixel form. We present a method for generating images directly in a high-level domain (e.g. brush strokes), without the need for real pairwise data. Specifically, we train a "canvas" network to imitate the mapping of high-level constructs to pixels, followed by a high-level "drawing" network which is optimized through this mapping towards solving a desired image recreation or translation task. We successfully discover sequential vector representations of symbols, large sketches, and 3D objects, utilizing only pixel data. We display applications of our method in image segmentation, and present several ablation studies comparing various configurations.

Review:
This paper presents an unsupervised method for generating images in a high-level domain (brush strokes and geometric primitives). The proposed system is comprised of two neural networks: the drawer D and a forward model C of an external renderer R. The latter is trained on the rollouts produced by sending random actions to R. The forward model is then freezed and used to train D, i.e., the network that repeatedly interacts (sends commands) with the C to produce a desired image. Since everything is differentiable, D can be optimized via regular gradient descent.
Pros:
+ The paper is written clearly and relatively easy to read
+ The idea to replace the non-differentiable renderer with a differentiable approximation makes sense. Pure RL setups (i.e., in [Ganin et al., 2018]) are quite sample inefficient and hard to train due to high variance of REINFORCE.
+ The proposed method is tested both in 2D (drawings and floor plans) and 3D (prisms) domains and yields relatively good results.
Cons:
- The datasets used in the paper are quite simplistic. I’m wondering how hard it is to train a forward model for more complex data. At the very least, I would want to see how the method handles the 3D experiment when the view is not axis-aligned and there is more variety in the number of primitives and their types.
- The performance of the method especially on drawings and floor plans is not excellent. The drawer tends to reproduce line art with small disjoint strokes - very different from how humans would accomplish this task. A similar observation can be made for floor plans (the system outputs small pixel-like boxes that tile bigger rooms). This suggests that recovered commands do not really correspond to the higher-level structure of the input. Unlike in RL approaches, injection of prior knowledge about the data (e.g., the floor plan should be reproduced using the minimum possible number of rectangles) seems problematic within the proposed framework.
- It’s unclear how to use the approach for non-continuous actions (e.g., choosing types of primitives in 3D or different instruments in music).
- It seems the method may suffer from significant exploration problems in more complex settings. Consider an image of a rectangle that the system should reproduce. Say, it initially outputs a box that doesn’t intersect with the target one. The gradient of l^2-distance between those two images in the pixel space is non-zero but it is zero w.r.t. the action taken since no small change of the action parameters would make the generated box intersect with the target (assuming that the target is far from the model’s output) so l^2 will stay the same.
- I would love to see some comparison (preferably quantitative - speed of training, quality of reconstructions, etc.) to an RL system. So far, in the paper, there is only one figure showing a couple of images produced by such system.
Notes/questions:
* Section 2, paragraph 2: The systems by Xie et al. and Ganin et al. are very distinct. The former models the appearance of a single stroke while the latter is more similar to the present paper and synthesizes the entire image using strokes of a predefined appearance.
* Section 3.3, paragraph 3: “pixel-wise maximum” - it seems to be a fairly restrictive setup which only works when the model increases intensity of pixels.
* Section 3.4: This is a straightforward idea and is not novel (e.g., already used in some demonstrations of the method in [Ganin et al., 2018])
* Section 4.2, paragraph 2: During training, do you use all the patches or randomly sample them? Is your loss computed per patch or for the entire image?
In summary, the paper presents an interesting idea but the execution needs some improvement (especially, in terms of evaluation) before the paper is ready to be accepted to the conference.
After going through the authors' comments and the revised version of the paper, I keep the rating as is. The paper needs a more convincing evaluation section as well as some clean up (e.g., references to figures and tables in the text)