Paper:
We present a novel approach to train a natural media painting using reinforcement learning. Given a reference image, our formulation is based on stroke-based rendering that imitates human drawing and can be learned from scratch without supervision. Our painting agent computes a sequence of actions that represent the primitive painting strokes. In order to ensure that the generated policy is predictable and controllable, we use a constrained learning method and train the painting agent using the environment model and follows the commands encoded in an observation. We have applied our approach on many benchmarks and our results demonstrate that our constrained agent can handle different painting media and different constraints in the action space to collaborate with humans or other agents.

Review:
This paper presents a reinforcement learning agent trained to interact with a painting environment in order to reproduce target images. The novel aspect of this work seems to be a version of the agent producing partial actions (i.e., portion of the action tuple is clamped to pre-specified values). This new agent receives the clamped components of the action as an additional conditioning input.
Pros:
+ In some cases, images produced by the system look appealing.
Cons:
- The writing of the manuscript could be significantly improved. I had a hard time understanding certain parts of the paper (e.g., what “constrained” means in the context of the present work). I got an impression that there was an effort to make things look more complex than they really are.
- The proposed model lacks novelty – there seems to be only one non-trivial contribution and I’m not entirely sure how useful it is. The authors never compare their system against a simple baseline when one just overrides the actions of the agent.
- In general, the evaluation section of the paper leaves a lot to be desired. The authors report some numbers (most of which are not even for the proposed model) but I’m struggling to make anything out of that information. Why should I care about them? What interesting conclusions can I draw from them? The paper never discusses this. On top of that, there are no baselines.
Notes/questions:
* At test time, the policy receives renders from a real environment. Could that create problems since it has only been trained on images synthesized by a neural surrogate?
* Abstract: “on many benchmarks” -> “on several benchmarks”
* Section 1: “we use a constraint representation along with …” – this sentence needs to be rewritten.
* Section 2: Missing reference – Neural Painters (Nakano, 19). Considers a neural surrogate of the libmypaint environment.
* Section 2, last paragraph:  “both methods can work on either a small action space or a small observation space” – how is the present model different? The action space is very similar (albeit continuous) to the existing approaches (i.e., relies on Bezier curves much like in (Ganin et al., 18)).
* Section 2, last paragraph: The last sentence needs to be rewritten (“uncontrollable agent” looks a bit strange)
* Section 3, first paragraph: “We highlight all …” -> “We describe all …”
* Section 4.1, first paragraph: The first sentence needs to be rewritten (“the corresponding canvas by the given action”).
* Section 4.1, first paragraph: “Unlike the previous …” – not true. (Ganin et al., 18) proposes to use this environment and (Nakano, 19) trains a neural surrogate for it.
* Section 4.4: “WGAN loss (Huang et al., 19)” – this WGAN loss was introduced in (Ganin et al., 18).
* Section 5: I feel like the paper could do a better job at explaining what “constraining” really means and justifying why it’s an interesting problem to solve. In my opinion, Eq. (3), for example, obscures rather than clarifies the notion of “constraints”.
* Section 5, paragraph 2: “For each different” -> “For each”
* Section 5.1, paragraph 4: “pi” -> “\pi”
* Figure 5: The caption almost overlaps with the text below.
* Section 6.3, paragraph 2: “l2” -> “l_2”
* Section 7, paragraph 2: “differing” -> “different”
I feel like the authors should perform a major re-writing of the manuscript before it’s ready for publication. Moreover, I failed to see any significantly novel aspects of the proposed system (maybe due to the poor presentation) and therefore I wouldn’t recommend the paper for acceptance.