S TROKE N ET : A N EURAL P AINTING E NVIRONMENT
A BSTRACT
We’ve seen tremendous success of image generating models these years.
Generating images through a neural network is usually pixel-based, which is fundamentally different from how humans create artwork using brushes.
To imitate human drawing, interactions between the environment and the agent is required to allow trials.
However, the environment is usually non-differentiable, leading to slow convergence and massive computation.
In this paper we try to address the discrete nature of software environment with an intermediate, differentiable simulation.
We present StrokeNet , a novel model where the agent is trained upon a wellcrafted neural approximation of the painting environment.
With this approach, our agent was able to learn to write characters such as MNIST digits faster than reinforcement learning approaches in an unsupervised manner.
Our primary contribution is the neural simulation of a real-world environment.
Furthermore, the agent trained with the emulated environment is able to directly transfer its skills to real-world software.
I NTRODUCTION
To learn drawing or writing, a person ﬁrst observes (encodes) the target image visually and uses a pen or a brush to scribble (decode), to reconstruct the original image.
For an experienced painter, he or she foresees the consequences before taking any move, and could choose the optimal action.
Stroke-based image generation is fairly different from traditional image generation problems due to the intermediate rendering program.
Raster-based deep learning approaches for image generation allow effective optimization using back-propagation.
While for stroke-based approaches, rather than learning to generate the image, it is more of learning to manipulate the painting program.
An intuitive yet potentially effective way to tackle the problem is to ﬁrst learn this mapping from “stroke data” to the resulting image with a neural network, which is analogous to learning painting experience.
An advantage of such a mapping over software is that it provides a continuous transformation.
For any painting program, the pixel values of an image are calcuated based on the coordinate points along the trajectory of an action.
Speciﬁc pixels are indexed by the discrete pixel coordinates, which cuts the gradient ﬂow with respect to the action.
In our implementation, the indexing is done by an MLP described in Section 3.
We further deﬁne “drawing” by giving a formal deﬁnition of “stroke”.
In our context, a “stroke” consists of color, brush radius, and a sequence of tuples containing the coordinate and pressure of each point along the trajectory.
We will later describe this in detail in Section 3.
Based on these ideas, we train a differentiable approximator of our painting software, which we call a “ generator ”.
We then tested the generator by training a vanilla CNN as an agent that encodes the image into “stroke” data as an input for the environment.
Our proposed architecture, StrokeNet , basically comprises the two components, a generator and an agent .
Finally, an agent is trained to write and draw pictures of several popular datasets upon the generator.
For the MNIST (LeCun & Cortes, 2010) digits, we evaluated the quality of the agent with a classiﬁer trained solely on the original MNIST dataset, and tested the classiﬁer on generated images.
We also compared our method with others to show the efﬁciency.
We explored the latent space of the agent as well.
R ELATED W ORK
Generative models such as VAEs(Kingma & Welling, 2013; Sohn et al., 2015) and
GANs(Goodfellow
et al., 2014; Mirza & Osindero, 2014; Radford et al., 2015; Arjovsky et al., 2017) have achieved huge success in image generation in recent years.
These models generate images directly to pixel-level and thus could be trained through back-propagation effectively.
To mimic human drawing, attempts have been made by both graphics and machine learning communities.
Traditionally, trial-and-error algorithms(Hertzmann, 2003) are designed to optimize stroke placement by minimizing an energy function, incorporating heuristics, e.g., constraining the number of strokes.
Concept learning is another example tackling this problem using Bayesian program learning (Lake et al., 2015).
Recent deep learning based approaches generally falls into two categories: RNN-based approaches and reinforcement learning.
For RNN-based approaches such as SketchRNN (Ha & Eck, 2017) and handwriting generation with RNN by Graves (Graves, 2013), they both rely on sequential datasets.
Thus for unpaired data, those models cannot be applied.
Another popular solution is to adopt reinforcement learning such as “artist agent”(Xie et al., 2012) and SPIRAL (Ganin et al., 2018).
These methods train an agent that interact with the painting environment.
For reinforcement learning tasks with large, continuous action space like this, the training process can be computationally costly and it could take the agent tens of epochs to converge.
To mitigate this situation, we simulate the environment in a differentiable manner much alike the idea in World Models (Ha & Schmidhuber, 2018; Schmidhuber, 1990; 2018), where an agent learns from a neural network simulated environment.
Similar approach is also used in character reconstruction for background denoising(Huang et al., 2018).
In our scenario, we train our generator (auto-encoder) by parts for ﬂexible stroke sequence length and image resolution, discussed in Section 3 and 4.
Differentiable rendering is an extensively researched topic in computer graphics.
It is used to solve inverse rendering problems.
Some differentiable renderers explicitly model the relationship between the parameters and observations (Loper & Black, 2014), others use neural network to approximate the result (Nguyen-Phuoc et al., 2018) since neural nets are powerful function approximators.
While little has been done on simulating 2D rendering process adopted in digital painting software, we used a generator neural network to meet our needs.
S TROKE N ET A RCHITECTURE AND E NVIRONMENT 3. S TROKE
We deﬁne a single stroke as follows,
[Equation]
where c ∈ R 3 stands for RGB color, scalar r for brush radius, and tuple ( x i , y i , p i ) for an anchor point on the stroke, consisting of x, y coordinate and pressure p , and n is the maximum number of points in a single stroke, in this case, 16 .
These values are normalized such that the coordinates correspond to the default OpenGL coordinate system.
[Equation]
for k = 1 , 2 , 3 and i = 1 , 2 , · · · , n .
We used absolute coordinates for each point.
It is notable that compared to the QuickDraw (Ha & Eck, 2017) dataset which contains longer lines, our strokes consist of much fewer points.
We consider many trajectory points redundant since the stroke lines can be ﬁtted by spline curves with fewer anchor points.
For example, to ﬁt a straight line, only two end-points are needed regardless of the length, in other words, stroke curves are usually scaleinvariant.
However, if we are to sample the data from a user input, we could have dozens of points along the trajectory.
Hence we made the assumption of being able to represent curves with a few
[Figure]
anchors.
We later showed that a single stroke with only 16 anchors is able to ﬁt most MNIST digits and generate twisted lines in Section 5.
We further assumed that longer and more complicated lines can be decomposed into simple segments and extended our experiments to include recurrent drawing of multiple strokes to generate more complex drawings.
3. G ENERATOR
The outline of the StrokeNet architecture is shown in Figure 1.
The generator takes s as input, and projects the stroke data with two MLPs.
One is the position encoder which encodes ( x i , y i , p i ) into 64 × 64 feature maps, the other, brush encoder encodes the color and radius of the brush to a single 64 × 64 feature map.
The color c (cid:48) is a single gray scale scalar whose value equals to 13 (cid:80) 3 k =
c k , while color strokes are approximated by channel mixing described in Section 3.4.
The features are then concatenated and passed to the (de)convolution layers.
To preserve the sequential and pressure information of each point ( x i , y i , p i ) , the position encoder ﬁrst maps ( x i , y i ) to the corresponding position onto a 64 × 64 matrix by putting a bright dot on that point.
This is modeled by a 2D Gaussian function with its peak scaled to 1, which simpliﬁes to:
[Equation]
for i = 1 , 2 , · · · , n where the value is calculated for each point ( x, y ) on the 64 × 64 map.
Denote this mapping from ( x i , y i ) to R 64 × 64 as pos :
[Equation]
By multiplying the corresponding pressure p i , we now have n position features, in our setup, sixteen.
This part of the generator is trained separately with random coordinates until it generates accurate and reliable signals.
However, if we directly feed these features into the (de)convolutional layers of the network, the generator fails partly due to the sparsity of the single brightness feature.
Instead, we take every two neighbouring feature maps and add them together (denoted by “reduce” in Figure 1.),
[Equation]
Now, each feature map f i represents a segment of the stroke.
By learning to connect and curve the n − 1 “segments”, we are able to reconstruct the stroke.
By appending the encoded color and radius data we now have the feature with shape 64 × 64 × n .
We then feed the features into three (de)convolutional layers with batch-normalization (Ioffe & Szegedy, 2015) activated by
LeakyReLU
(Xu et al., 2015).
The last layer is activated by tanh.
B r u s h
D a t a
C oo r d i n a t e
P r e ss u r e
G e n e r a t o r × +
C NN C NN
F C F C
[Figure]
3. A GENT
The agent is a VGG (Simonyan & Zisserman, 2014)-like CNN that encodes the target image into its underlying stroke representation s .
Three parallel FC-decoders with different activations are used to decode position (tanh), pressure (sigmoid) and brush data (sigmoid) from the feature.
We used average-pooling instead of max-pooling to improve gradient ﬂow.
For the recurrent version of
StrokeNet, two separate CNNs are trained for the target image and the drawing frame, as shown in
Figure 2.
In practice the target image feature is computed once for all steps.
3. E NVIRONMENT
We ﬁrst built a painting software using JavaScript and WebGL.
We later tailored this web application for our experiment.
The spline used to ﬁt the anchor points is centripetal CatmullRom (Catmull & Rom, 1974; Barry & Goldman, 1988).
A desirable feature about Catmull-Rom spline is that the curve goes through all control points, unlike the more commonly used Bezier curve (Sederberg & Farouki, 1992).
We then interpolate through the sampled points and draw circles around each center point as shown in Figure 3.
For each pixel inside a circle, its color depends on various factors including attributes of the brush, blending algorithm, etc.
Our generator is trained on the naive brush.
When it comes to the color blending of two frames, the generator is fed with the mean value of input
RGB color as a gray scale scalar, and its output is treated as an alpha map.
Normalization and alpha-blending is then performed to yield the next color frame, to simulate real blending algorithm underlying the software.
Denote the generator output at time-step t by q ( t ) ∈ R 256 × 256 , the frame image by r ( t ) ∈ R 3 × 256 × 256 , RGB color of the brush by c ∈ R 3 , the blending process is approximated as follows,
[Equation]
for k = 1 , 2 , 3 corresponding to the RGB channels, where J denotes a 256 × 256 all-one matrix.
T RAINING M ETHODS 4. D ATASET FOR G ENERATOR
For the generator, we synthesize a large amount of samples, each of length n .
We would like to capture both the randomness and the smoothness of human writing, thus it is natural to incorporate chaos , most notably, the motion of three-body (Nielsen et al., 2001).
Start
End
There is no closed-form solution to three-body problem, and error accumulates in simulation using numerical methods, leading to unpredictable and chaotic results.
We simulate three-body motion in space (z-component for pressure) with random initial conditions and sample the trajectories as strokes for our dataset.
The simulation is done with a set of equations using Newton’s universal law of gravitation:
[Equation]
where P i ( i = 1 , 2 , 3 , P i ∈ R 3 ) denotes the position of the three objects respectively, (cid:126)F i denotes the gravitational force exerted on the i th object.
In our simulation we set mass m 1 = m 2 = m 3 =and gravitational constant G = 5 × 10 − 5 .
We also always keep our camera (origin point) at the center of the triangle formed by the three objects to maintain relatively stable “footage”.
Using this method we collected about 600K images since there is virtually no cost to generate samples.
Samples from the dataset are shown in Figure 4.
4. D ATASETS FOR A GENT
To prove the effectivess of our neural environment, we trained an agent to perform drawing task on several popular datasets, from characters to drawings, with the generator part frozen.
For MNIST and Omniglot, we trained an agent to draw the characters within one stroke.
We later trained the recurrent StrokeNet on more complex datasets like QuickDraw and KanjiVG (Ofusa et al., 2017).
We resized all the input images to 256 × 256 with anti-alias and paddings.
4. T RAINING M ETHODOLOGY
At ﬁrst we train the position encoder guided by function pos that maps a coordinate to a 64 ×matrix with l 2 distance to measure the loss.
Next we freeze the position encoder and train the other parts of the generator, again with l 2 loss to measure the performance on the three-body dataset.
It can be found that smaller batch size results in more accurate images.
We trained the generator with a batch size of 64 until the loss no longer improves.
We then set the batch size to 32 to sharpen the neural network.
To train the agent , we freeze the generator .
Denote the agent loss as l agent , the generated image and ground-truth image as i gen and i gt respectively, the loss is deﬁned as:
[Equation]
where P k =
[ x k , y k , p k ] T is the data describing the k th anchor point on the stroke.
Here the summation term constrains the average distance between neighbouring points, where λ denotes the penalty strength.
If we drop this term, the agent fails to learn the correct order of the points in a stroke because the generator itself is, after all, not robust to all cases of input, and is very likely to produce wrong results for sequences with large gaps between neighbouring points.
E XPERIMENTS
All experiments are conducted on a single NVIDIA Tesla P40 GPU.
We ﬁrst experimented with single step StrokeNet on MNIST and Omniglot, then we experimented recurrent StrokeNet with QuickDraw and Kanji.
For the MNIST dataset, we later enforced a Gaussian prior to the latent variable and explored the latent space of the agent by linear interpolation.
Finally, for a quantitative evaluation of the model, we trained a classiﬁer on MNIST and tested the classiﬁer with images generated by the agent.
The close accuracies indicate the quality of the generated images.
5. S INGLE -S TEP S TROKE N ET
It can be seen that a single stroke provides rich expressive power for different shapes.
The generator generalizes well to unseen stroke patterns other than the synthesized three-body dataset.
On the Omniglot dataset, since many characters consist of multiple strokes while the agent can only draw one, the agent tries to capture the contour of the character.
learns to “sketch” the is the
5. R ECURRENT -S TEP S TROKE N ET
For more complex datasets multiple steps of strokes are needed.
Again the agent does pretty well to capture the contour of the given image.
However, it seems that the agent has trouble to recover the details of the pictures and tends to smear inside the boundaries with thick strokes.
[Figure]
( - ) + = ( - ) + =
[Figure]
5. L ATENT S PACE E XPLORATION
To convert the agent into a latent space generative model, we experimented with the VAE version of the agent, where the feature obtained from the last layer of CNN is projected into two vectors representing the means µ and standard deviations (activated by softplus) σ , both ofdimensions.
A vector noise of i.i.d.
Gaussian U ∼ N (0 , I ) is sampled, latent variable z is given by
[Equation]
We did latent space interpolation with the agent trained on MNIST.
The simple data led to easily interpretable results.
Since the images are generated by strokes, the digits transform smoothly to one another.
That is to say, the results looked as if we were directly interpolating the stroke data.
Results are shown in Figure 9 and 10.
5. P ERFORMANCE E VALUATION
In order to evaluate the agent, we trained a 5-layer CNN classiﬁer solely on pre-processed
MNIST dataset, which is also the input to the MNIST agent.
The size of the image is 256 × 256 , so there is some performance drop to the classiﬁcation task compared to standard 28 × 28 images.
The classiﬁer is then used to evaluate the paired test-set image generated by the agent.
The accuracies reﬂect the quality of the generated images.
We also compared the l 2 loss with SPIRAL on MNIST to illustrate that our method has the advantage of faster convergence over reinforcement learning approaches, shown in Figure 11.
TEST DATA ACCURACY
Pre-processed images 90.82%
Agent Output (3 steps) 88.43%
Agent Output (1 step) 79.33%
Agent Output (1 step, VAE) 67.21%
[Figure] [Figure]
[Figure] [Figure]
D ISCUSSION
For future work, there are several major improvements we want to make both to the network structure and to the algorithm.
The recurrent structure adopted here is of the simplest form.
We use this setup because we consider drawing as a Markov process, where the current action only depends on what the agent sees, the target image and the previous frame.
More advanced structures like LSTM (Hochreiter &
Schmidhuber, 1997) or GRU (Chung et al., 2014) may boost the performance.
A stop sign can be also introduced to determine when to stop drawing, which can be useful in character reconstruction.
For the agent, various attention mechanism could be incorporated to help the agent focus on undrawn regions, so that smear and blurry scribbles might be prevented.
Secondly, The generator and the agent were trained as two separate parts throughout the experiment.
We can somehow train them as a whole: during the training of the agent, store all the intermediate stroke data.
After a period of training, sample images from the real environment with the stroke data just collected, and train the generator with the data.
By doing so in an iterative manner, the generator could ﬁt better to the current agent and provide more reliable reconstructions, while a changing generator may potentially provide more valuable overall gradients.
It is also found useful to add a bit of randomness to the learning rate.
Since different decoders of the agent learn at different rates, stochasticity results in more appealing results.
For example, the agent usually fails to generalize to color images because it always sticks with one global average color (as shown in Figure 12).
However, it sometimes generates appealing results with some randomness added during the training.
As a result of this immobility, the way agent writes is dull compared to humans and reinforcement learning agents like SPIRAL.
For instance, when writing the digit “8”, the agent is simply writing “3” with endpoints closed.
Also, the agent avoids to make intersecting strokes over all datasets, although such actions are harmless and should be totally encouraged and explored!
Thus, random sampling techniques could be added to the decision making process to encourage bolder moves.
Finally, for the evaluation metrics, the naive l 2 loss can be combined with adversarial learning.
If paired sequential data is available, we believe adding it to training will also improve the results.
C ONCLUSION
In this paper we bring a proof-of-concept that an agent is able to learn from its neural simulation of an environment.
Especially when the environment is deﬁnite given the action, or contains a huge action space which requires massive computation, the proposed approach could be useful.
Our primary contribution is that we devised a model-based method to approximate non-differentiable environment with neural network, and the agent trained with our method converges quickly on several datasets.
It is able to adapt its skills to real world.
Hopefully such approaches can be useful when dealing with more difﬁcult reinforcement learning problems.
A CKNOWLEDGEMENTS
This work was partially supported by the National Natural Science Foundation of China (U1711262, U1811264, 11501204).
We thank our anonymous reviewers for their valuable feedback and opinions.
A PPENDIX 8. E NVIRONMENT D ETAILS
Let P i
=
[ x i , y i ]
T denote the coordinate of a sampled point.
For a curve deﬁned by points
P 0 , P 1 , P 2 , P 3 , the spline can be produced by:
[Equation]
where
[Equation]
with α = 0 . 5 , t 0 =
and i = 0 , 1 , 2 ,
By interpolating t from t 1 to t 2 linearly, we generate the curve between P 1 and P 2 .
The pressure values between neighbouring points are interpolated linearly.
8. L OSSES
[Figure]
[Figure] [Figure]