L AYOUT GAN: G ENERATING G RAPHIC L AYOUTS WITH W IREFRAME D ISCRIMINATOR
Anonymous authors
Paper under double-blind review
A BSTRACT
Layouts are important for graphic design and scene generation.
We propose a novel generative adversarial network, named as LayoutGAN, that synthesizes graphic layouts by modeling semantic and geometric relations of 2D elements.
The generator of LayoutGAN takes as input a set of randomly placed 2D graphic elements and uses self-attention modules to reﬁne their semantic and geometric parameters jointly to produce a meaningful layout.
Accurate alignment is critical for good layouts.
We thus propose a novel differentiable wireframe rendering layer that maps the generated layout to a wireframe image, upon which a CNNbased discriminator is used to optimize the layouts in visual domain.
We validate the effectiveness of LayoutGAN in various experiments including MNIST digit generation, document layout generation, clipart abstract scene generation and tangram graphic design.
I NTRODUCTION
Graphic design is an important visual communication tool in our modern world, encompassing everything from book covers to magazine layouts to web design.
Whereas methods for generating realistic natural-looking images have made signiﬁcant progress lately, e.g., with Generative Adversarial Networks (GANs) Karras et al.
(2017), methods for creating designs are far more primitive.
This is, in part, due to the difﬁculty of ﬁnding data representations suitable for learning.
Graphic designs are normally composed by scalable primitive objects, such as polygons, curves and ellipses, instead of pixels laid on a regular lattice.
The crucial semantic meaning of graphic design depends on the presence of elements, their attributes and their relations to other elements, which can be reﬂected by the layout of these elements on a page.
The visual perception of design depends on the arrangement of these elements; misalignment of two elements of just a few millimeters can ruin the design.
Training from images of designs using conventional GANs synthesizes the layouts in pixel-level, thus do not precisely characterize dependence among different elements and would be unlikey to capture arrangement and layout styles well.
Modeling such highly-structured data using neural networks is of great interest as they usually represent human abstract knowledge about the visual world Zitnick & Parikh (2013); Song et al.
and how these knowledge are expressed via documents and designs Deka et al.
(2017a)
; Yang et al.
Instead of generating pixel-level images like conventional GANs, this paper introduces LayoutGAN, a novel GAN to explore the synthesis process of structured data, which directly synthesizes a set of relational graphic elements representing semantic layouts for graphic design.
In our network, each element is represented by its class label and its geometric parameters, e.g., polygon keypoints.
The generator takes as input graphic elements with randomly-sampled classes and geometric parameters, and arranges them in a design; the output is the contextually reﬁned class probabilities and geometric parameters of the design elements.
The generator has the desirable property of being permutationinvariant: it will generate the same layout if we re-order the input elements.
Following up the synthesized and real structured data, we propose two kinds of discriminator networks.
The ﬁrst one, is similar in structure to the generator: it operates directly on the class probabilities and geometric parameters of the elements, and learns to extract the semantic and geometric relations among different elements for layout optimization from graphic domain.
Though effective, it is not sensitive enough to the misalignment and occlusion of different elements, which is crucial to graphic layout designs.
The second one is in visual domain.
Like a human viewer who judge a graphic design by looking at its rasterized image, the spatial-semantic relations among different elements can be well reﬂected in visual domain by mapping them to 2D layouts.
Then CNNs can be used for layout optimization as they are specialized in distinguishing visual patterns including but not limited to misalignment and occlusion.
However, the key challenge is how to map the geometric parameters to pixel-level layouts differentiably.
Apparently, one trial is to render the graphic elements into bitmap masks using STN Jaderberg et al.
But we found the ﬁlled pixels within the design elements such as polygons cause occlusion issue and are ineffective for back-propagation (for example, a small polygon hides inside a larger one).
We experimented with bitmap mask rendering but it was not successful.
In this paper, we propose a novel differentiable wireframe rendering layer that raterizes both synthesized and real structured data of graphic elements into wireframe images, upon which a standard CNN can be used to optimize the layout across the visual and the graphic domain.
The wireframe rendering discriminator has several advantages.
First, convolution layers are very good at extracting spatial patterns of images so that they are more sensitive to alignment.
Second, the rendered wireframes make it visible how two elements may overlap and thus the network is alleviated from inferring the occlusions that may occur in other renderings such as masks.
We evaluate the LayoutGAN for several different tasks.
In the MNIST experiments, we represent each digit by a set of points, which is the simplest geometric form to generate digits for sanity test and evaluate the output with inception scores Salimans et al.
In document experiments, we generate page layouts with a set of semantically-labeled bounding boxes, and compare with the results from conventional GANs.
In clipart abstract scene experiments, we show our LayoutGAN can accurately model objects’ pairwise semantic and geometric relations by arranging the layout of a set of speciﬁc scene elements to synthesize clipart abstract scenes.
We further demonstrate that our method can generalize to more complex elements in tangram graphic design and show how our LayoutGAN can be used to optimize randomly-perturbed layouts.
In summary, this work makes the following contributions.
Unlike conventional GANs that generates pixel-level images structurally, the proposed LayoutGAN directly synthesizes structured data, that is a set of graphic elements to form meaningful layouts, which is totally resolution free.
A novel differentiable wireframe rendering layer is proposed to rasterize graphic elements to wireframe images, allowing the discriminator to judge alignment from discrete element arrangements.
R ELATED W ORK
Structured data generation.
Convolutional networks have been shown successful for generating data in regular lattice, such as images Radford et al.
, videos Vondrick et al.
and 3D volumes Yan et al.
; Wu et al.
When generating highly-structured data, such as texts Donahue et al.
and programs Reed & De Freitas (2015), recurrent networks are often the ﬁrst choice Sutskever et al.
(2014), esepcially equipped with attention Bahdanau et al.
and
memory modules Graves et al.
Recently, researchers show that convolutional networks can be also used to synthesize sequences van den Oord et al.
; Van Den Oord et al.
(2016) using auto-regressive models.
However, in many cases, a datum cannot be expressed as a sequence Vinyals et al.
(2015), but just a set of elements, e.g. point clouds.
Fan et al.
(2017) propose a point set generation network for synthesizing 3D point clouds of the object shape from a single images.
It is further paired with a point set classiﬁcation network Qi et al.
(2017) for auto-encoding 3D point clouds Achlioptas et al.
Our work extends the set repsentation to more general priminitive objects, i.e. semantically-labeled polygons.
Meanwhile, researchers also model the structured data of connected elements using graph convolutions Kipf & Welling (2016).
Data-driven graphic design.
Automating layout is a classic problem in graphic design Hurst et al.
O’Donovan et al.
(2014a) formulate an energy function by assembling various heuristic visual cues and design principles to optimize single-page layouts, and extend this to an interactive tool O’Donovan et al.
The model parameters are learned from a small number of example designs.
Pang et al.
(2016) optimize layout for desired gaze direction.
Deka et al.
(2017a) collect a mobile app design database for harnessing data-driven applications and present preliminary results of learning similarities of pixel-level textural/non-textual masks for design search, but do not learn models from this data.
Swearngin et al.
(2018) propose an interactive system that converts example design screenshots to vector graphics for designers to re-use and edit.
Bylinskii et al.
(2017) analyze the visual importance of graphic designs and use saliency map as the driving force to assist retargeting and thumbnailing.
Previous methods have learned models for other graphic design elements, such as fonts O’Donovan et al.
(2014b)
and colors O’Donovan et al.
These are orthogonal to the layout problem, and could be combined in future work. No previous method has learned to create design or layout from large datasets, and no previous work has applied GANs to layout.
[Figure]
fc
[Figure]
fc fc fc fc
Decoder
CNN
Differentiable WireframeRendering
Wireframe Layout f(p 1 ,x 1 ) f(p 2 ,x 2 ) … .
f(p i ,x i ) … .
f(p n ,x n ) f(p 1 ,x 1 ) f(p 2 ,x 2 ) ….
f(p i ,x i ) ….
f(p n ,x n ) f ’ (p 1 ,x 1 ) f'(p 2 ,x 2 )
[Figure]
f'(p i ,x i )
[Figure]
f'(p n
[Figure]
Probabilities GeometricParameters {(p 1 ’, θ 1 ’)
[Figure]
… ,(p N ’, θ N ’)} p 1 ’ θ 1 ’ p 2 ’ θ 2 ’ ●●● p N ’ θ N ’
[Figure]
for layout optimization.
3D scene synthesis.
Interior scene synthesis and furniture layout generation draws great interest in graphics community.
Early approaches focus on optimizaton of hand-crafted design principles Merrell et al.
and learning statistical priors of pairwise object relationships Fisher et al.
(2012) due to limited data.
Kai Wang & Ritchie (2018) recently propose a sequential decision making approach to indoor scene synthesis.
In each step, a CNN is trained to predict either location or category of one object by looking at the rendered top-down views.
This bears resemblance to our wireframe rendering discriminator in spirit of using convolutions to capture spatial patterns of layouts.
L AYOUT GAN 3. O VERVIEW
Instead of generating pixel-level images, the LayoutGAN synthesizes a set of (total number of N) relational graphic elements represented as { ( p 1 , θ 1 ) , · · · , ( p N , θ N ) } , where p is the class probability and θ = (cid:2) θ 1 , ..., θ m (cid:3) denote geometric parameters (total number of m ) that can be in various forms for different graphic layouts.
For example, θ =
[ X, Y ] for 2D point set generation (MNIST digit) and θ = (cid:2) X 1 , Y 1 , X 2 , Y 2 (cid:3) for bounding box generation (document layout).
It can be easily extended to other geometric forms depending on different tasks, for example, θ =
[ X, Y, S, F ] for graphic layouts considering position, scale and ﬂip simultaneously (clipart abstract scene).
In conjunction with class probability p , LayoutGAN is ﬂexible to synthesize graphic layouts of different elements with various geometric forms as needed.
To synthesize semantic layouts of relational graphic elements, we introduce new designs in both the generator and the discriminator of our LayoutGAN.
The generator aims to learn a function G that takes as input layout z
= { ( p 1 , θ 1 ) , · · · , ( p N , θ N ) } consisting of initial graphic elements with randomly sampled class labels and geometric parameters ( p i , θ i ) , ( i = 1 , · · · , N ) , and outputs contextually reﬁned ( p i (cid:48) , θ i (cid:48) ) for element i in z to form a new graphical layout
G ( z ) = (cid:8) ( p 1 (cid:48) , θ 1 (cid:48) ) , · · · , ( p N (cid:48) , θ N (cid:48) ) (cid:9) which resembles a real graphic layout.
Note that different from vanilla GANs where z represents a low-dimensional latent variable, our z represents initial random graphic layout that has the same structure as the real one.
While the discriminator learns to capture the semantic and geometric relations among different elements for layout optimization from both the graphic domain and the visual domain.
Next, we go into the details of the generator and discriminator design.
3. G ENERATOR NETWORK ARCHITECTURE
As shown in Figure 1, the generator takes as input a set of graphic elements with random class labels and geometric parameters sampled from Uniform and Gaussian distribution respectively.
An encoder consisting of a multilayer perceptron network (implemented as multiple fully connected layers) ﬁrst embeds the class label and geometric parameters of each graphic element.
The relation module implemented as self-attention module inspired by Wang et al.
(2017) is then used to reﬁne the embedded features of each graphic element contextually by modeling its semantic and spatial relations with all the other elements in the set.
Denote f ( p i , θ i ) as the embedded feature of the graphic element i , its reﬁned feature representation f (cid:48)
( p i , θ i ) can be obtained through a contextual residual learning process, which is deﬁned as:
[Equation]
Here j is the element index and the unary function U computes a representation of the embedded feature f ( p j , θ j ) of element j .
The pairwise function H computes a scalar value representing the relation between elements i and j .
Thus, all the other elements j (cid:54) =
i contribute to the feature reﬁnement of element i by summing up their relations.
The response is normalized by the total number of elements in the set, N .
The weight matrix W r computes a linear embedding, producing the contextual residual to be added on f ( p i , θ i ) for feature reﬁnement.
In our experiments, we deﬁne
H as a dot-product:
[Equation]
where δ ( f ( p i , θ i ))
=
W δ f ( p i , θ i ) and φ ( f ( p j , θ j )) =
W φ f ( p j , θ j ) are two linear embeddings.
We stack T = 4 relation modules for feature reﬁnement in our experiments.
Finally, a decoder consisting of another multilayer perceptron network followed by two branches of fully connected layer with sigmoid activation are used to map the reﬁned feature of each element back to class probabilities and geometric parameters respectively.
Optionally, Non-Maximum Suppression (NMS) can be applied to remove duplicated elements.
3. D ISCRIMINATOR NETWORK ARCHITECTURES
The discriminator aims to distinguish between the synthesized and the real layouts.
We have explored two approaches from graphic and visual domains respectively.
For the graphic domain solution, denoted as the relation-based discriminator, is to directly extract the relations among different graphic elements in the parameter space for layout optimization.
In contrast, the visual domain solution, denoted as the wireframe rendering discriminator, a differentiable wireframe rendering layer is proposed to map graphic elements to 2D wireframe images, thus CNNs are applied to optimize the layout from visual domain.
3.3. R ELATION - BASED DISCRIMINATOR
The relation-based discriminator embeds the input graphic elements and extracts their graphical relations for layout optimization from graphic domain.
It takes as input a set of graphic elements represented by class probability and geometric parameters, and feeds them to an encoder consisting of a multilayer perceptron network for feature embedding f ( p i , θ i ) and then extract their global graphical relations among different elements g ( r ( p 1 , θ 1 ) , · · · , r ( p N , θ N )) where r ( p i , θ i ) is obtained from the similar relation module as that in generator but without shortcut connection, and g is a max-pooling function Qi et al.
Thus, the global relations among all graphic elements can be modeled, upon which a classiﬁer composed by a multilayer perception network with sigmoid activation is applied for real/fake prediction.
3.3. W IREFRAME RENDERING DISCRIMINATOR
To take advantage of CNNs to learn visual patterns for effcient layout optimization, a key issue is to map graphic elements to 2D images in a differentiable way, which is essentially a process of rasterization.
Let’s consider to rasterize a graphic layout with N elements, denoted as { ( p 1 , θ 1 ) , ..., ( p N , θ N ) } onto a target image
I ( X t , Y t ) , where ( X t , Y t ) is the location in pre-deﬁned regular grid.
Assuming there are C semantic classes for each element, the target image I is thus of
C channels.
The pixel I ( X t , Y t ) on the location ( X t , Y t ) in the rendered image can be calculated through class-wise maximum operation of the rendered class probability distributions on ( X t , Y t ) of all elements, generally formulated as:
[Equation]
where R (( p i , θ i ) , ( X t , Y t )) denotes the rendering process as:
[Equation]
(X 1, Y 1 ) (X 2, Y 2 )
[Figure]
Triangle 𝑃·𝑅𝑒𝐿𝑢 1−
𝑋 𝑡 −𝑋 1 𝑅𝑒𝐿𝑢(1−|𝑌 𝑡 −𝑌 1 |)
(X 1, Y 1 ) 𝑃·𝑅𝑒𝐿𝑢(1−|𝑌
𝑡 −𝑌 1 |)
𝑃·𝑅𝑒𝐿𝑢(1−|𝑌
𝑡 −𝑌 2 |) 𝑃 · 𝑅 𝑒 𝐿 𝑢− 𝑋 𝑡 − 𝑋𝑃 · 𝑅 𝑒 𝐿 𝑢− 𝑋 𝑡 − 𝑋(X 1, Y 1 )
(X 3, Y 3 ) (X 2, Y 2 ) 𝑃·𝑅𝑒𝐿𝑢(1−|𝑌
𝑡 −𝑌 2 |) where function F computes the rasterization, which varies for different geometric forms of graphic elements.
In this paper, we focus on polygons θ i =
( X 1 i , Y 1 i , ..., X Ki , Y Ki ) with K keypoints.
We start with the simplest geometric form, a single keypoint θ i =
( X 1 i , Y 1 i ) for element i .
We implement an interpolation kernel k for its rasterization.
Its spatial rendering response on ( X t , Y t ) in the rendered image can be written as:
[Equation]
We adopt bilinear interpretation Johnson et al.
(2016), corresponding to the kernel k ( d ) = max (0 , 1 − | d | ) (implemented as ReLU activation ), as shown in Figure 2.
As R (( p i , θ i ) , ( X t , Y t )) is a linear function of the class probability and the coordinates, gradients can be propagated backward in to them both.
We validate such rendering design for MNIST digit generation, detailed experiments can be seen in Section 4.1.
We now consider more complex polygons.
Assuming an element is a rectangle, or bounding box represented by its top-left and bottom-right coordinates θ =
( X 1 , Y 1 , X 2 , Y 2 ) , which is very common in various designs.
Speciﬁcally, considering a rectangle i with coordinates ( X 1 i , Y 1 i , X
i , Y 2 i ) , as shown in Figure 2, the black grids represent the locations in the rendered image and the orange dotted box represents the rectangle being rasterized in the rendered image.
For a wireframe representation, only the points near the boundary of the dotted box (lie in blue solid line) are related to the rectangle, so its spatial rendering response on ( X t , Y t ) can be formulated as:
[Equation]
where b ( d ) =
min(max(0 , d ) , 1) constraining the rendering to nearby pixels.
We further describe the wireframe rendering process of another geometric form, triangle.
For triangle i represented by its three vertices’ coordinates θ i =
( X 1 i , Y 1 i , X
i , Y 2 i , X 3 i , Y 3 i ) , its spatial rendering response on ( X t , Y t ) in the rendered image can be calculated as:
[Equation]
Through this wireframe rendering process, gradients can be propagated backward to both the class probability and geometric parameters of the graphic elements for joint optimization.
A CNN consisting of 3 convolutional layers followed by a fully connected layer with sigmoid activation is then used for predicting real/fake graphic layouts.
E XPERIMENTS
The implementation is based on TensorFlow Abadi et al.
The network parameters are initialized from zero-mean Gaussian with standard deviation of 0 .
All the networks are optimized using Adam Kingma & Ba (2014) with a ﬁxed learning rate of 0 . 00002 .
Detailed architectures can be found in the appendix.
Rendered
[Figure]
[Figure]
Refined Location
[Figure]
[Table] [Table] 4. MNIST DIGITS GENERATION
MNIST is a handwritten digit database consisting of 60,000 training and 10,000 testing images.
For each image, we extract the locations of 128 randomly-selected foreground pixels as the graphic representation so that digit generation can be formulated as layout generation of 2D points.
In
Figure 3a, each image shows 8 × 8 digits rendered from point layouts.
The left and middle images are the generated samples by the LayoutGAN with wireframe rendering discriminator and the one with relation-based discriminator, respectively.
The right image shows the digits rendered from ground-truth point layouts.
One can see that the LayoutGAN with either discriminator well capture various digit patterns by collaboratively reﬁning the coordinates of all the points through relation modeling.
Particularly, the wireframe rendering discriminator helps generate more compact, better aligned point layouts.
For quantitative evaluation, we train a multilayer perceptron network for digits classiﬁcation, which achieves an accuracy of 98 .
91% on MNIST test set, and use it to compute the inception score Salimans et al.
(2016) of 10,000 generated samples.
As shown in Table 1, The
LayoutGAN with wireframe rendering discriminator achieves a higher inception score than the one with relation-based discriminator.
To shed light on the relational reﬁnement process of LayoutGAN, we mark each input random point with a location-speciﬁc color, and track their reﬁned locations in the generated layouts, as shown in Figure 3b.
One can see that colors change gradually along the strokes, showing the network learns some contextually consistent local displacements of the points.
4. D OCUMENT SEMANTIC LAYOUT GENERATION
One document page consists of a number of regions with different semantic roles, such as heading, paragraph, table, ﬁgure, caption and list.
Each region is represented by a bounding box.
Modeling layouts of semantic regions is critical for document analysis, retargeting and synthesis.
In real document data, these semantically-labeled bounding boxes are often well aligned according to some axises and their placement follows some particular patterns, such as heading always appears above paragraph or table (some document page examples can be found in the appendix).
In this experiment, we focus on one-column layouts with no more than 9 bounding boxes that may belong to 6 semantic classes as mentioned above.
We are interested how these spatial-semantic patterns can be captured by our network.
For training data, we collect totally around 25,000 layouts from real document pages.
We also implement a baseline method that represents the layout by semantic masks as used in Yang et al.
; Deka et al. (2017a).
Particularly, we render all semantic regions into masks and train a DCGAN Radford et al.
(2015) for generating mask layouts.
Then we extract the connected mask regions of each semantic class and output enclosing bounding boxes of all regions.
Figureshows some representative generation results (each row consists of 6 samples, high-resolution results can be found in the appendix).
The ﬁrst row shows the results from DCGAN.
One can see that the generated results suffer from misalignment among regions.
In addition, different semantic blocks are in poor arrangement, showing the network cannot well capture the spatial-semantic patterns through layout rendering in pixel-level.
The third row shows the results from LayoutGAN with wireframe rendering discriminator.
We retrieve the most similar real layouts from the training set in the last row as references.
It can be seen that LayoutGAN can well capture different document layout patterns with clear deﬁnitions of instance-level semantic bounding boxes.
To validate the advantage of wireframe rendering discriminator, we compare the generated results with those from the LayoutGAN with relation-based discriminator.
As shown in the second row of
[Figure] [Figure] Figure 4, the latter can also capture different layout patterns but sometimes suffers from overlapping and misalignment issues.
As the bounding boxes in real document layouts are either left- or centeraligned with no overlappings with each other (except for caption with ﬁgure/table), we propose two metrics to quantitatively measure the quality of the generated layouts.
The ﬁrst one is overlapping index, which is the percentage of total overlapping area among any two bounding boxes inside the whole page.
The second one is alignment index which is calculated by ﬁnding the minimum standard deviation of either the left or the center coordinates of all the bounding boxes.
Tableprovides quantitative comparisons of real layouts and synthesized layouts from LayoutGAN with two different discriminators.
One can see that LayoutGAN with wireframe rendering discriminator achieves lower value in both overlapping index and alignment index than the one with relation-based discriminator, validating the superiority of the proposed wireframe rendering method for layout generation.
Similar conclusion can also be drawn by analyzing the loss functions.
We add shift perturbations to the bounding boxes of real layouts and feed them to both discriminators to examine their loss behaviors.
Figure 6 visualizes the loss landscape of both discriminators corresponding to different extent of shift perturbation.
One can see that the loss surface in terms of shift perturbation of the wireframe rendering discriminator is much smoother than that of relation-based discriminator. [Figure] [Table]
C LIPART ABSTRACT SCENE GENERATION For
better visualizing the element’s pairwise semantic and geometric relations captured by the LayoutGAN, we consider abstract scene generation, that is to synthesize abstract scenes by arranging a set of speciﬁc clipart scene elements (boy, girl, glasses, hat, sun, and tree in our experiments) tailored from the Clipart dataset Zitnick et al.
To synthesize a reasonable abstract scene, we ﬁrst use LayoutGAN to generate the layout of inner scene elements by representing each of them as semantically-labeled bounding box with normalized center coordinates, width and height, and ﬂip attribute.
Then, we render the corresponding clipart image of each scene element onto a background image according to the predicted position (center coordinates), scale (width and height) and ﬂip attribute, forming synthesized abstract scenes.
Note that it is challenging as accurate objects’ pairwise spatial-semantic relations are required for reasonable scenes, for example, the glasses/hat should be exactly on the eyes/head of the boy/girl with matching scales and ﬂips simultaneously.
As shown in Figure 5, the ﬁrst three rows show the generated layouts and corresponding rendered scenes by DCGAN, the LayoutGAN with relation-based discriminator and the one with wireframe rendering discriminator respectively.
The last row shows some samples rendered from ground-truth scene layouts.
It can be seen that compared to DCGAN and the LayoutGAN with relation-based discriminator, the one with wireframe rendering discriminator can capture the objects’ pairwise relations more precisely, forming accurate scene layouts (for example, the glasses/hat accurately lies on the eyes/head of the person with varying scales and ﬂips), validating the superiority of the pro- [Figure] [Figure] [Figure]
posed wireframe rendering strategy. In addition, we conduct a user study involving 20 respondents for a subjective evaluation of the results. Speciﬁcally, we randomly sample a same number of abstract scenes synthesized by the above three models. A subject is asked to rate the synthetic scenes in terms of three scores (Excellent, Fair, Poor) by following three guidelines: 1) whether they have coherent overall structures, 2) whether different objects are placed in reasonable relative positions, scales and ﬂips, and 3) whether there are duplicate objects.
We compute the rating percentage of all sampled scenes for each model.
As shown in Table 3, the results of LayoutGAN with the wireframe rendering discriminator receive better ratings, which suggests that they are preferred by respondents.
4. T ANGRAM GRAPHIC DESIGN
The tangram is a dissection puzzle aiming to form a speciﬁc shape using seven pieces of ﬂat shapes without overlapping.
The seven pieces include two large right triangles, one medium right triangle, two small right triangles, one square and one parallelogram, which can be assembled to form a square of side one unit and having area one square unit when choosing a unit of measurement.
We collect totally 149 tangram graphic designs including animals, people and objects.
In our experiments, we consider eight rotation/reﬂection poses for each piece.
Given the seven pieces initialized with random location and ground-truth pose and class, the LayoutGAN with wireframe rendering discriminator is trained to reﬁne their conﬁgurations jointly to form reasonable layouts.
Note that it is a challenging task due to the complex conﬁgurations and limited data.
We perform two experiments.
The ﬁrst one is a perturbation recovering test, in which we ﬁrst add some random perturbations to piece locations of real layouts and train a LayoutGAN to recover the true layouts.
Figure 7 shows that our network is able to pull back the displaced shapes to the right locations, conﬁrming that the network successfully ﬁgures out the relations between different graphic elements.
We further train another LayoutGAN to generate tangram designs from purely random initialization.
In Figure 9 (high-resolution results can be found in the appendix), each row represents the sampled progressive generated results and the retrieved similar real tangrams.
We also present some generated results of DCGAN and a sequential approach that follows Wang et al.
(2018) for comparison.
One can see from Figure 8 that DCGAN cannot well model the spatial relations among different shapes while the sequential model suffers from accumulated error.
In contrast, the LayoutGAN can generate meaningful tangrams like fox and person, although others may be hard to interpret, as shown in the penultimate row.
C ONCLUSION
In this paper, we have proposed a novel LayoutGAN for generating layouts of relational graphic elements.
Different from traditional GANs that generate images in pixel-level, the LayoutGAN can directly output a set of relational graphic elements.
A novel differentiable wireframe rendering layer was proposed to rasterize the generated graphic elements to wireframe images, making it feasible to leverage CNNs as discriminator for better layout optimization from visual domain.
Future works include adding a content representation, such as text, icon and picture to each graphic element.
A PPENDIX 6. R EAL DOCUMENT PAGES
For better understanding the document semantic layouts
[Figure]
we show some layout samples and their corresponding real document pages, as shown in Figure
[Figure]
6. H IGH - RESOLUTION SYNTHESIZED DOCUMENT LAYOUTS
[Figure] [Figure]
Relation-based
Wireframe rendering
Real data
Girl Sun Tree Boy
Glasses Hat
DCGAN
[Figure] [Figure]
6. D ETAILED ARCHITECTURE DESIGN OF THE GENERATOR
The generator takes as input a set of graphic elements with random class labels and geometric parameters sampled from Uniform and Gaussian distribution respectively.
An encoder consisting of three fully connected layers ﬁrst embeds the semantic and geometric parameters of each graphic element.
Two cascaded relation modules implemented as self-attention module followed by a basic residual block (bottleneck building block) form one relation block.
We cascade two such relation blocks (totally 4 relation modules) for contextual feature reﬁnement.
Finally, a decoder consisting of several fully connected layers followed by two heads (implemented as the fully connected layer) with sigmoid activation are used to decode the reﬁned feature of each element back to class probability distributions and geometric parameters respectively. 6. S YNTHESIZED MOBILE APP LAYOUT DESIGNS We further conduct a new experiment of mobile app layout design by using the RICO dataset Deka et al. (2017b).
For better understanding the mobile app layouts, we ﬁrst show some layout samples and their corresponding real mobile app screenshots, as shown in Figure 5.
SemanticLayouts
[Figure]
Figure 6 shows the results from LayoutGAN and the retrieved most-similar real layouts from the training set as references.
Both the wireframe and the corresponding mask layouts are provided for better visualization.
Synthesized Layouts
[Figure]