Original: M ULTIMODAL A TTENTION FOR L AYOUT S YNTHESIS IN D IVERSE D OMAINS
Revised: M ULTIMODAL A TTENTION FOR L AYOUT S YNTHESIS IN D IVERSE D OMAINS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We address the problem of scene layout generation for diverse domains such as images, mobile applications, documents and 3D objects.
Revised: We address the problem of scene layout generation for diverse domains such as images, mobile applications, documents and 3D objects.

Original: Most complex scenes, natural or human-designed, can be expressed as a meaningful arrangement of simpler compositional graphical primitives.
Revised: Most complex scenes, natural or human-designed, can be expressed as a meaningful arrangement of simpler compositional graphical primitives.

Original: Generating a new layout or extending an existing layout requires understanding the relationships between these primitives.
Revised: Generating a new layout or extending an existing layout requires understanding the relationships between these primitives.

Original: To do this, we propose a multimodal attention framework, MMA, that leverages self-attention to learn contextual relationships between layout elements and generate novel layouts in a given domain.
Revised: To do this, we propose a multimodal attention framework, MMA, that leverages self-attention to learn contextual relationships between layout elements and generate novel layouts in a given domain.

Original: Our framework allows us to generate a new layout either from an empty set or from an initial seed set of primitives, and can easily scale to support an arbitrary of primitives per layout.
Revised: Our framework allows us to generate a new layout either from an empty set or from an initial seed set of primitives, and can easily scale to support an arbitrary of primitives per layout.

Original: Further, our analyses show that the model is able to automatically capture the semantic properties of the primitives.
Revised: Further, our analyses show that the model is able to automatically capture the semantic properties of the primitives.

Original: We propose simple improvements in both representation of layout primitives, as well as training methods to demonstrate competitive performance in very diverse data domains such as object bounding boxes in natural images (COCO bounding boxes), documents (PubLayNet), mobile applications (RICO dataset) as well as 3D shapes (PartNet).
Revised: We propose simple improvements in both representation of layout primitives, as well as training methods to demonstrate competitive performance in very diverse data domains such as object bounding boxes in natural images (COCO bounding boxes), documents (PubLayNet), mobile applications (RICO dataset) as well as 3D shapes (PartNet).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In the real world, there exists a strong relationship between different objects that are found in the same environment (Torralba & Sinha, 2001; Shrivastava & Gupta, 2016).
Revised: In the real world, there exists a strong relationship between different objects that are found in the same environment (Torralba & Sinha, 2001; Shrivastava & Gupta, 2016).

Original: For example, a dining table usually has chairs around it, a surfboard is found near the sea, horses do not ride cars, etc. .
Revised: For example, a dining table usually has chairs around it, a surfboard is found near the sea, horses do not ride cars, etc. .

Original: Biederman (2017) provided strong evidence in cognitive neuroscience that perceiving and understanding a scene involves two related processes: perception and comprehension .
Revised: Biederman (2017) provided strong evidence in cognitive neuroscience that perceiving and understanding a scene involves two related processes: perception and comprehension .

Original: Perception deals with processing the visual signal or the appearance of a scene.
Revised: Perception deals with processing the visual signal or the appearance of a scene.

Original: Comprehension deals with understanding the schema of a scene, where this schema (or layout) can be characterized by contextual relationships ( e.g. , support, occlusion, and relative likelihood, position, and size) between objects.
Revised: Comprehension deals with understanding the schema of a scene, where this schema (or layout) can be characterized by contextual relationships ( e.g. , support, occlusion, and relative likelihood, position, and size) between objects.

Original: For generative models that synthesize scenes, this evidence underpins the importance of two factors that contribute to the realism or plausibility of a generated scene: layout, i.e. , the arrangement of different objects, and their appearance (in terms of pixels).
Revised: For generative models that synthesize scenes, this evidence underpins the importance of two factors that contribute to the realism or plausibility of a generated scene: layout, i.e. , the arrangement of different objects, and their appearance (in terms of pixels).

Original: Therefore, generating a realistic scene necessitates both these factors to be plausible.
Revised: Therefore, generating a realistic scene necessitates both these factors to be plausible.

Original: The advancements in the generative models for image synthesis have primarily targeted plausibility of the appearance signal by generating incredibly realistic images often with a single entity such as faces (Karras et al., 2019; 2017), or animals (Brock et al., 2018; Zhang et al., 2018).
Revised: The advancements in the generative models for image synthesis have primarily targeted plausibility of the appearance signal by generating incredibly realistic images often with a single entity such as faces (Karras et al., 2019; 2017), or animals (Brock et al., 2018; Zhang et al., 2018).

Original: In the case of large and complex scenes, with a lot of strong non-local relationships between different elements, most methods require proxy representations for layouts to be provided as inputs ( e.g. , scene graph, segmentation mask, sentence).
Revised: In the case of large and complex scenes, with a lot of strong non-local relationships between different elements, most methods require proxy representations for layouts to be provided as inputs ( e.g. , scene graph, segmentation mask, sentence).

Original: We argue that to plausibly generate large and complex scenes without such proxies, it is necessary to understand and generate the layout of a scene, in terms of contextual relationships between various objects present in the scene.
Revised: We argue that to plausibly generate large and complex scenes without such proxies, it is necessary to understand and generate the layout of a scene, in terms of contextual relationships between various objects present in the scene.

Original: The layout of a scene, capturing what primitives occupy what parts of the scene, is an incredibly rich representation.
Revised: The layout of a scene, capturing what primitives occupy what parts of the scene, is an incredibly rich representation.

Original: Learning to generate layouts itself is a challenging problem due to the variability of real-world or human-designed layouts.
Revised: Learning to generate layouts itself is a challenging problem due to the variability of real-world or human-designed layouts.

Original: Each layout is composed of a small fraction of possible objects, objects can be present in a wide range of locations, the number of objects varies for each scene and so do the contextual relationships between objects.
Revised: Each layout is composed of a small fraction of possible objects, objects can be present in a wide range of locations, the number of objects varies for each scene and so do the contextual relationships between objects.

Original: [Figure]
Revised: [Figure]

Original: Formally, a scene layout can be represented as an unordered set of graphical primitives.
Revised: Formally, a scene layout can be represented as an unordered set of graphical primitives.

Original: The primitive itself can be discrete or continuous depending on the data domain.
Revised: The primitive itself can be discrete or continuous depending on the data domain.

Original: For example, in the case of layout of documents, primitives can be bounding boxes from discrete classes such as ‘text’, ‘image’, or ‘caption’, and in case of 3D objects, primitives can be 3D occupancy grids of parts of the object such as ‘arm’, ‘leg’, or ‘back’ in case of chairs.
Revised: For example, in the case of layout of documents, primitives can be bounding boxes from discrete classes such as ‘text’, ‘image’, or ‘caption’, and in case of 3D objects, primitives can be 3D occupancy grids of parts of the object such as ‘arm’, ‘leg’, or ‘back’ in case of chairs.

Original: Additionally, in order to make the primitives compositional, we represent each primitive by a location vector with respect to the origin, and a scale vector that deﬁnes the bounding box enclosing the primitive.
Revised: Additionally, in order to make the primitives compositional, we represent each primitive by a location vector with respect to the origin, and a scale vector that deﬁnes the bounding box enclosing the primitive.

Original: Again, based on the domain, these location and scale vectors can be 2D or 3D. A generative model for layouts should be able to look at all existing primitives and propose the placement and attributes of a new one.
Revised: Again, based on the domain, these location and scale vectors can be 2D or 3D. A generative model for layouts should be able to look at all existing primitives and propose the placement and attributes of a new one.

Original: We propose a novel
Revised: We propose a novel

Original: Multimodal Attention framework (MMA) that ﬁrst maps the different parameters of the primitive independently to a ﬁxed-length continuous latent vector, followed by a masked Transformer decoder to look at representations of existing primitives in layout and predict the next parameter.
Revised: Multimodal Attention framework (MMA) that ﬁrst maps the different parameters of the primitive independently to a ﬁxed-length continuous latent vector, followed by a masked Transformer decoder to look at representations of existing primitives in layout and predict the next parameter.

Original: Our generative framework can start from an empty set, or a set of primitives, and can iteratively generate a new primitive one parameter at a time.
Revised: Our generative framework can start from an empty set, or a set of primitives, and can iteratively generate a new primitive one parameter at a time.

Original: Moreover, by predicting either to stop or to generate the next primitive, our sequential approach can generate variable length layouts.
Revised: Moreover, by predicting either to stop or to generate the next primitive, our sequential approach can generate variable length layouts.

Original: Our approach can be readily plugged into scene generation frameworks (e.g., Layout2Image (Zhao et al., 2019), GauGAN (Park et al., 2019b)) or stand-alone applications that require generating layouts or templates with/without user interaction.
Revised: Our approach can be readily plugged into scene generation frameworks (e.g., Layout2Image (Zhao et al., 2019), GauGAN (Park et al., 2019b)) or stand-alone applications that require generating layouts or templates with/without user interaction.

Original: For instance, in the UI design of mobile apps and websites, an automated model for generating plausible layouts can signiﬁcantly decrease the manual effort and cost of building such apps and websites.
Revised: For instance, in the UI design of mobile apps and websites, an automated model for generating plausible layouts can signiﬁcantly decrease the manual effort and cost of building such apps and websites.

Original: Finally, a model to create layouts can potentially help generate synthetic data for various tasks tasks (Yang et al., 2017; Capobianco & Marinai, 2017;
Revised: Finally, a model to create layouts can potentially help generate synthetic data for various tasks tasks (Yang et al., 2017; Capobianco & Marinai, 2017;

Original: Chang et al., 2015; Wu et al., 2017b;a).
Revised: Chang et al., 2015; Wu et al., 2017b;a).

Original: To the best of our knowledge, MMA is the ﬁrst framework to perform competitively with the stateof-the-art approaches in 4 diverse data domains.
Revised: To the best of our knowledge, MMA is the ﬁrst framework to perform competitively with the stateof-the-art approaches in 4 diverse data domains.

Original: We evaluate our model using existing metrics proposed for different domains such as Jensen-Shannon Divergence, Minimum matching distance, and Coverage in case of 3D objects, Inception Score and Fr ´ echet Inception Distance for COCO, and
Revised: We evaluate our model using existing metrics proposed for different domains such as Jensen-Shannon Divergence, Minimum matching distance, and Coverage in case of 3D objects, Inception Score and Fr ´ echet Inception Distance for COCO, and

Original: Negative Log-likelihood of the test set in case of app wireframes and documents.
Revised: Negative Log-likelihood of the test set in case of app wireframes and documents.

Original: Qualitative analysis of the framework also demonstrates that our model captures the semantic relationships between objects automatically (without explicitly using semantic embeddings like word2vec Mikolov et al.
Revised: Qualitative analysis of the framework also demonstrates that our model captures the semantic relationships between objects automatically (without explicitly using semantic embeddings like word2vec Mikolov et al.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Generative models.
Revised: Generative models.

Original: Deep generative models based on CNNs such as variational auto-encoders (VAEs) (Kingma & Welling, 2013), and generative adversarial networks (GANs) (Goodfellow et al., 2014) have recently shown a great promise in terms of faithfully learning a given data distribution and sampling from it.
Revised: Deep generative models based on CNNs such as variational auto-encoders (VAEs) (Kingma & Welling, 2013), and generative adversarial networks (GANs) (Goodfellow et al., 2014) have recently shown a great promise in terms of faithfully learning a given data distribution and sampling from it.

Original: There has also been research on generating data sequentially (Oord et al., 2016; Chen et al., 2020) even when the data has no natural order (Vinyals et al., 2015).
Revised: There has also been research on generating data sequentially (Oord et al., 2016; Chen et al., 2020) even when the data has no natural order (Vinyals et al., 2015).

Original: Many of these approaches often rely on low-level information (Gupta et al., 2020b) such as pixels while generating images (Brock et al., 2018;
Revised: Many of these approaches often rely on low-level information (Gupta et al., 2020b) such as pixels while generating images (Brock et al., 2018;

Original: Karras et al., 2019), videos (Vondrick et al., 2016), or 3D objects (Wu et al., 2016; Yang et al., 2019; Park et al., 2019a; Gupta et al., 2020a) and not on semantic and geometric structure in the data.
Revised: Karras et al., 2019), videos (Vondrick et al., 2016), or 3D objects (Wu et al., 2016; Yang et al., 2019; Park et al., 2019a; Gupta et al., 2020a) and not on semantic and geometric structure in the data.

Original: Scene generation.
Revised: Scene generation.

Original: Generating 2D or 3D scenes conditioned on sentence (Li et al., 2019d; Zhang et al., 2017;
Revised: Generating 2D or 3D scenes conditioned on sentence (Li et al., 2019d; Zhang et al., 2017;

Original: Reed et al., 2016), a scene graph (Johnson et al., 2018; Li et al., 2019a; Ashual &
Revised: Reed et al., 2016), a scene graph (Johnson et al., 2018; Li et al., 2019a; Ashual &

Original: Wolf, 2019), a layout (Dong et al., 2017; Hinz et al., 2019;
Revised: Wolf, 2019), a layout (Dong et al., 2017; Hinz et al., 2019;

Original: Isola et al., 2016; Wang et al., 2018b) or an existing image (Lee et al., 2018) has drawn a great interest in vision community.
Revised: Isola et al., 2016; Wang et al., 2018b) or an existing image (Lee et al., 2018) has drawn a great interest in vision community.

Original: Given the input, some works generate a ﬁxed layout and diverse scenes (Zhao et al., 2019), while other works generate diverse layouts and scenes (Johnson et al., 2018; Li et al., 2019d).
Revised: Given the input, some works generate a ﬁxed layout and diverse scenes (Zhao et al., 2019), while other works generate diverse layouts and scenes (Johnson et al., 2018; Li et al., 2019d).

Original: These methods involve pipelines often trained and evaluated end-to-end, and surprisingly little work has been done to evaluate the layout generation component itself.
Revised: These methods involve pipelines often trained and evaluated end-to-end, and surprisingly little work has been done to evaluate the layout generation component itself.

Original: Layout generation serves as a complementary task to these works and can be combined with these methods.
Revised: Layout generation serves as a complementary task to these works and can be combined with these methods.

Original: In this work, we evaluate the layout modeling
Revised: In this work, we evaluate the layout modeling

Original: [Figure]
Revised: [Figure]

Original: capabilities of two of the recent works (Johnson et al., 2018; Li et al., 2019d) that have layout generation as an intermediate step.
Revised: capabilities of two of the recent works (Johnson et al., 2018; Li et al., 2019d) that have layout generation as an intermediate step.

Original: We also demonstrate the results of our model with Layout2Im (Zhao et al., 2019) for image generation.
Revised: We also demonstrate the results of our model with Layout2Im (Zhao et al., 2019) for image generation.

Original: Layout generation.
Revised: Layout generation.

Original: The automatic generation of layouts is an important problem in graphic design.
Revised: The automatic generation of layouts is an important problem in graphic design.

Original: Many of the recent data-driven approaches use data speciﬁc constraints in order to model the layouts.
Revised: Many of the recent data-driven approaches use data speciﬁc constraints in order to model the layouts.

Original: For example, Wang et al.
Revised: For example, Wang et al.

Original: (2018a; 2019);
Revised: (2018a; 2019);

Original: Li et al.
Revised: Li et al.

Original: (2019c)
Revised: (2019c)

Original: ; Ritchie et al.
Revised: ; Ritchie et al.

Original: (2019) generates topdown view indoor rooms layouts but make several assumptions regarding the presence of walls, roof etc. , and cannot be easily extended to other datasets.
Revised: (2019) generates topdown view indoor rooms layouts but make several assumptions regarding the presence of walls, roof etc. , and cannot be easily extended to other datasets.

Original: In this paper, we focus on approaches that have fewer domain-speciﬁc constraints.
Revised: In this paper, we focus on approaches that have fewer domain-speciﬁc constraints.

Original: LayoutGAN (Li et al., 2019b) uses a GAN framework to generate semantic and geometric properties of a ﬁxed number of scene elements.
Revised: LayoutGAN (Li et al., 2019b) uses a GAN framework to generate semantic and geometric properties of a ﬁxed number of scene elements.

Original: LayoutVAE (Jyothi et al., 2019) starts with a label set, i.e. , categories of all the elements present in the layout, and then generates a feasible layout of the scene.
Revised: LayoutVAE (Jyothi et al., 2019) starts with a label set, i.e. , categories of all the elements present in the layout, and then generates a feasible layout of the scene.

Original: Zheng et al.
Revised: Zheng et al.

Original: (2019) attempt to generate document layouts given the images, keywords, and category of the document.
Revised: (2019) attempt to generate document layouts given the images, keywords, and category of the document.

Original: Patil et al.
Revised: Patil et al.

Original: (2019) proposes a method to construct hierarchies of document layouts using a recursive variational autoencoder and sample new hierarchies to generate new document layouts.
Revised: (2019) proposes a method to construct hierarchies of document layouts using a recursive variational autoencoder and sample new hierarchies to generate new document layouts.

Original: Manandhar et al.
Revised: Manandhar et al.

Original: (2020) develops an auto-encoding framework for layouts using Graph Networks.
Revised: (2020) develops an auto-encoding framework for layouts using Graph Networks.

Original: 3D-PRNN (Zou et al., 2017), PQNet (Wu et al., 2020) and ComplementMe Sung et al.
Revised: 3D-PRNN (Zou et al., 2017), PQNet (Wu et al., 2020) and ComplementMe Sung et al.

Original: (2017), generates 3D shapes via sequential part assembly.
Revised: (2017), generates 3D shapes via sequential part assembly.

Original: While 3D-PRNN generates only bounding boxes, PQ-Net and ComplementMe can synthesize complete 3D shapes starting with a partial or no input shape.
Revised: While 3D-PRNN generates only bounding boxes, PQ-Net and ComplementMe can synthesize complete 3D shapes starting with a partial or no input shape.

Original: Our approach offers several advantages over current layout generation approaches without sacriﬁcing their beneﬁts.
Revised: Our approach offers several advantages over current layout generation approaches without sacriﬁcing their beneﬁts.

Original: By factorizing primitives into structural parameters and compositional geometric parameters, we can generate high-resolution primitives using distributed representations and consequently, complete scenes.
Revised: By factorizing primitives into structural parameters and compositional geometric parameters, we can generate high-resolution primitives using distributed representations and consequently, complete scenes.

Original: The autoregressive nature of the model allows us to generate layouts of arbitrary lengths as well as start with partial layouts.
Revised: The autoregressive nature of the model allows us to generate layouts of arbitrary lengths as well as start with partial layouts.

Original: Further, modeling the position and size of primitives as discrete values (as discussed in § 3.1) helps us realize better performance on datasets, such as documents and app wireframes, where bounding boxes of layouts are typically axis-aligned.
Revised: Further, modeling the position and size of primitives as discrete values (as discussed in § 3.1) helps us realize better performance on datasets, such as documents and app wireframes, where bounding boxes of layouts are typically axis-aligned.

Original: We evaluate our method both quantitatively and qualitatively with state-of-the-art methods speciﬁc to each dataset and show competitive results in very diverse domains.
Revised: We evaluate our method both quantitatively and qualitatively with state-of-the-art methods speciﬁc to each dataset and show competitive results in very diverse domains.

Original:  O UR A PPROACH
Revised:  O UR A PPROACH

Original: In this section, we introduce our attention network in the context of the layout generation problem.
Revised: In this section, we introduce our attention network in the context of the layout generation problem.

Original: We ﬁrst discuss our representation of layouts for primitives belonging to different domains.
Revised: We ﬁrst discuss our representation of layouts for primitives belonging to different domains.

Original: Next, we discuss the Multimodal Attention (MMA) framework and show how we can leverage previous advances such as Transformers (Vaswani et al., 2017) to model the probability distribution of layouts.
Revised: Next, we discuss the Multimodal Attention (MMA) framework and show how we can leverage previous advances such as Transformers (Vaswani et al., 2017) to model the probability distribution of layouts.

Original: MMA allows us to learn non-local semantic relationships between layout primitives and also gives us the ﬂexibility to work with variable length layouts.
Revised: MMA allows us to learn non-local semantic relationships between layout primitives and also gives us the ﬂexibility to work with variable length layouts.

Original: 3. L AYOUT R EPRESENTATION
Revised: 3. L AYOUT R EPRESENTATION

Original: Given a dataset of layouts, a single layout instance can be deﬁned as a graph G with n nodes, where each node i ∈ { 1 , . . . , n } is a graphical primitive.
Revised: Given a dataset of layouts, a single layout instance can be deﬁned as a graph G with n nodes, where each node i ∈ { 1 , . . . , n } is a graphical primitive.

Original: We assume that the graph is fully-connected, and let the attention network learn the relationship between nodes.
Revised: We assume that the graph is fully-connected, and let the attention network learn the relationship between nodes.

Original: The nodes can have structural or semantic information associated with them.
Revised: The nodes can have structural or semantic information associated with them.

Original: For each node, we project the information associated with it to a d model -dimensional space represented by feature vector s i .
Revised: For each node, we project the information associated with it to a d model -dimensional space represented by feature vector s i .

Original: Note that the information itself can be discrete ( e.g. , part category), continuous ( e.g. , color), or multidimensional vectors ( e.g. , signed distance function of the part) on some manifold.
Revised: Note that the information itself can be discrete ( e.g. , part category), continuous ( e.g. , color), or multidimensional vectors ( e.g. , signed distance function of the part) on some manifold.

Original: Speciﬁcally, in our ShapeNet experiments, we use an MLP to project part embedding to d model -dimensional space, while in the 2D layout experiments, we use a learned d model -dimensional category embedding which is equivalent to using an MLP with zero bias to project one-hot encoded category vectors to the latent space.
Revised: Speciﬁcally, in our ShapeNet experiments, we use an MLP to project part embedding to d model -dimensional space, while in the 2D layout experiments, we use a learned d model -dimensional category embedding which is equivalent to using an MLP with zero bias to project one-hot encoded category vectors to the latent space.

Original: Each primitive also carries geometric information g i which we factorize into a position vector and a scale vector.
Revised: Each primitive also carries geometric information g i which we factorize into a position vector and a scale vector.

Original: For the layouts in R 2 such as images or documents, g i =
Revised: For the layouts in R 2 such as images or documents, g i =

Original: [ x i , y i , h i , w i ] , where ( x, y ) are the coordinates of the centroid of primitive and ( h, w ) are the height and width of the bounding box containing the primitive, normalized with respect to the dimensions of the entire layout.
Revised: [ x i , y i , h i , w i ] , where ( x, y ) are the coordinates of the centroid of primitive and ( h, w ) are the height and width of the bounding box containing the primitive, normalized with respect to the dimensions of the entire layout.

Original: Representing geometry with discrete variables.
Revised: Representing geometry with discrete variables.

Original: We apply an 8-bit uniform quantization on each of the geometric ﬁelds and model them using Categorical distribution.
Revised: We apply an 8-bit uniform quantization on each of the geometric ﬁelds and model them using Categorical distribution.

Original: Discretizing continuous signals is a practice adopted in previous works for image generation such as PixelCNN++ (Salimans et al., 2017), however, to the best of our knowledge, it has been unexplored in the layout modeling task.
Revised: Discretizing continuous signals is a practice adopted in previous works for image generation such as PixelCNN++ (Salimans et al., 2017), however, to the best of our knowledge, it has been unexplored in the layout modeling task.

Original: We observe that even though discretizing coordinates introduces approximation errors, it allows us to express arbitrary distributions which we ﬁnd particularly important for layouts with strong symmetries such as documents and app wireframes.
Revised: We observe that even though discretizing coordinates introduces approximation errors, it allows us to express arbitrary distributions which we ﬁnd particularly important for layouts with strong symmetries such as documents and app wireframes.

Original: We project each geometric ﬁeld of the primitive independently to the same d model -dimension, such that i th primitive in R 2 can be represented as ( s i , x i , y i , h i , w i ) .
Revised: We project each geometric ﬁeld of the primitive independently to the same d model -dimension, such that i th primitive in R 2 can be represented as ( s i , x i , y i , h i , w i ) .

Original: We concatenate all the elements in a ﬂattened sequence of their parameters.
Revised: We concatenate all the elements in a ﬂattened sequence of their parameters.

Original: We also append embeddings of two additional parameters s (cid:104) bos (cid:105) and s (cid:104) eos (cid:105) to denote start and end of sequence.
Revised: We also append embeddings of two additional parameters s (cid:104) bos (cid:105) and s (cid:104) eos (cid:105) to denote start and end of sequence.

Original: Our layout in R 2 can now be represented by a sequence of 5 n + 2 latent vectors
Revised: Our layout in R 2 can now be represented by a sequence of 5 n + 2 latent vectors

Original: [Equation]
Revised: [Equation]

Original: For brevity, we use θ j , j ∈ { 1 , . .
Revised: For brevity, we use θ j , j ∈ { 1 , . .

Original: , 5 n + 2 } to represent any element in the above sequence.
Revised: , 5 n + 2 } to represent any element in the above sequence.

Original: We can now pose the problem of modeling this joint distribution as product over series of conditional distributions using chain rule:
Revised: We can now pose the problem of modeling this joint distribution as product over series of conditional distributions using chain rule:

Original: [Equation]
Revised: [Equation]

Original: 3. M ODEL ARCHITECTURE AND TRAINING
Revised: 3. M ODEL ARCHITECTURE AND TRAINING

Original: Our overall architecture is surprisingly simple and shown in Fig.
Revised: Our overall architecture is surprisingly simple and shown in Fig.

Original: Given an initial set of K visible primitives (where K can be 0 when generating from scratch), our attention based model takes as input, a random permutation of the visible nodes, π = ( π 1 , . .
Revised: Given an initial set of K visible primitives (where K can be 0 when generating from scratch), our attention based model takes as input, a random permutation of the visible nodes, π = ( π 1 , . .

Original: . , π K ) , and consequently a sequence of d model -dimensional vectors ( θ 1 , . .
Revised: . , π K ) , and consequently a sequence of d model -dimensional vectors ( θ 1 , . .

Original: , θ 5 K ) .
Revised: , θ 5 K ) .

Original: We ﬁnd this to be an important step since by factorizing primitive representation into geometry and structure ﬁelds, our attention module can explicitly assign weights to individual coordinate dimensions.
Revised: We ﬁnd this to be an important step since by factorizing primitive representation into geometry and structure ﬁelds, our attention module can explicitly assign weights to individual coordinate dimensions.

Original: The attention module is similar to Transformer
Revised: The attention module is similar to Transformer

Original: Decoder (Vaswani et al., 2017) and consists of L attention layers, each of which consists of (a) a masked multi-head attention layer ( h attn ), and (b) fully connected feed forward layer ( h fc ).
Revised: Decoder (Vaswani et al., 2017) and consists of L attention layers, each of which consists of (a) a masked multi-head attention layer ( h attn ), and (b) fully connected feed forward layer ( h fc ).

Original: Each sublayer also adds residual connections (He et al., 2016) and LayerNorm (Ba et al., 2016).
Revised: Each sublayer also adds residual connections (He et al., 2016) and LayerNorm (Ba et al., 2016).

Original: [Equation]
Revised: [Equation]

Original: where l denotes the layer index.
Revised: where l denotes the layer index.

Original: Masking is performed such that θ only attends to all the input latent vectors as well as previous predicted latent vectors.
Revised: Masking is performed such that θ only attends to all the input latent vectors as well as previous predicted latent vectors.

Original: The output at the last layer corresponds to next parameter.
Revised: The output at the last layer corresponds to next parameter.

Original: At training and validation time, we use teacher forcing, i.e. , instead of using output of previous step, we use groundtruth sequences to train our model efﬁciently.
Revised: At training and validation time, we use teacher forcing, i.e. , instead of using output of previous step, we use groundtruth sequences to train our model efﬁciently.

Original: Loss.
Revised: Loss.

Original: We use a softmax layer to get probabilities if the next parameter is discrete.
Revised: We use a softmax layer to get probabilities if the next parameter is discrete.

Original: Instead of using a standard cross-entropy loss, we minimize KL-Divergence between softmax predictions and output one-hot distribution with Label Smoothing (Szegedy et al., 2016), which prevents the model from becoming overconﬁdent.
Revised: Instead of using a standard cross-entropy loss, we minimize KL-Divergence between softmax predictions and output one-hot distribution with Label Smoothing (Szegedy et al., 2016), which prevents the model from becoming overconﬁdent.

Original: If the next parameter is continuous, we use an L 1 loss.
Revised: If the next parameter is continuous, we use an L 1 loss.

Original: [Equation]
Revised: [Equation]

Original: 3D Primitive Auto-encoding.
Revised: 3D Primitive Auto-encoding.

Original: PartNet dataset (Yu et al., 2019) consists of 3D objects decomposed into simpler meaningful primitives, such as chairs are composed of back, arms, 4 legs, and so on.
Revised: PartNet dataset (Yu et al., 2019) consists of 3D objects decomposed into simpler meaningful primitives, such as chairs are composed of back, arms, 4 legs, and so on.

Original: We pose the problem of 3D shape generation as generating a layout of such primitives.
Revised: We pose the problem of 3D shape generation as generating a layout of such primitives.

Original: We use Chen & Zhang (2019)’s approach to ﬁrst encode voxel-based represent of primitive to d model -dimensional latent space using 3D CNN.
Revised: We use Chen & Zhang (2019)’s approach to ﬁrst encode voxel-based represent of primitive to d model -dimensional latent space using 3D CNN.

Original: An MLP based implicit parameter decoder projects the latent vector to the surface occupancy grid of the primitive.
Revised: An MLP based implicit parameter decoder projects the latent vector to the surface occupancy grid of the primitive.

Original: Order of primitives.
Revised: Order of primitives.

Original: One of the limitations of an autoregressive modeling approach is that sequence of primitives is an important consideration, in order to train the generative model, even if the layout doesn’t have a natural deﬁned order Vinyals et al. (2015).
Revised: One of the limitations of an autoregressive modeling approach is that sequence of primitives is an important consideration, in order to train the generative model, even if the layout doesn’t have a natural deﬁned order Vinyals et al. (2015).

Original: To generate a layout from any partial layout, we use a random permutation of primitives as input to the model.
Revised: To generate a layout from any partial layout, we use a random permutation of primitives as input to the model.

Original: For the output, we always generate the sequences in raster order of centroid of primitives, i.e. , we order the primitives in ascending order of their ( x, y, z ) coordinates.
Revised: For the output, we always generate the sequences in raster order of centroid of primitives, i.e. , we order the primitives in ascending order of their ( x, y, z ) coordinates.

Original: In our experiments, we observed that the ordering of elements is important for model training.
Revised: In our experiments, we observed that the ordering of elements is important for model training.

Original: Note that similar limitations are faced by contemporary works in layout generation (Jyothi et al., 2019;
Revised: Note that similar limitations are faced by contemporary works in layout generation (Jyothi et al., 2019;

Original: Li et al., 2019d; Hong et al., 2018; Wang et al., 2018a), image generation (Salimans et al., 2017; Gregor et al., 2015) and 3D shape generation (Wu et al., 2020;
Revised: Li et al., 2019d; Hong et al., 2018; Wang et al., 2018a), image generation (Salimans et al., 2017; Gregor et al., 2015) and 3D shape generation (Wu et al., 2020;

Original: Zou et al., 2017).
Revised: Zou et al., 2017).

Original: Generating a distribution over an order-invariant set of an arbitrary number of primitives is an exciting problem and we will explore it in future research.
Revised: Generating a distribution over an order-invariant set of an arbitrary number of primitives is an exciting problem and we will explore it in future research.

Original: Other details.
Revised: Other details.

Original: In our base model, we use d model = 512 , L = 6 , and n head = 8 (number of multiattention heads).
Revised: In our base model, we use d model = 512 , L = 6 , and n head = 8 (number of multiattention heads).

Original: Label smoothing uses an (cid:15) = 0 . 1 , and λ = 1 .
Revised: Label smoothing uses an (cid:15) = 0 . 1 , and λ = 1 .

Original: We use Adam optimizer (Kingma & Ba, 2014) with β 1 = 0 .
Revised: We use Adam optimizer (Kingma & Ba, 2014) with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 99 and learning rate 10 − 4 ( 10 − 5 for PartNet).
Revised: 99 and learning rate 10 − 4 ( 10 − 5 for PartNet).

Original: We use early stopping based on validation loss.
Revised: We use early stopping based on validation loss.

Original: In the ablation studies provided in § B, we show that our model is quite robust to these choices, as well as other hyperparameters (layout resolution, ordering of elements, ordering of ﬁelds).
Revised: In the ablation studies provided in § B, we show that our model is quite robust to these choices, as well as other hyperparameters (layout resolution, ordering of elements, ordering of ﬁelds).

Original: To sample a new layout, we can start off with just a start of sequence embedding or an initial set of primitives.
Revised: To sample a new layout, we can start off with just a start of sequence embedding or an initial set of primitives.

Original: Several decoding strategies are possible to recursively generate primitives from the initial set.
Revised: Several decoding strategies are possible to recursively generate primitives from the initial set.

Original: In samples generated for this work, unless otherwise speciﬁed, we have used nucleus sampling (Holtzman et al., 2019), with top- p = 0 . 9 which has been shown to perform better as compared to greedy sampling and beam search (Steinbiss et al., 1994).
Revised: In samples generated for this work, unless otherwise speciﬁed, we have used nucleus sampling (Holtzman et al., 2019), with top- p = 0 . 9 which has been shown to perform better as compared to greedy sampling and beam search (Steinbiss et al., 1994).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we discuss the qualitative and quantitative performance of our model on different datasets.
Revised: In this section, we discuss the qualitative and quantitative performance of our model on different datasets.

Original: Evaluation of generative models is hard, and most quantitative measures fail in providing a good measure of novelty and realism of data sampled from a generative model.
Revised: Evaluation of generative models is hard, and most quantitative measures fail in providing a good measure of novelty and realism of data sampled from a generative model.

Original: We will use datasetspeciﬁc quantitative metrics used by various baseline approaches and discuss their limitations wherever applicable.
Revised: We will use datasetspeciﬁc quantitative metrics used by various baseline approaches and discuss their limitations wherever applicable.

Original: We will provide the code and pretrained models to reproduce the experiments.
Revised: We will provide the code and pretrained models to reproduce the experiments.

Original: 4. 3D S HAPE SYNTHESIS ( ON P ART N ET DATASET )
Revised: 4. 3D S HAPE SYNTHESIS ( ON P ART N ET DATASET )

Original: PartNet is a large-scale dataset of common 3D shapes that are segmented into semantically meaningful parts.
Revised: PartNet is a large-scale dataset of common 3D shapes that are segmented into semantically meaningful parts.

Original: We use two of the largest categories of PartNet - Chairs and Lamp.
Revised: We use two of the largest categories of PartNet - Chairs and Lamp.

Original: We voxelize the shapes into 64 3 and train an autoencoder to learn part embeddings similar to the procedure followed by PQ-Net (Wu et al., 2020).
Revised: We voxelize the shapes into 64 3 and train an autoencoder to learn part embeddings similar to the procedure followed by PQ-Net (Wu et al., 2020).

Original: Overall, we had 6305 chairs and 1188 lamps in our datasets.
Revised: Overall, we had 6305 chairs and 1188 lamps in our datasets.

Original: We use the ofﬁcial train, validation, and test split from PartNet in our experiments.
Revised: We use the ofﬁcial train, validation, and test split from PartNet in our experiments.

Original: Although it is fairly trivial to extend our method to train for the class-conditional generation of shapes, in order to compare with baselines fairly, we train separate models for each of the categories.
Revised: Although it is fairly trivial to extend our method to train for the class-conditional generation of shapes, in order to compare with baselines fairly, we train separate models for each of the categories.

Original: Generated Samples.
Revised: Generated Samples.

Original: Fig.
Revised: Fig.

Original: 3 shows examples of shape completion from the PartNet dataset.
Revised: 3 shows examples of shape completion from the PartNet dataset.

Original: Given a random primitive, we use our model to iteratively predict the latent shape encoding of the next part,
Revised: Given a random primitive, we use our model to iteratively predict the latent shape encoding of the next part,

Original: [Table]
Revised: [Table]

Original: as well its position and scale in 3D. We then use the part decoder to sample points on the surface of the object.
Revised: as well its position and scale in 3D. We then use the part decoder to sample points on the surface of the object.

Original: For visualization, we use the marching cubes algorithm to generate a mesh and render the mesh using a ﬁxed camera viewpoint
Revised: For visualization, we use the marching cubes algorithm to generate a mesh and render the mesh using a ﬁxed camera viewpoint

Original: [Table]
Revised: [Table]

Original: Quantitative Evaluation.
Revised: Quantitative Evaluation.

Original: The output of our model is point clouds sampled on the surface of the 3D shapes.
Revised: The output of our model is point clouds sampled on the surface of the 3D shapes.

Original: We use Chamfer Distance (CD) and Earth Mover’s Distance (EMD) to compare two point clouds.
Revised: We use Chamfer Distance (CD) and Earth Mover’s Distance (EMD) to compare two point clouds.

Original: Following prior work, we use 4 different metrics to compare the distribution of shapes generated from the model and shapes in the test dataset: (i) Jensen Shannon Divergence (JSD) computes the KL divergence between marginal distribution of point clouds in generated set and test set, (ii) Coverage (Cov) - compares the distance between each point in generated set to its nearest neighbor in test set, (iii) Minimum Matching Distance (MMD) - computes the average distance of each point in test set to its nearest neighbor in generated set, and (iv) 1-nearest neighbor accuracy (1-NNA) uses a 1-NN classiﬁer see if the nearest neighbor of a generated sample is coming from generated set or test set.
Revised: Following prior work, we use 4 different metrics to compare the distribution of shapes generated from the model and shapes in the test dataset: (i) Jensen Shannon Divergence (JSD) computes the KL divergence between marginal distribution of point clouds in generated set and test set, (ii) Coverage (Cov) - compares the distance between each point in generated set to its nearest neighbor in test set, (iii) Minimum Matching Distance (MMD) - computes the average distance of each point in test set to its nearest neighbor in generated set, and (iv) 1-nearest neighbor accuracy (1-NNA) uses a 1-NN classiﬁer see if the nearest neighbor of a generated sample is coming from generated set or test set.

Original: Our model performs competitively with existing approaches to generate point clouds.
Revised: Our model performs competitively with existing approaches to generate point clouds.

Original: 4. L AYOUTS FOR NATURAL SCENES (COCO B OUNDING BOXES )
Revised: 4. L AYOUTS FOR NATURAL SCENES (COCO B OUNDING BOXES )

Original: COCO bounding boxes dataset is obtained using bounding box annotations in COCO Panopticdataset (Lin et al., 2014).
Revised: COCO bounding boxes dataset is obtained using bounding box annotations in COCO Panopticdataset (Lin et al., 2014).

Original: We ignore the images where the isCrowd ﬂag is true following the LayoutVAE (Jyothi et al., 2019) approach.
Revised: We ignore the images where the isCrowd ﬂag is true following the LayoutVAE (Jyothi et al., 2019) approach.

Original: The bounding boxes come from all 80 thing and 91 stuff categories.
Revised: The bounding boxes come from all 80 thing and 91 stuff categories.

Original: Our ﬁnal dataset has 118280 layouts from COCO train split with a median length of 42 elements and 5000 layouts from COCO valid split with a median length of 33 .
Revised: Our ﬁnal dataset has 118280 layouts from COCO train split with a median length of 42 elements and 5000 layouts from COCO valid split with a median length of 33 .

Original: We use the ofﬁcial validation split from COCO as test set in our experiments, and use 5% of the training data as validation.
Revised: We use the ofﬁcial validation split from COCO as test set in our experiments, and use 5% of the training data as validation.

Original: Baseline Approaches.
Revised: Baseline Approaches.

Original: We compare our work with 4 previous methods.
Revised: We compare our work with 4 previous methods.

Original: LayoutGAN (Li et al., 2019b) is a GAN based layout generation framework, starting with a noise vector sampled from gaussian distribution to generate a bounding box layours.
Revised: LayoutGAN (Li et al., 2019b) is a GAN based layout generation framework, starting with a noise vector sampled from gaussian distribution to generate a bounding box layours.

Original: Since the method always generate ﬁxed number of bounding boxes, it uses non-maximum suppression (NMS) to remove duplicates.
Revised: Since the method always generate ﬁxed number of bounding boxes, it uses non-maximum suppression (NMS) to remove duplicates.

Original: LayoutVAE (Jyothi et al., 2019) uses consists of two separate autoregressive VAE models.
Revised: LayoutVAE (Jyothi et al., 2019) uses consists of two separate autoregressive VAE models.

Original: The method assumes categories of elements present in a generated layout to be known.
Revised: The method assumes categories of elements present in a generated layout to be known.

Original: First, CountVAE generates counts of each of the elements of the label set, and then BoundingBoxVAE, generates the location and size of each occurrence of the bounding box.
Revised: First, CountVAE generates counts of each of the elements of the label set, and then BoundingBoxVAE, generates the location and size of each occurrence of the bounding box.

Original: ObjGAN (Li et al., 2019d) is a GAN framework for text to image synthesis.
Revised: ObjGAN (Li et al., 2019d) is a GAN framework for text to image synthesis.

Original: An intermediate step in their image synthesis approach is to generate a bounding box layout given a sentence using a BiLSTM (trained independently).
Revised: An intermediate step in their image synthesis approach is to generate a bounding box layout given a sentence using a BiLSTM (trained independently).

Original: We
Revised: We

Original: I n p u t L a y o u t
Revised: I n p u t L a y o u t

Original: C o m p l e t e d L a y o u t
Revised: C o m p l e t e d L a y o u t

Original: R ea l + L2 I m
Revised: R ea l + L2 I m

Original: O u r s + L

Revised: O u r s + L

Original: I m
Revised: I m

Original: L a y o u t
Revised: L a y o u t

Original: V A E + L

Revised: V A E + L

Original: I m adopt this step of the ObjGAN framework to our problem setup by provide categories of all layout elements as input to the ObjGAN and synthesize all the elements’ bounding boxes.
Revised: I m adopt this step of the ObjGAN framework to our problem setup by provide categories of all layout elements as input to the ObjGAN and synthesize all the elements’ bounding boxes.

Original: sg2im (Johnson et al., 2018) attempts to generate images given scene graph of the image by ﬁrst generating a layout of the scene using graph convolutions and then using the layout to generate complete scene using
Revised: sg2im (Johnson et al., 2018) attempts to generate images given scene graph of the image by ﬁrst generating a layout of the scene using graph convolutions and then using the layout to generate complete scene using

Original: GANs.
Revised: GANs.

Original: Since sg2im requires a scene graph input, following the approach of (Jyothi et al., 2019), we create a scene graph from the input and reproduce the input layout using the scene graph.
Revised: Since sg2im requires a scene graph input, following the approach of (Jyothi et al., 2019), we create a scene graph from the input and reproduce the input layout using the scene graph.

Original: Since LayoutVAE and LayoutGAN are not open source, we implemented our own version of these baseline.
Revised: Since LayoutVAE and LayoutGAN are not open source, we implemented our own version of these baseline.

Original: Note that, like many GAN models, LayoutGAN was notoriously hard to train and our implementation (and hence results) might differ from author’s implementation despite our best efforts.
Revised: Note that, like many GAN models, LayoutGAN was notoriously hard to train and our implementation (and hence results) might differ from author’s implementation despite our best efforts.

Original: We were able to reproduce LayoutVAE’s results on COCO dataset as proposed in the original paper and train our own models for different datasets using the same hyperparameters.
Revised: We were able to reproduce LayoutVAE’s results on COCO dataset as proposed in the original paper and train our own models for different datasets using the same hyperparameters.

Original: We also re-purpose
Revised: We also re-purpose

Original: ObjGAN and sg2im using guidelines mentioned in LayoutVAE.
Revised: ObjGAN and sg2im using guidelines mentioned in LayoutVAE.

Original: Although evaluating generative models is challenging, we attempt to do a fair comparison to the best of our abilities.
Revised: Although evaluating generative models is challenging, we attempt to do a fair comparison to the best of our abilities.

Original: For our model (and others), we keep architecture hyperparameters same across the datasets.
Revised: For our model (and others), we keep architecture hyperparameters same across the datasets.

Original: We also train different baselines for same number of epochs in corresponding datasets.
Revised: We also train different baselines for same number of epochs in corresponding datasets.

Original: Some of the ablation studies are provided in the appendix.
Revised: Some of the ablation studies are provided in the appendix.

Original: Generated Samples.
Revised: Generated Samples.

Original: Fig.
Revised: Fig.

Original: 4 shows layout completion task using our model on COCO dataset.
Revised: 4 shows layout completion task using our model on COCO dataset.

Original: Although the model is trained with all 171 categories, in the ﬁgure we only show ‘thing’ categories for clarity.
Revised: Although the model is trained with all 171 categories, in the ﬁgure we only show ‘thing’ categories for clarity.

Original: We also use the generated layouts for a downstream application of scene generation (Zhao et al., 2019).
Revised: We also use the generated layouts for a downstream application of scene generation (Zhao et al., 2019).

Original: Semantics Emerge via Layout.
Revised: Semantics Emerge via Layout.

Original: We posited earlier that capturing layout should capture contextual relationships between various elements.
Revised: We posited earlier that capturing layout should capture contextual relationships between various elements.

Original: We provide further evidence of our argument in Fig. 6.
Revised: We provide further evidence of our argument in Fig. 6.

Original: We visualize the 2D-tsne plot of the learned embeddings for categories.
Revised: We visualize the 2D-tsne plot of the learned embeddings for categories.

Original: We observe that supercategories from COCO are clustered together in the embedding space of the model.
Revised: We observe that supercategories from COCO are clustered together in the embedding space of the model.

Original: Certain categories such as window-blind and curtain (which belong to different super-categories) also appear close to each other.
Revised: Certain categories such as window-blind and curtain (which belong to different super-categories) also appear close to each other.

Original: These observations are in line with observations made by Gupta et al.
Revised: These observations are in line with observations made by Gupta et al.

Original: (2019) who use visual co-occurence to learn category embeddings.
Revised: (2019) who use visual co-occurence to learn category embeddings.

Original: Quantitative
Revised: Quantitative

Original: evaluation
Revised: evaluation

Original: Following the approach of LayoutVAE, we compute negative loglikelihoods (NLL) of all the layouts in validation data using importance sampling.
Revised: Following the approach of LayoutVAE, we compute negative loglikelihoods (NLL) of all the layouts in validation data using importance sampling.

Original: NLL approach is good for evaluating validation samples, but fails for generated samples.
Revised: NLL approach is good for evaluating validation samples, but fails for generated samples.

Original: Ideally, we would like to evaluate the performance of a generative model on a downstream task.
Revised: Ideally, we would like to evaluate the performance of a generative model on a downstream task.

Original: To this end, we employ Layout2Im (Zhao et al., 2019) to generate an image from the layouts generated by each of the method.
Revised: To this end, we employ Layout2Im (Zhao et al., 2019) to generate an image from the layouts generated by each of the method.

Original: We compute Inception Score (IS) and Fr´echet Inception Distance (FID) to compare quality and diversity of generated images.
Revised: We compute Inception Score (IS) and Fr´echet Inception Distance (FID) to compare quality and diversity of generated images.

Original: Our method is competitive with existing approaches in both these metrics, and outperforms existing approaches in terms of NLL.
Revised: Our method is competitive with existing approaches in both these metrics, and outperforms existing approaches in terms of NLL.

Original: Note that ObjGAN and LayoutVAE are conditioned on the label set.
Revised: Note that ObjGAN and LayoutVAE are conditioned on the label set.

Original: So we provide labels of objects present in the each validation layout as input.
Revised: So we provide labels of objects present in the each validation layout as input.

Original: The task for the model is to then predict the number and postition of these objects.
Revised: The task for the model is to then predict the number and postition of these objects.

Original: Hence, these methods have unfair advantage over our method and
Revised: Hence, these methods have unfair advantage over our method and

Original: ObjGAN indeed performs better than our method and LayoutGAN, which are unconditional.
Revised: ObjGAN indeed performs better than our method and LayoutGAN, which are unconditional.

Original: We clearly outperform LayoutGAN on IS and FID metrics.
Revised: We clearly outperform LayoutGAN on IS and FID metrics.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: I n p u t L a y o u t
Revised: I n p u t L a y o u t

Original: C o m p l e t e d
Revised: C o m p l e t e d

Original: L a y o u t
Revised: L a y o u t

Original: O u r s L a y o u t V A E
Revised: [Figure]

Original: 4. M OBILE APP WIREFRAMES (RICO) AND D OCUMENT LAYOUTS (P UB L AY N ET )Rico
Revised: 4. M OBILE APP WIREFRAMES (RICO) AND D OCUMENT LAYOUTS (P UB L AY N ET )

Original:  Mobile App Wireframes.
Revised: Rico Mobile App Wireframes.

Original: Rico mobile app dataset (Deka et al., 2017; Liu et al., 2018) consists of layout information of more than 66000 unique UI screens from over 9300 android apps.
Revised: Rico mobile app dataset (Deka et al., 2017; Liu et al., 2018) consists of layout information of more than 66000 unique UI screens from over 9300 android apps.

Original: Each layout consists of one or more of the 25 categories of graphical elements such as text, image, icon etc.
Revised: Each layout consists of one or more of the 25 categories of graphical elements such as text, image, icon etc.

Original: A complete list of these elements is provided in the supplementary material.
Revised: A complete list of these elements is provided in the supplementary material.

Original: Overall, we get 62951 layouts in Rico with a median length of 36.
Revised: Overall, we get 62951 layouts in Rico with a median length of 36.

Original: Since the dataset doesn’t have ofﬁcial splits, we use 5% of randomly selected layouts for validation and 15% for testing.
Revised: Since the dataset doesn’t have ofﬁcial splits, we use 5% of randomly selected layouts for validation and 15% for testing.

Original: PubLayNet.
Revised: PubLayNet.

Original: PubLayNet (Zhong et al., 2019) is a large scale document dataset consisting of over 1 .million articles collected from PubMed Central.
Revised: PubLayNet (Zhong et al., 2019) is a large scale document dataset consisting of over 1 .million articles collected from PubMed Central.

Original: The layouts are annotated with 5 element categories - text, title, list, label, and ﬁgure.
Revised: The layouts are annotated with 5 element categories - text, title, list, label, and ﬁgure.

Original: We ﬁlter out the document layouts with over 128 elements.
Revised: We ﬁlter out the document layouts with over 128 elements.

Original: Our ﬁnal dataset has 335703 layouts from PubLayNet train split with a median length of 33 elements and 11245 layouts from PubLayNet dev split with a median length of 36 .
Revised: Our ﬁnal dataset has 335703 layouts from PubLayNet train split with a median length of 33 elements and 11245 layouts from PubLayNet dev split with a median length of 36 .

Original: We use the provided dev split as our test set and 5% of the training data for validation.
Revised: We use the provided dev split as our test set and 5% of the training data for validation.

Original: Generated layout samples.
Revised: Generated layout samples.

Original: Fig.
Revised: Fig.

Original: 8 and 9 shows some of the generated samples of our model from RICO mobile app wireframes and PubLayNet documents.
Revised: 8 and 9 shows some of the generated samples of our model from RICO mobile app wireframes and PubLayNet documents.

Original: Note that both the datasets share similarity in terms of distribution of elements, such as high coverage in terms of space, very little collision of elements, and most importantly alignment of the elements along both x and y-axes.
Revised: Note that both the datasets share similarity in terms of distribution of elements, such as high coverage in terms of space, very little collision of elements, and most importantly alignment of the elements along both x and y-axes.

Original: Our method is able to preserve most of these properties as we discuss in the next section.
Revised: Our method is able to preserve most of these properties as we discuss in the next section.

Original: Fig.
Revised: Fig.

Original: 10 shows multiple completions done by our model for the same initial element.
Revised: 10 shows multiple completions done by our model for the same initial element.

Original: Comparison with baselines.
Revised: Comparison with baselines.

Original: We use the same baselines for evaluation as discussed previously in § 4.2.
Revised: We use the same baselines for evaluation as discussed previously in § 4.2.

Original: Fig.
Revised: Fig.

Original: 9 shows that our method is able to preserve alignment between bounding boxes better
Revised: 9 shows that our method is able to preserve alignment between bounding boxes better

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: than competing methods.
Revised: than competing methods.

Original: Note that we haven’t used any post-processing in order to generate these layouts.
Revised: Note that we haven’t used any post-processing in order to generate these layouts.

Original: Our hypothesis is that (1) discretization of size/position, and (2) decoupling geometric ﬁelds in the attention module, are particularly useful in datasets with aligned boxes
Revised: Our hypothesis is that (1) discretization of size/position, and (2) decoupling geometric ﬁelds in the attention module, are particularly useful in datasets with aligned boxes

Original: [Table]
Revised: [Table]

Original: To measure this performance quantitatively, we introduce 2 important statistics.
Revised: To measure this performance quantitatively, we introduce 2 important statistics.

Original: Overlap represents the intersection over union (IoU) of various layout elements.
Revised: Overlap represents the intersection over union (IoU) of various layout elements.

Original: Generally in these datasets, elements do not overap with each other and Overlap is small.
Revised: Generally in these datasets, elements do not overap with each other and Overlap is small.

Original: Coverage indicates the percentage of canvas covered by the layout elements.
Revised: Coverage indicates the percentage of canvas covered by the layout elements.

Original: Table 2 shows that layouts generated by our method resemble real data statistics better than LayoutGAN and LayoutVAE.
Revised: Table 2 shows that layouts generated by our method resemble real data statistics better than LayoutGAN and LayoutVAE.

Original:  C ONCLUSION .
Revised:  C ONCLUSION .

Original: We propose MMA, a multimodal attention framework to generate layouts of graphical elements.
Revised: We propose MMA, a multimodal attention framework to generate layouts of graphical elements.

Original: Our model uses self-attention model to capture contextual relationship between different layout elements and generate novel layouts, or complete partial layouts.
Revised: Our model uses self-attention model to capture contextual relationship between different layout elements and generate novel layouts, or complete partial layouts.

Original: We show that our model performs competitively with the state-of-the-art approaches for very diverse datasets such as Rico Mobile
Revised: We show that our model performs competitively with the state-of-the-art approaches for very diverse datasets such as Rico Mobile

Original: App Wireframes, COCO bounding boxes, PubLayNet documents, and 3D shapes.
Revised: App Wireframes, COCO bounding boxes, PubLayNet documents, and 3D shapes.

Original: There are a few limitations of our approach.
Revised: There are a few limitations of our approach.

Original: First, our model requires a layout or a scene to be decomposed into compositional primitives.
Revised: First, our model requires a layout or a scene to be decomposed into compositional primitives.

Original: In many cases, such primitives might not be even deﬁned.
Revised: In many cases, such primitives might not be even deﬁned.

Original: Second, like most data-driven approaches, generated layouts are dominated by high frequency objects or shapes in the dataset.
Revised: Second, like most data-driven approaches, generated layouts are dominated by high frequency objects or shapes in the dataset.

Original: We can control the diversity to some extent using improved sampling techniques, however, generating diverse layouts that not only learn from data, but also from human priors or pre-deﬁned rules is an important direction of research which we will continue to explore.
Revised: We can control the diversity to some extent using improved sampling techniques, however, generating diverse layouts that not only learn from data, but also from human priors or pre-deﬁned rules is an important direction of research which we will continue to explore.

Original: Appendix A A RCHITECTURE AND TRAINING DETAILS
Revised: Appendix A A RCHITECTURE AND TRAINING DETAILS

Original: In all our R 2 experiments, our base model consists of d model = 512 , L = 6 , n head = 8 , precision =and d ff = 2048 .
Revised: In all our R 2 experiments, our base model consists of d model = 512 , L = 6 , n head = 8 , precision =and d ff = 2048 .

Original: We also use a dropout of 0 .
Revised: We also use a dropout of 0 .

Original: 1 at the end of each feedforward layer for regularization.
Revised: 1 at the end of each feedforward layer for regularization.

Original: We ﬁx the the maximum number of elements in each of the datasets to 128 which covers over 99.9% of the layouts in each of the COCO, Rico and PubLayNet datasets.
Revised: We ﬁx the the maximum number of elements in each of the datasets to 128 which covers over 99.9% of the layouts in each of the COCO, Rico and PubLayNet datasets.

Original: We also used Adam optimizer Kingma & Ba (2014) with initial learning rate of 10 − 4 .
Revised: We also used Adam optimizer Kingma & Ba (2014) with initial learning rate of 10 − 4 .

Original: We train our model forepochs for each dataset with early stopping based on maximum log likelihood on validation layouts.
Revised: We train our model forepochs for each dataset with early stopping based on maximum log likelihood on validation layouts.

Original: Our COCO Bounding Boxes model takes about 1 day to train on a single NVIDIA GTX1080 GPU.
Revised: Our COCO Bounding Boxes model takes about 1 day to train on a single NVIDIA GTX1080 GPU.

Original: Batching matters a lot to improve the training speed.
Revised: Batching matters a lot to improve the training speed.

Original: We want to have evenly divided batches, with absolutely minimal padding.
Revised: We want to have evenly divided batches, with absolutely minimal padding.

Original: We sort the layouts by the number of elements and search over this sorted list to use ﬁnd tight batches for training.
Revised: We sort the layouts by the number of elements and search over this sorted list to use ﬁnd tight batches for training.

Original: In all our R 3 experiments, we change d model = 128 , and learning rate to 10 − 5 .
Revised: In all our R 3 experiments, we change d model = 128 , and learning rate to 10 − 5 .

Original: B A BLATION STUDIES
Revised: B A BLATION STUDIES

Original: We evaluate the importance of different model components with negative log-likelihood on COCO layouts.
Revised: We evaluate the importance of different model components with negative log-likelihood on COCO layouts.

Original: The ablation studies show the following:
Revised: The ablation studies show the following:

Original: Small, medium and large elements: NLL of our model for COCO large, medium, and small boxes is 2.4, 2.5, and 1.8 respectively.
Revised: Small, medium and large elements: NLL of our model for COCO large, medium, and small boxes is 2.4, 2.5, and 1.8 respectively.

Original: We observe that even though discretizing box coordinates introduces approximation errors, it later allows our model to be agnostic to large vs small objects.
Revised: We observe that even though discretizing box coordinates introduces approximation errors, it later allows our model to be agnostic to large vs small objects.

Original: Varying precision: Increasing it allows us to generate ﬁner layouts but at the expense of a model with more parameters.
Revised: Varying precision: Increasing it allows us to generate ﬁner layouts but at the expense of a model with more parameters.

Original: Also, as we increase the precision, NLL increases, suggesting that we might need to train the model with more data to get similar performance (Table 3).
Revised: Also, as we increase the precision, NLL increases, suggesting that we might need to train the model with more data to get similar performance (Table 3).

Original: Size of embedding: Increasing the size of the embedding d model improves the NLL, but at the cost of increased number of parameters (Table 4).
Revised: Size of embedding: Increasing the size of the embedding d model improves the NLL, but at the cost of increased number of parameters (Table 4).

Original: Model depth: Increasing the depth of the model L , does not signiﬁcantly improve the results (Table 5).
Revised: Model depth: Increasing the depth of the model L , does not signiﬁcantly improve the results (Table 5).

Original: We ﬁx the L = 6 in all our experiments.
Revised: We ﬁx the L = 6 in all our experiments.

Original: Ordering of the elements: Adding position encoding, makes the self-attention layer dependent to the ordering of elements.
Revised: Ordering of the elements: Adding position encoding, makes the self-attention layer dependent to the ordering of elements.

Original: In order to make it depend less on the ordering of input elements, we take randomly permute the sequence.
Revised: In order to make it depend less on the ordering of input elements, we take randomly permute the sequence.

Original: This also enables our model to be able to complete any partial layout.
Revised: This also enables our model to be able to complete any partial layout.

Original: Since output is predicted sequentially, our model is not invariant to the order of output sequence also.
Revised: Since output is predicted sequentially, our model is not invariant to the order of output sequence also.

Original: In our experiments, we observed that predicting the elements in a simple raster scan order of their position improves the model performance both visually and in terms of negative log-likelihood.
Revised: In our experiments, we observed that predicting the elements in a simple raster scan order of their position improves the model performance both visually and in terms of negative log-likelihood.

Original: This is intuitive as ﬁlling the elements in a pre-deﬁned order is an easier problem.
Revised: This is intuitive as ﬁlling the elements in a pre-deﬁned order is an easier problem.

Original: We leave the task of optimal ordering of layout elements to generate layouts for future research.
Revised: We leave the task of optimal ordering of layout elements to generate layouts for future research.

Original: (Table 6).
Revised: (Table 6).

Original: Discretization strategy: Instead of the factorizing location in x-coordinates and y-coordinates, we tried predicting them at once (refer to the Split-xy column of Table 6).
Revised: Discretization strategy: Instead of the factorizing location in x-coordinates and y-coordinates, we tried predicting them at once (refer to the Split-xy column of Table 6).

Original: This increases the vocabulary size of the model (since we use H × H possible locations instead of H alone) and in turn the number of hyper-parameters with decline in model performance.
Revised: This increases the vocabulary size of the model (since we use H × H possible locations instead of H alone) and in turn the number of hyper-parameters with decline in model performance.

Original: An upside of this approach is that generating new layouts takes less time as we have to make half as many predictions for each element of the layout (Table 6).
Revised: An upside of this approach is that generating new layouts takes less time as we have to make half as many predictions for each element of the layout (Table 6).

Original: Loss: We tried two different losses, label smoothing (M¨uller et al., 2019) and NLL.
Revised: Loss: We tried two different losses, label smoothing (M¨uller et al., 2019) and NLL.

Original: Although optimizing using NLL gives better validation performance in terms of NLL (as is expected), we do not ﬁnd much difference in the qualitative performance when using either loss function.
Revised: Although optimizing using NLL gives better validation performance in terms of NLL (as is expected), we do not ﬁnd much difference in the qualitative performance when using either loss function.

Original: (Table 6)
Revised: (Table 6)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: C V ISUALIZING ATTENTION
Revised: C V ISUALIZING ATTENTION

Original: The self-attention based approach proposed enables us to visualize which existing elements are being attending to while the model is generating a new element.
Revised: The self-attention based approach proposed enables us to visualize which existing elements are being attending to while the model is generating a new element.

Original: This is demonstrated in Figure
Revised: This is demonstrated in Figure

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D L AYOUT V ERIFICATION
Revised: D L AYOUT V ERIFICATION

Original: Since in our method it is straightforward to compute likelihood of a layout
Revised: Since in our method it is straightforward to compute likelihood of a layout

Original: [Table]
Revised: [Table]

Original: we can use our method to test if a given layout is likely or not.
Revised: we can use our method to test if a given layout is likely or not.

Original: Figure 12 shows the NLL given by our model by doing leftright and top-down inversion of layouts in COCO (following Li et al.
Revised: Figure 12 shows the NLL given by our model by doing leftright and top-down inversion of layouts in COCO (following Li et al.

Original: (2019b)).
Revised: (2019b)).

Original: In case of COCO, if we ﬂip a layout left-right, we observe that layout remains likely, however ﬂipping the layout upside decreases the likelihood (or increases the NLL of the layout).
Revised: In case of COCO, if we ﬂip a layout left-right, we observe that layout remains likely, however ﬂipping the layout upside decreases the likelihood (or increases the NLL of the layout).

Original: This is intuitive since it is unlikely to see fog in the bottom of an image, while skis on top of a person.
Revised: This is intuitive since it is unlikely to see fog in the bottom of an image, while skis on top of a person.

Original: E
Revised: E

Original: M
Revised: M

Original: ORE SEMANTICS IN LEARNED CATEGORY EMBEDDINGS
Revised: ORE SEMANTICS IN LEARNED CATEGORY EMBEDDINGS

Original: Table 7 captures the most frequent bigrams and trigrams (categories that co-occur) in real and synthesized layouts.
Revised: Table 7 captures the most frequent bigrams and trigrams (categories that co-occur) in real and synthesized layouts.

Original: Table 8 shows word2vec
Revised: Table 8 shows word2vec

Original: (Mikolov et al., 2013) style analogies being captured by embeddings learned by our model.
Revised: (Mikolov et al., 2013) style analogies being captured by embeddings learned by our model.

Original: Note that the model was trained to generate layouts and we did not specify any additional objective function for analogical reasoning task.
Revised: Note that the model was trained to generate layouts and we did not specify any additional objective function for analogical reasoning task.

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Bbox center (x) sky-other airplane clouds person sea road car
Revised: Bbox center (x) sky-other airplane clouds person sea road car

Original: [Figure]
Revised: [Figure]

Original: F D ATASET S TATISTICS
Revised: F D ATASET S TATISTICS

Original: In this section, we share statistics of different elements and their categories in our dataset.
Revised: In this section, we share statistics of different elements and their categories in our dataset.

Original: In particular, we share the total number of occurrences of an element in the trai ning dataset (in descending
Revised: In particular, we share the total number of occurrences of an element in the trai ning dataset (in descending

Original: [Table]
Revised: [Table]

Original: order) and the total number of distinct layouts an element was present in throughout the training data
Revised: order) and the total number of distinct layouts an element was present in throughout the training data

Original: [Table]
Revised: [Table]

Original: Tables 9, 9 show the statistics for Rico wireframes, and table 10 show the statistics for PubLayNet documents
Revised: Tables 9, 9 show the statistics for Rico wireframes, and table 10 show the statistics for PubLayNet documents

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: G C OORDINATE E MBEDDING
Revised: G C OORDINATE E MBEDDING

Original: Just like in Fig.
Revised: Just like in Fig.

Original: 6, we project the embedding learned by our model on COCO in a 2-d space using
Revised: 6, we project the embedding learned by our model on COCO in a 2-d space using

Original: TSNE.
Revised: TSNE.

Original: In the absence of explicit constraints on the learned embedding, the model learns to cluster together all the coordinate embedding in a distinct space, in a ring-like manner.
Revised: In the absence of explicit constraints on the learned embedding, the model learns to cluster together all the coordinate embedding in a distinct space, in a ring-like manner.

Original: H C OMPARISON WITH P OLY G EN
Revised: H C OMPARISON WITH P OLY G EN

Original: We would like to highlight some similarities of our framework with the recently proposed autoregressive generative model for 3D meshes, PolyGen Nash et al.
Revised: We would like to highlight some similarities of our framework with the recently proposed autoregressive generative model for 3D meshes, PolyGen Nash et al.

Original: While both works adopt
Revised: While both works adopt

Original: Transformer Decoder for autoregressive modeling, the usage differs in the following aspects:
Revised: Transformer Decoder for autoregressive modeling, the usage differs in the following aspects:

Original: • PolyGen models mesh vertices as nodes.
Revised: • PolyGen models mesh vertices as nodes.

Original: Advantage of this method is that it allows modelling high resolution 3D objects.
Revised: Advantage of this method is that it allows modelling high resolution 3D objects.

Original: However challenge is that sequence lengths for high resolution meshes can be very high and it can be very difﬁcult to model them using self-attention (whose memory requirements grow proportionally to the square of sequence length).
Revised: However challenge is that sequence lengths for high resolution meshes can be very high and it can be very difﬁcult to model them using self-attention (whose memory requirements grow proportionally to the square of sequence length).

Original: • We on the other hand separate out attributes (not just coordinates but also height, width, category and (or) SDF encoding) of parts of 3D objects which are typically fewer in number.
Revised: • We on the other hand separate out attributes (not just coordinates but also height, width, category and (or) SDF encoding) of parts of 3D objects which are typically fewer in number.

Original: Deep Networks based SDF encoding are an active area of research and the current SOTA methods don’t provide high resolution results as mesh based methods.
Revised: Deep Networks based SDF encoding are an active area of research and the current SOTA methods don’t provide high resolution results as mesh based methods.

Original: • Our model predicts future elements in order, but we randomize the order of the input elements.
Revised: • Our model predicts future elements in order, but we randomize the order of the input elements.

Original: This allows us to do partial layout completion.
Revised: This allows us to do partial layout completion.

Original: I N EAREST NEIGHBORS
Revised: I N EAREST NEIGHBORS

Original: To see if our model is memorizing the training dataset, we compute nearest neighbors of generated layouts using chamfer distance on top-left and bottom-right bounding box coordinates of layout elements.
Revised: To see if our model is memorizing the training dataset, we compute nearest neighbors of generated layouts using chamfer distance on top-left and bottom-right bounding box coordinates of layout elements.

Original: Figure 15 shows the nearest neighbors of some of the generated layouts from the training dataset.
Revised: Figure 15 shows the nearest neighbors of some of the generated layouts from the training dataset.

Original: We note that nearest neighbor search for layouts is an active area of research.
Revised: We note that nearest neighbor search for layouts is an active area of research.

Original: Generated
Revised: Generated

Original: [Figure]
Revised: [Figure]

Revised: NN

Revised: J M ORE EXAMPLES FOR L AYOUT TO I MAGE

Revised: Layouts for natural scenes are cluttered and hard to qualitatively evaluate even for a trained user.

Revised: Here we share some more sample layouts generated from two methods used in the paper.

Revised: Figureshows some extra sample layouts and corresponding images generated using Layout2Im tool.

Revised: Existing layout to image methods don’t work as well as free-form image generation methods but are arguably more beneﬁcial in downstream applications.

Revised: We hope that improving layout generation will aid the research community to develop better scene generation tools both in terms of diversity and quality.

Revised: [Figure]