T ell2 D esign: A Dataset for Language-Guided Floor Plan Generation (2023)

Abstract: We consider the task of generating designs directly from natural language descriptions, and consider floor plan generation as the initial research area. Language conditional generative models have recently been very successful in generating high-quality artistic images. However, designs must satisfy different constraints that are not present in generating artistic images, particularly spatial and relational constraints. We make multiple contributions to initiate research on this task. First, we introduce a novel dataset, Tell2Design (T2D), which contains more than 80k floor plan designs associated with natural language instructions. Second, we propose a Sequence-to-Sequence model that can serve as a strong baseline for future research. Third, we benchmark this task with several text-conditional image generation models. We conclude by conducting human evaluations on the generated samples and providing an analysis of human performance. We hope our contributions will propel the research on language-guided design generation forward.

## introduction recently, text-conditional generative ai models @xcite @xcite @xcite have demonstrated impressive results in generating high-fidelity images. such models generally focus on understanding high-level visual concepts from sentence-level descriptions, and the generated images are valued for looking realistic and being creative, thereby being more suitable for generating artwork. however, besides less constrained generation like artworks, generating designs that meet various requirements specified the balcony, located on the eastern side of the unit, north of the bathroom, juts out from the rest of the unit. it is 4' wide from west to east, and the entire balcony is located eastward of the easternmost wall of the bedroom, common area, and bathroom. the balcony spans 11 ' from north to south, and its northern wall is shared with the southern wall of the kitchen. the kitchen and balcony both jut out approximately 4' to the east of the unit. the bathroom is located on the central-eastern side of the unit between the common area and the master bedroom. while the northern wall of the master bedroom and the southern wall of the common area are both roughly 12' wide ... ## related work text-conditioned image generation image generation is a well-studied problem, and the most popular techniques have been applied for both unconditional image generation and text-conditional settings. early works apply auto-regressive models @xcite , or train gans @xcite @xcite @xcite with publicly available image captioning datasets to synthesize realistic images conditioned on sentence-level captions. other works have adopted the vq-vae technique @xcite to text-conditioned image generation by concatenating sequences of text tokens with image tokens and feeding them into autoregressive transformers @xcite @xcite . more recently, some works have applied diffusion models @xcite @xcite @xcite @xcite @xcite and received wide success in image generation, outperforming other approaches in fidelity and diversity, without training instability and mode collapse issues @xcite @xcite . however, these models operate on extracting high-level visual concepts from the short text and produce artwork-like images that are expected to be realistic and creative, thereby not suitable for generating designs that must satisfy various user/client requirements. layout generation layout generation is essentially a design process that requires meeting domain-specific constraints, where the desirable layouts could be documents, natural scenes, mobile phone uis, and indoor scenes. for example, pub-laynet @xcite is proposed to generate machine-annotated scientific documents with five different element categories text, title, figure, list, and table. rico @xcite is introduced to develop user interface designs for mobile applications, which contains button, toolbar, etc. sun rgb-d @xcite presents a combined scene-understanding task, including indoor scenes from three other datasets @xcite @xcite . moreover, icvt @xcite aims to produce advertisement poster layouts automatically, where the image background is given as input. the above methods are designed for different layout domains and cannot be directly applied to floor plan design. moreover, none of them has considered generating the layout design directly from languages. floor plan generation several methods have been proposed to generate floor plan designs automatically @xcite @xcite @xcite @xcite . most of these methods generate floor plans conditioned on certain constraints, such as room types, adjacencies, and boundaries. for example, merrell et al. (2010) generate buildings with interior floor plans for computer graphics applications using bayesian networks without considering any human preferences. liu et al. (2013) present an interactive tool to generate desired floor plan following a set of manually defined rules. hua (2016) particularly focus on generating floor plans with irregular regions. wu et al. (2018) cast the generation as a mixed integer quadratic programming problem where some floor plan components are formulated into a set of inequality constraints. more recently, wu et al. (2019) propose a cnn-based method to determine the location of different rooms given boundary images as a constraint. chen et al. (2020) provide a small amount of template-based artificial verbal commands and manually parses them to scene graphs for guiding the generation. in summary, existing methods represent the intrinsic components of floor plans in several specific formats as generation constraints. some formats are straightforward, such as boundary images, but they only specify limited constraints and lead to less controllable generation. other formats, such as scene graphs and inequalities, can incorporate more information but require specific domain knowledge and extra-human efforts in pre-processing. we instead provide a unified and natural way of conditioning the floor plan generation with a set of language instructions that is much more flexible and user-friendly to characterize floor plans with various constraints. ## tell2design dataset in this section, we introduce how we construct our t2d dataset, followed by the data analysis and a discussion of the main dataset challenges. ## conclusion in this paper, we initiate the research of a novel language-guided design generation task, with a specific focus on the floor plan domain as a start. we formulate it as language-guided floor plan generation and introduce tell2design (t2d), a large-scale dataset that features floor plans with natural language instructions in describing user preferences. we propose a seq2seq model as a strong baseline and compare it with several text-conditional image generation models. experimental results demonstrate that the design generation task brings up several challenges and is not well-solved by existing text-conditional image generation techniques. human evaluations assessing the degree of alignment between text and design, along with the human performance on the task, expose the challenge of understanding fuzzy and entangled information, and the nature of design diversity in our task. we hope this paper will serve as a foundation and propel future research on the task of the language-guided design generation.