T ext L ap: Customizing Language Models for Text-to-Layout Planning (2024)

Abstract: Automatic generation of graphical layouts is crucial for many real-world applications, including designing posters, flyers, advertisements, and graphical user interfaces. Given the incredible ability of Large language models (LLMs) in both natural language understanding and generation, we believe that we could customize an LLM to help people create compelling graphical layouts starting with only text instructions from the user. We call our method TextLap (text-based layout planning). It uses a curated instruction-based layout planning dataset (InsLap) to customize LLMs as a graphic designer. Human annotators are asked to build a benchmark to evaluate different layout planning models. We demonstrate the effectiveness of TextLap and show that it outperforms strong baselines, including GPT-4 based methods, for document generation and graphical design benchmarks.

## introduction traditional layout generation tasks typically involve organizing text and graphical elements into a pleasing and professional looking arrangement. this is often time-consuming and difficult for users, especially those without training or design skill. our goal is to condition an llm so that it allows users to generate professional looking layouts by simply inputting text instructions. we are inspired by the rapid advancement of generative models for text @xcite , images @xcite , and videos @xcite and recent works that create layouts using graphical elements such as category, position, or size as input @xcite @xcite . we advance the field by customizing llms as text-guided layout generation models that require only a text description of the desired layout and optionally text descriptions for visual elements as input, enabling layout planning solely in the text modality. we believe that building an llm-based layout generation offers a more user-friendly approach to achieving desired designs, allowing text instructions to guide the process-an aspect that is challenging to incorporate in traditional settings. creating 2d graphical layouts from text alone is challenging for several reasons. first, text descriptions of 2d arrangements are usually underspecified. there are multiple ways to describe a 2d layout, and there are multiple 2d layouts that can meet a description. furthermore, comprehending 2d spatial relations @xcite is challenging for llms due to their autoregressive nature and 1d positional embedding @xcite . even understanding the meaning of coordinate numbers in a description is sometimes difficult for llms. to enable the generation of 2d designs from text, we build an instruction-based layout planning (in-stlap) dataset to customize large language models. we then use instlap to train our text-based layout planning (textlap) model through supervised fine-tuning of an llm, enabling it to understand 2d spatial relationships and generate or modify bounding box coordinates based on text prompts. figure 1 illustrates example inputs and outputs of the proposed textlap model. as an llm, textlap enables users to iteratively refine layout designs through natural language conversations. initial text instructions provide an initial draft, and users can further customize their designs by interacting with the model. textlap generates text-coherent layouts in response to users' iterative requests, enabling them to quickly find the most suitable custom template for their design. we believe that textlap significantly reduces the time required to create designs, enhances design efficiency, and plays an important role in assisting graphic designers. moreover, textlap also serves as a layout planning component for image generators, greatly enhancing the layout coherence of generated images according to text prompts. our contributions are threefold: • we proposed a novel text-to-layout tasks aiming to enhance design efficiency and designed evaluation dataset and protocals. ## creating the instlap dataset to customize a language model for layout planning, we developed the instruction to layout planning (instlap) dataset which comprises filtered and enhanced image-caption pairs from ms-coco (2017) @xcite , a subset of the laion datset @xcite , and crelllo @xcite . ## model training we construct the textlap model by fine-tuning the vicuna-1.5 7b @xcite model using the fastchat framework @xcite . the model is trained on the instlap-train set with both 4 14278 object layouts and text layouts. figure 2 presents examples of conversations from the instlap-train dataset. we train the model using both the closedset layout prompt and the prompt enhanced with object augmentations. the training aims to enable the model to extract objects and separate visualtext elements when no specific target is provided. additionally, the model is trained to focus on userspecified keywords or modification instructions. training details are included in appendix b.1. we use multiple text formats for layout representation, including lists of integer/float numbers and css structures. however, we notice that multidigit coordinates are often tokenized into multiple tokens, potentially complicating the model's numerical understanding. we integrated discrete coordinates as single tokens into the tokenizer to mitigate this. ## conclusion this study addresses the text-to-layout task by creating an instructional dataset built upon available resources with the assistance of gpt-4v to finetune a large language model for layout planning. the fine-tuned model, textlap, outperforms gpt-4 in object layout planning and can generate layouts with both text and object when trained on textonly and object-only layouts. textlap provides a framework to addressing real-world graphic design challenges by building instruction-following layout datasets. it is desired if the image rendering models can be finetuned or jointly trained with instlap. however, it is beyond the scope of this paper as instlap aims to unveiling the graphic design ability of large language models.