Automatic Layout Planning for Visually-Rich Documents with Instruction-Following Models (2024)

Abstract: Recent advancements in instruction-following models have made user interactions with models more user-friendly and efficient, broadening their applicability. In graphic design, non-professional users often struggle to create visually appealing layouts due to limited skills and resources. In this work, we introduce a novel multimodal instruction-following framework for layout planning, allowing users to easily arrange visual elements into tailored layouts by specifying canvas size and design purpose, such as for book covers, posters, brochures, or menus. We developed three layout reasoning tasks to train the model in understanding and executing layout instructions. Experiments on two benchmarks show that our method not only simplifies the design process for non-professionals but also surpasses the performance of few-shot GPT-4V models, with mIoU higher by 12% on Crello. This progress highlights the potential of multimodal instruction-following models to automate and simplify the design process, providing an approachable solution for a wide range of design tasks on visually-rich documents.

## introduction the creation of visually-rich documents (e.g., posters, brochures, book covers, digital advertisements, etc) using available visual components, poses a significant challenge for both professionals and amateurs in the design field. central to this challenge is the task of arranging these components in an efficient and aesthetically pleasing manner, a process known to be both tedious and time-consuming. existing toolkits such as adobe express 1 , canva 2 , and picsart 3 , usually provide fixed templates to users. these templates, while useful, often fail to fully accommodate the varied and evolving design needs of users, thereby 1 https://www.adobe.com/express/ 2 https://www.canva.com/ 3 https://picsart.com/ potentially limiting creative expression. existing research on automatic layout planning @xcite @xcite often requires detailed annotations and poses addition constraints on fixed canvas ratios, thereby diminishing user-friendliness and adaptability. recent advancements in large language models (llms) have showcased their remarkable ability to follow human instructions and execute specified tasks @xcite @xcite , introducing a new level of flexibility and control in human-computer interaction. alongside these developments, we have witnessed the emergence of instruction-tuned multimodal models @xcite @xcite , extending the capabilities of llms to understand and process information across both textual and visual domains. this progression naturally raises the question of the potential application of instruction-following models in the complex domain of multimodal layout planning. however, employing these models for layout planning presents significant challenges, as the task requires intricate reasoning abilities, including but not limited to, cross-referencing multiple images and performing numerical calculations. ## results & analysis quantitative results effects of #component figure 3 (a) reveals that all listed models exhibit high miou for templates with a single component. flexdm's miou shows slight fluctuations, stabilizing around 50%. in contrast, miou for doclap and gpt-4(v) decreases as the number of components increases, indicating that more complex scenarios involving more visual components might pose challenges to current instruction-following models. effects of component size figure 3 (b) demonstrates a linear correlation between the relative size of a single visual component and the iou of the model prediction with the ground truth for all models assessed. this suggests that smaller visual components pose a greater challenge for precise placement in accordance with the ground truth during layout planning. typically, these small components, such as logos, small text boxes, or decoraground-truth ours gpt4v flexdm ground-truth ours gpt4v ds-gan figure 5: qualitative comparisons for layout planning results on posterlayout. gpt-4v w/ 1-shot learning. demonstrative examples figure 4 shows examples from crello while figure 5 shows examples from posterlayout.