A Bounding Box is Worth One Token - Interleaving Layout and Text in a Large Language Model for Document Understanding (2025)

Abstract: Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Layout andText in a Large Language Model (LayTextLLM) for document understanding. LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in KIE and VQA. Comprehensive benchmark evaluations reveal significant improvements of LayTextLLM, with a 15.2% increase on KIE tasks and 10.7% on VQA tasks compared to previous SOTA OCR-based LLMs. All resources are available at URL masked for anonymous review.

## introduction recent research has increasingly explored the use of large language models (llms) or multimodal large language models (mllms) @xcite @xcite @xcite @xcite @xcite @xcite @xcite for document-oriented visual question answering (vqa) and key information extraction (kie). a line of research utilizes off-the-shelf ocr tools to extract text and spatial layouts, which are then combined with llms to address visually rich * equal contribution †corresponding author document understanding (vrdu) tasks. these approaches assume that most valuable information for document comprehension can be derived from the text and its spatial layouts, viewing spatial layouts as "lightweight visual information" @xcite . following this premise, several studies @xcite @xcite @xcite have explored various approaches that integrate spatial layouts with text for llms and achieves results that are competitive with those of mllms. the most natural method to incorporate layout information is by treating spatial layouts as tokens, which allows for the seamless interleaving of text and layout into a unified text sequence @xcite @xcite . for example, @xcite employ format such as "harrisburg 78|09" to represent ocr text and corresponding layout, where "har-risburg" is ocr text and "78|09" indicates the mean of the horizontal and vertical coordinates, respectively. similarly, @xcite use " @xcite " to represent layout information. these approaches can effectively take advantage of autoregressive characteristics of llms and is known as the "coordinateas-tokens" scheme @xcite . in contrast, docllm @xcite explores interacting spatial layouts with text through a disentangled spatial attention mechanism that captures crossalignment between text and layout modalities. however, we believe that both of the previous approaches have limitations. as shown in figure 1 , coordinate-as-tokens significantly increases the number of tokens. additionally, to accurately comprehend coordinates and enhance zero-shot capabilities, this scheme often requires few-shot incontext demonstrations and large-scale language models, such as chatgpt davinci-003 (175b) @xcite , which exacerbates issues related to sequence length and gpu resource demands. al though docllm does not increase sequence length, its performance may be improved by more effectively leveraging the autoregressive traits of llms. to address these problems, this paper explores a simple yet effective approach to enhance the interaction between spatial layouts and text -interleaving layout and text in a large language model (laytextllm) for document understanding. adhering to the common practice of interleaving any modality with text @xcite @xcite , we specifically apply this principle to spatial layouts. in particular, we map each bounding box to a single embedding, which is then interleaved with its corresponding text. as shown in figure 1 , laytextllm significantly outperforms the 175b models, while only slightly increasing or even reducing the sequence length compared to docllm. our contributions can be listed as follows: ## method in this section, we introduce laytextllm. we begin by detailing the model architecture, which features an innovative spatial layout projector (section 3.1) that transforms four-dimensional layout coordinates into a single-token embedding. next, we present three layout-text alignment pretraining tasks: line-level layout decoding, textto-layout prediction, and layout-to-text prediction (section 3.2.1) to ensure a seamless integration of layout and text understanding. finally, we describe the incorporation of spatially-grounded key information extraction as a auxiliary task during supervised fine-tuning (sft) (section 3.2.2), to enhance the performance in kie tasks. ## conclusion we propose laytextllm, interleaving spatial layouts and text to improve predictions through an innovative slp, the layout-text alignment pretraining and the sg-kie tasks. extensive experiments show the effectiveness of laytextllm.