D oc LLM : A Layout-Aware Generative Language Model for Multimodal Document Understanding (2024)

Abstract: Enterprise documents such as forms, receipts, reports, and other such records, often carry rich semantics at the intersection of textual and spatial modalities. The visual cues offered by their complex layouts play a crucial role in comprehending these documents effectively. In this paper, we present DocLLM, a lightweight extension to traditional large language models (LLMs) for reasoning over visual documents, taking into account both textual semantics and spatial layout. Our model differs from existing multimodal LLMs by avoiding expensive image encoders and focuses exclusively on bounding box information to incorporate the spatial layout structure. Specifically, the cross-alignment between text and spatial modalities is captured by decomposing the attention mechanism in classical transformers to a set of disentangled matrices. Furthermore, we devise a pre-training objective that learns to infill text segments. This approach allows us to address irregular layouts and heterogeneous content frequently encountered in visual documents. The pre-trained model is fine-tuned using a large-scale instruction dataset, covering four core document intelligence tasks. We demonstrate that our solution outperforms SotA LLMs on 14 out of 16 datasets across all tasks, and generalizes well to 4 out of 5 previously unseen datasets.

## introduction documents with rich layouts, including invoices, contracts, and forms, constitute a significant portion of enterprise corpora, and the automatic analysis of these documents offer considerable advantages @xcite . although document ai (docai) has made tremendous progress, there remains a significant performance gap in real-world applications due to the complex layouts, bespoke type-setting and template diversity exhibited by * equal contribution. conventional large language models (llms) such as gpt-3.5 @xcite , llama @xcite or falcon @xcite primarily accept text-only inputs and assume that the documents exhibit simple layouts and uniform formatting. they are not suitable for document intelligence tasks, which are inherently multi-modal, requiring the understanding of both text content and visual layout cues. numerous vision-language frameworks @xcite that can process documents as images and capture the interactions between textual and visual modalities do exist. however, these frameworks necessitate the use of complex vision backbone architectures @xcite to encode image information, and often make use of spatial information as an auxiliary contextual signal @xcite . in this paper, we present docllm, a lightweight extension to standard llms that excels in several visually rich form understanding tasks. unlike traditional llms, it models both spatial layouts and text semantics, and therefore is intrinsically multimodal. the spatial layout information is incorporated through bounding box coordinates of the text tokens obtained typically using optical character recognition (ocr), and does not rely on a complex vision encoder component. consequently, our solution preserves the causal decoder architecture, introduces only a marginal increase in the model size, and has reduced processing times. we demonstrate that merely including the spatial layout structure is sufficient for various document intelligence tasks such as form understanding, table alignment and visual question answering. existing efforts to incorporate spatial layout information typically involve either concatenating spatial and textual embeddings @xcite or summing the two @xcite . in contrast, we treat the spatial information as a distinct modality and compute its inter-dependency with the text modality in a disentangled manner @xcite . specifically, we extend the self-attention mechanism of transformers to include new attention scores that capture cross-modal relationships. there is often a correlation between the content, position and size of the fields in a form and hence representing their alignments at various abstraction levels across the transformer layers can enhance document understanding. visual documents often feature heterogeneous content, irregular layouts, and disjointed text segments. a classical next token prediction in selfsupervised pre-training can be restrictive for these documents since the preceding tokens may not always be relevant due to the diverse arrangements of text. to tackle this issue, we propose two modifications to the pre-training objective: (a) adopting cohesive blocks of text that account for broader contexts, and (b) implementing an infilling approach by conditioning the prediction on both preceding and succeeding tokens. due to these modifications, the model is better equipped to address misaligned text, contextual completions, intricate layouts, and mixed data types. although text spans and infilling tasks have been studied before @xcite , our solution is tailored for visual documents with an emphasis on semantically coherent blocks. we tune docllm on instruction data curated from multiple datasets for several document intelligence tasks including key information extraction (kie), natural language inference (nli), visual question-answering (vqa) and document classification (cls). the modifications introduced by docllm enhance the performance of llama2-7b model by 15-60% in four of five datasets unseen during training. our contributions include: (1) a lightweight extension to llms designed for understanding visual documents. (2) a disentangled spatial attention mechanism that captures cross-alignment between text and layout modalities. 2 related work general purpose models. by treating a document as text content, many text based llms @xcite @xcite ) can be directly utilized for document intelligence tasks. despite the remarkable capabilities provided by these llms, their lack of understanding of visual elements and layouts can be severely limiting in the docai context. although multi-modal llms @xcite @xcite @xcite ) that explicitly include image information can account for visual signals, they often struggle to recognize specific structures and patterns prevalent in enterprise documents @xcite . instead of relying on generalized training, a model tailored for visually rich document understanding (vrdu) tasks can gain a nuanced comprehension of the language, formats and data structures unique to these types of documents. ## discussion impact. docllm enables language models to go beyond plain text settings and offers immediate utility in visually rich document understanding tasks. by accommodating complex layout structures, docllm allows documents with rich layouts to be included in the pre-training corpus without requiring extensive preprocessing. the explicit modeling of spatial relationships enables perceiving the documents as inherently structured knowledge. flexibility. the support for multi-page documents, implemented through page breaks and document boundaries, enhances the model's ability to comprehend documents with diverse lengths. this overcomes the constraints of small multimodal models that can handle only a single page and multimodal llms mainly designed for images. limitations. the use of english-language datasets derived from limited enterprise domains (such as iit-cdip) may introduce inherent representational biases in vrdu models, including docllm. also, docllm may be vulnerable to inaccurate bounding box information produced by an ocr engine. 10 however, several modern off-the-shelf solutions can robustly extract text from documents (hegghammer, 2022), mitigating this issue. docllm's support for long-form documents is restricted by its context length. increasing the model size and allowing unbounded context length during inference can address this limitation. docllm is trained and designed to capture the characteristics of enterprise documents. consequently, its efficacy may be restricted when applied to documents outside this domain, such as presentation decks or marketing reports. finally, docllm may not excel at complex reasoning tasks, especially those requiring a deep understanding of numerical concepts. see appendix f for additional discussion. ## conclusions we introduced docllm, a lightweight extension to traditional llms, tailored for generative reasoning over documents with rich layouts. docllm eschews expensive image encoders and instead utilizes bounding box information to capture the spatial layout structure of documents. this is achieved through a disentangled attention mechanism that models cross-alignment between text and spatial modalities. notably, our model addresses the challenges posed by irregular layouts and heterogeneous content using a learning to infill pre-training objective. tuning the model on a carefully curated instruction dataset provides a flexible interface for interactions. our evaluation across various document intelligence tasks demonstrates that docllm surpasses equivalent models both for in-domain and out-of-domain datasets. in the future, we plan to infuse vision into docllm in a lightweight manner.