L ayout P ointer: A Spatial-Context Adaptive Pointer Network for Visual Information Extraction (2024)

Abstract: Visual Information Extraction (VIE), as a crucial task of Document Intelligence, involves two primary sub-tasks: Semantic Entity Recognition (SER) and Relation Extraction (RE). However, VIE faces two significant challenges. Firstly, most existing models inadequately utilize spatial information of entities, often failing to predict connections or incorrectly linking spatially distant entities. Secondly, the improper input order of tokens challenges in extracting complete entity pairs from documents with multi-line entities when text is extracted via PDF parser or OCR. To address these challenges, we propose LayoutPointer, a Spatial-Context Adaptive Pointer Network. LayoutPointer explicitly enhances spatial-context relationships by incorporating 2D relative position information and adaptive spatial constraints within self-attention. Furthermore, we recast the RE task as a specialized cycle detection problem, employing a unique tail-to-head pointer to restore the semantic order among multi-line entities. To better evaluate the effectiveness of our proposed method, we reconstruct a multi-line dataset named MLFUD, which more accurately reflects real-world scenarios. Fine-tuning experimental results on FUNSD, XFUND, and MLFUD datasets demonstrate that LayoutPointer significantly outperforms existing state-of-the-art methods in F1 scores for RE tasks (e.g., 5.71% improvement on XFUND using LayoutPointer _{\text{BASE-X}} over LayoutLMv3).

## introduction the visually-rich document understanding (vrdu) task involves leveraging artificial intelligence to extract and interpret information from digitized or scanned documents @xcite @xcite information extraction (vie), as a critical component within vrdu, involves two primary sub-tasks: semantic entity recognition (ser) and relation extraction (re). ser focuses on extracting specific entities from documents, while re aims to identify relations between entities. in recent years, multimodal pre-trained models @xcite @xcite @xcite @xcite @xcite @xcite , which integrate image, text, and layout information, have demonstrated strong capabilities in the vie task. however, the vie task currently faces two primary challenges. firstly, most existing models inadequately handle the explicit spatial relationships between entities. we test the capabilities of the state-of-the-art base model layoutlmv3 @xcite in the re task. as shown in fig. 1a , layoutlmv3 fails to identify five entity links in the "name of account" column of a table. moreover, fig. 1b demonstrates that, when processing forms with a left-right structural layout, layoutlmv3 incorrectly connects entities that are spatially distant and belong to different rows, such as "lorillard" and "donald". these findings highlight the limitations of layoutlmv3 in perceiving spatial distance and layout patterns between entities. one potential solution involves applying relative distance thresholds along the x and y axes to refine relationship predictions. however, the effectiveness of this solution strongly relies on the specific threshold settings, making it difficult to achieve precise control across different scenarios. secondly, the improper input order of tokens makes it difficult for existing methods to extract complete entity pairs from documents with multiline texts. when extracting text from real-world documents using a pdf parser or ocr, it is common for an entity to be divided across multiple lines. a case in point is illustrated in fig. 2 , where "requirements laboratory" is split into "requirements" and "laboratory" on separate rows after ocr processing. tokens within the same entity will be scrambled when sorted by 2d coordinates as coordinates cannot be shared between them. existing vie schemes @xcite primarily depend on head features of contiguous entities to identify relations. however, the method falls short in reconstructing the correct semantic order of entities, leading to incomplete results. to address the above two challenges, we propose a spatial-context adaptive pointer network, named layoutpointer. layoutpointer consists of three main components: (1) feature extractor, leveraging layoutlmv3 for multimodal feature extraction; (2) spatial-context adaptive enhancer, building on multi-layer encoders that explicitly introduces 2d relative position information and adaptive spatial constraints through rc-rope and soft layout mask mechanisms in self-attention; to evaluate the effectiveness of our proposed method, we extract samples from public datasets funsd and xfund and reconstruct a multiline dataset, named mlfud. fine-tuning experimental results reveal that layoutpointer outperforms current state-of-the-art methods, improving f1 scores by 5.68% over geolayoutlm and 5.71% over layoutlmv3 for the re task on funsd and xfund, respectively, and significantly improving by 30.29% over layoutlmv3 on mlfud. h 0 h 1 h ... l 0 (opt.) label embedding l 1 l... slm texts & boxs ocr/ pdf parser document image linear n encoder × sca attention add & norm ffn sca attention add & norm bros (hong et al., 2022) introduces the spade decoder for re. kvpformer (hu et al., 2023) employs a coarse-to-fine answer prediction approach, which identifies each question first and then selects the most likely answer. geolayoutlm (luo et al., 2023) design a transformer-based relation head with geometric pre-training. most models utilize predefined spatial relative position information or 2d position embeddings to handle 2d coordinates. this paper investigates the integration of explicit adaptive spatial information to improve the performance of re through finetuning only. additionally, we explore methods for extracting relations between multi-line entities.