P ² N et: Parallel Pointer-based Network for Key Information Extraction with Complex Layouts (2025)

Abstract: Key Information Extraction (KIE) is a challenging multimodal task aimed at extracting structured value entities from visually rich documents. Despite recent advancements, two major challenges remain. First, existing datasets typically feature fixed layouts and a limited set of entity categories, while current methods are based on a full-shot setting that is difficult to apply in real-world scenarios, where new entity categories frequently emerge. Secondly, current methods often treat key entities simply as parts of the OCR-parsed context, neglecting the positive impact of the relationships between key-value entities. To address the first challenge, we introduce a new large-scale, human-annotated dataset, Complex Layout document for Key Information Extraction (CLEX). Comprising 5,860 images with 1,162 entity categories, CLEX is larger and more complex than existing datasets. It also primarily focuses on the zero-shot and few-shot KIE tasks, which are more aligned with real-world applications. To tackle the second challenge, we propose the Parallel Pointer-based Network (P²Net). This model frames KIE as a pointer-based classification task and effectively leverages implicit relationships between key-value entities to enhance extraction. Its parallel extraction mechanism enables simultaneous and efficient extraction of multiple results. Experiments on widely-used datasets, including SROIE, CORD, and the newly introduced CLEX, demonstrate that P²Net outperforms existing state-of-the-art methods (including GPT-4V) while maintaining fast inference speeds.

## introduction key information extraction (kie) aims to extract value entities (i.e., the text contents with their layout coordinates) from visually rich documents in the semi-structured document @xcite , such as forms, or scanned documents. for instance, * corresponding author figure 1 : an example of the kie. the key and value entities are marked in blue and orange boxes, respectively. the dotted arrows indicate implicit relations between key-value pairs. the content of the table is synthesized and some values are decorated for privacy purposes. as shown in fig. 1 , given an input visually rich document and a query "organization", kie aims to extract the value entity "natural resources bureau". kie plays a critical role in many downstream tasks, such as document information registration @xcite and document understanding @xcite . many tasks and datasets @xcite @xcite @xcite @xcite have been proposed for kie across various domains. in addition, a series of pre-trained multi-modal models @xcite @xcite were presented. based on these models, some methods @xcite @xcite @xcite leveraged biaffine model, question answering (qa), generation, graph-based, or token path prediction methods to extract the value entities. recently, many multimodal large language models (mllms) @xcite @xcite were proposed, contributing to the kie task. despite the promising results, those datasets and methods still suffer from the following two challenges: to address the challenge of existing datasets in terms of layout variety and entity categories, we propose a new large-scale human-annotated dataset named complex layout document for key information extraction (clex), which includes 5,860 scanned document images with 155 document types and 1,162 entity categories. in order to meet the practical needs and make the model adaptable to the scenario of emerging unseen entities, we mainly consider zero/few-shot kie task. specifically, this task leverages a set of known category samples, followed by the introduction of zero or a few number (1/5/10) of samples from previously unseen document categories for training the model. the model's performance is then evaluated on these new, unseen document types. to tackle the second challenge, we propose the parallel pointer-based network (p 2 net). this model frames kie as a pointer-based classification task, where key and value entities can point to each other. by doing so, it effectively captures the implicit relationships between key entities and keyvalue pairs, facilitating more accurate extraction of value entities. additionally, p²net incorporates multiple questions into the input context and utilizes token-linking operations to generate results for each question simultaneously. this parallel extraction mechanism greatly enhances the speed of p²net, improving both efficiency and performance. we conduct extensive few/zero-shot kie experiments on the sroie @xcite , cord @xcite , and the newly proposed clex datasets. the results demonstrate that p 2 net outperforms existing state-of-the-art methods, and the inference speed increases 6.4 times compared to traditional qa method on clex. in summary, the contributions of this paper are: ## related work kie datasets. there are some datasets released in the kie research area. for instance, funsd @xcite ) contains 199 noisy scanned documents in english with 4 entity categories. after that, xfund @xcite extend it into 7 different languages. several other datasets, such as maten @xcite , sroie @xcite , cord @xcite , ephoie @xcite , kleaster nda and kleaster charity @xcite , vrdu-registration form and vrdu-ad-buy form @xcite , poie @xcite were proposed based on different fields. however, the layouts of existing kie datasets are relatively fixed, making it hard to be applied to the real scenario where new entities are constantly emerging. to alleviate this problem, we propose the few/zero-shot kie task, and introduce a new large-scale dataset with complex layouts named clex with 5,860 images and 1,162 entity categories. please note that different from traditional few/zero-shot kie task @xcite , in this work, the models are first pre-trained on a set of known category samples, and then trained by zero or several unseen category samples. the motivation of this setting is to leverage the migration ability of the models from known category samples to those unknown category samples. source image number entity type language sroie receipts 1000 4 english cord receipts 1000 30 english ephoie paper head 1494 10 chinese funsd forms 199 4 english xfund forms 199 4 7 languages kleaster nda edgar 540 4 english kleister charity uk charity commission 2778 8 english vrdu-registration form foreign agents registration act 1915 6 english poie product 3000 21 english clex (ours) invoices, certificates, etc. 5860 1162 chinese/english ## clex dataset in this section, we introduce the data construction process of the proposed complex layout document for key information extraction (clex) dataset, as well as its statistics. ## conclusion in this work, to simulate the real-world situation that has various types of complex layout styles and unseen entities are emerging, we propose the clex, a human-annotated complex layout kie dataset with 5,860 images and 1,162 entity categories. and we mainly consider the few/zero-shot kie task. we also propose p 2 net. by leveraging several token linking operations, p 2 net incorporates the implicit clues from keys and key-value pairs and extracts value entities in parallel. the zero/few-shot kie experiments on relevant datasets illustrate that p 2 net outperforms state-of-the-art methods with a fast inference speed.