RE^2 : Region-Aware Relation Extraction from Visually Rich Documents (2024)

Abstract: Current research in form understanding predominantly relies on large pre-trained language models, necessitating extensive data for pre-training. However, the importance of layout structure (i.e., the spatial relationship between the entity blocks in the visually rich document) to relation extraction has been overlooked. In this paper, we propose \textbf{RE} gion-Aware \textbf{R} elation \textbf{E} xtraction ( \bf{RE^2} ) that leverages region-level spatial structure among the entity blocks to improve their relation prediction. We design an edge-aware graph attention network to learn the interaction between entities while considering their spatial relationship defined by their region-level representations. We also introduce a constraint objective to regularize the model towards consistency with the inherent constraints of the relation extraction task. To support the research on relation extraction from visually rich documents and demonstrate the generalizability of \bf{RE^2} , we build a new benchmark dataset, {DiverseForm} , that covers a wide range of domains. Extensive experiments on {DiverseForm} and several public benchmark datasets demonstrate significant superiority and transferability of \bf{RE^2} across various domains and languages, with up to 18.88% absolute F-score gain over all high-performing baselines

## introduction visually rich documents (vrds) encompass various types such as invoices, questionnaire forms, financial forms, legal documents, and so on. these documents possess valuable layout information that aids in comprehending their content. recent research @xcite @xcite has focused on extracting key information, such as entities and relations, from vrds by leveraging their layout structures and optical character recognition (ocr) results foot_0 . figure 1 shows an example where entity recognition aims to identify blocks of text in certain categories, such as question(q), answer(a), and header(h). relation extraction further predicts the links among the entities, especially q-a links indicating that the a block is the corresponding answer to the q block. extracting key information, especially relations in vrds is a challenging task. though similar to traditional extraction tasks in text-only natural language processing (nlp) @xcite , inferring relations in vrds poses additional challenges. they require not only understanding the semantic meaning of entities but also taking into account the layout information, e.g., the spatial structures among the entity blocks in original vrds. previous studies mainly focused on combining the text and layout with language model pre-training @xcite @xcite @xcite @xcite or encoding the local layout information by constructing super-tokens @xcite @xcite @xcite . however, the layout of the vrds, especially the relative spatial relationship among the entity blocks, is still yet to be effectively explored for relation extraction. to this end, we propose region-aware relation extraction (re 2 ) that leverages region-level spatial structures among the entities to reason about their relations foot_1 . specifically, given the question and answer entities from each vrd, we define three categories of region-level representations for each entity block, through which we further characterize the relative spatial relationship between each pair of question and answer entities. we then employ a layout-aware pre-trained language model (i.e., layoutxlm @xcite ) to encode the entities and an edge-aware graph attention network (egat) to further learn the interaction between the question and answer entities in a bipartite graph while considering their spatial relationship. to ensure each answer is linked to at most one question, we design a constraint-based learning objective to guide the learning process, in combination with the relation classification objective. to validate the effectiveness of re 2 , we conduct extensive experiments on various benchmark datasets for a wide range of languages and domains. we evaluate re 2 on two public datasets funsd @xcite and xfund @xcite , under supervised, multitask transfer, and zero-shot cross-lingual transfer settings. we also create a new benchmark dataset diverse-form that covers diverse domains, such as veterans affairs, visa applications, tax documents, air transport and so on, and evaluate re 2 for crossdomain transfer. experimental results show that re 2 outperforms the previous state-of-the-art approaches with a large margin on (almost) all languages and domains across all settings. our ablation studies also verify the significant benefit of the region-level spatial structures of entity blocks for relation extraction. the contributions of this work are summarized as follows: ## related work recent research on visually rich document information extraction shows that incorporating 2d positional embedding and layout coordinates into the pre-trained language models improves vrd understanding @xcite @xcite . @xcite ) models the spatial relationship of fine and coarse-grained visual elements based on intersection over union (iou) and focuses only on named entity recognition task. incorporating relative spatial positions of entities is essential for relation extraction task. (luo et al., 2023) incorporates the relative spatial relation between entities on a fine-grained level and serves as a task for model pre-training. to deal with the variation of relation definitions, docrel (li et al., 2022) proposes a contrastive learning framework that utilizes the coherence of existing relations in diverse enhanced positive views to generate relation representations. zhang et al. (2021) further explores entity relation extraction as dependency parsing, incorporating minimum vertical and horizontal distances between the entities as layout heuristics. compared with all these studies, our approach is the first to propose and incorporate multi-granular spatial structures among the entities, which have been shown to significantly improve relation extraction from vrds. graph attention networks (gat) @xcite have proven to be efficient for learning on graph-structured data @xcite . this is exemplified by the work graphdoc @xcite , a multimodal graph attention-based model that simultaneously utilizes text, layout, and image information for visually rich document understanding. though several studies @xcite @xcite have explored gnns for entity extraction from vrds, we are the first to design edge-aware gat to improve relation extrac-paragraph/ tabular-level bounding box entity-level bounding box layoutxlm biaffine classifier relation extraction head edge-aware graph attention network (egat) entity representation graph construction @xmath0: overview of the region-level relation extraction (re 2 ) framework. a bipartite graph of question and answer entities is constructed. in the egat layer, the representation of each entity is updated based on the attention scores of its first-order neighbors. tion from vrds, which presents additional challenges, encompassing spatial analysis to determine entity layout on the page and semantics between entities for identifying relations. gnns have also been applied to relation extraction from textual documents (zhu et al., 2019; guo et al., 2019; zhang et al., 2018). however, these methods cannot be directly adapted to relation extraction from vrds due to the fundamental differences in document formats, structures, and the key challenges encountered in relation extraction: text-only documents primarily rely on linguistic cues and phrases for relation extraction, whereas vrds necessitate consideration of both semantics and spatial context. given that, we innovatively incorporate a multigranular layout heuristic into an edge-aware graph attention network, placing greater emphasis on capturing more fine-grained layout structures. ## approach given a visually rich document d, a set of question entities @xmath1, q 2 , ..., q m } and answers @xmath2, a 2 , ..., a n }, we aim to identify all the connected pairs (q, a) where @xmath3, indicating that a is the corresponding answer of q. each q i or a j can be denoted as {[w 0 , w 1 , • • • , w t ], (x 0 , y 0 , x 1 , y 1 )}, where [w 0 , w 1 , • • • , w t ] is the sequence of words denoting the entity span and (x 0 , y 0 , x 1 , y 1 ) is the coordinates for the entity bounding box. figure 2 illustrates our re 2 framework that aims to leverage region-level spatial structures among the question and answer blocks to detect their association. ## experiment setting and hyperparameters the nvidia a40 gpu was utilized for all finetuning tasks. paragraph-level regions are created using easyocr through horizontal merging of text boxes when their distance is within 2, and vertical merging is performed when the distance is within 1, with the paragraph flag set to true. the model undergoes end-to-end training, incorporating finetuning of the layoutxlm base model. the egat layers and relation extraction head are trained from scratch, employing 2 egat layers for all experiments. the training process consists of 5000 steps with a batch size of 4, a learning rate of 5e-5, and a warm-up ratio of 0.1. cross-entropy loss is weighted at 1, and constraint loss is weighted at 0.02. ## conclusion in this work, we propose a novel entity relation extraction model, re 2 , that incorporates layout heuristics and constraints that are generalizable across different languages. experimental results on 8 different languages and our proposed dataset diverseform show the effectiveness of our proposed method under four settings (languagespecific, cross-lingual zero-shot, multi-lingual finetuning, and cross-domain transfer).