Global Structure Knowledge-Guided Relation Extraction Method for Visually-Rich Document (2023)

Abstract: Visual Relation Extraction (VRE) is a powerful means of discovering relationships between entities within visually-rich documents. Existing methods often focus on manipulating entity features to find pairwise relations, yet neglect the more fundamental structural information that links disparate entity pairs together. The absence of global structure information may make the model struggle to learn long-range relations and easily predict conflicted results. To alleviate such limitations, we propose a GlObal Structure knowledge-guided relation Extraction (GOSE) framework. GOSE initiates by generating preliminary relation predictions on entity pairs extracted from a scanned image of the document. Subsequently, global structural knowledge is captured from the preceding iterative predictions, which are then incorporated into the representations of the entities. This “generate-capture-incorporate” cycle is repeated multiple times, allowing entity representations and global structure knowledge to be mutually reinforced. Extensive experiments validate that GOSE not only outperforms existing methods in the standard fine-tuning setting but also reveals superior cross-lingual learning capabilities; indeed, even yields stronger data-efficient performance in the low-resource setting.

## introduction visually-rich document understanding (vrdu) aims to automatically analyze and extract key information from scanned/digital-born documents, such as forms and financial receipts @xcite . since visually-rich documents (vrds) usually contain diverse structured information, visual relation extraction (vre), as a critical part of vrdu, has recently attracted extensive attention from both the academic and industrial communities @xcite @xcite . the vre task aims to identify relations between semantic entities in vrds, severing as the essential basis of mapping vrds to structured information which is closer to the human comprehension process of the vrds @xcite . recently, inspired by the success of pre-training in visually-rich document understanding @xcite @xcite , many fine-tuning works predict relation for each entity pair independently, according to local semantic entity representations derived from the pre-trained model. although existing vre methods have achieved promising improvement, they ignore global structure information, i.e. dependencies between entity pairs. without considering global structure knowledge, the model may easily predict conflicted links and struggle to capture longrange relations. taking the state-of-the-art model lilt @xcite as an example, as shown in figure 1 (a), although each relational entity pair predicted by the lilt may make sense semantically, there are conflicted relational entity pairs such as (labels, blue) and (closures, white), as well as (tear tape, white) and (cartons, white), whose link crosses each other from a global view. this phenomenon indicates that methods only using local features do not have the sufficient discriminatory ability for conflicted predictions. furthermore, as shown in figure 1 (b), even though lilt accurately identifies the relational entity pairs (no.of stores, 19) and (no.of stores, 18), lilt still hardly learns long-range relations such as (no.of stores, 16) and (no.of stores, 17). our intuition is that global structure knowledge can help the model learn long-range relations. the model can predict the relational entity pair (no.of stores, 17) by analogy with the global structure consistency between (no.of stores, 17) and (no.of stores, 18). in this paper, we present the first study on leveraging global structure information for visual relation extraction. we focus on how to effectively mine and incorporate global structure knowledge into existing fine-tuning methods. it has the following two challenges: (1) huge mining space. considering n entities in a vrd, the computational complexity of capturing dependencies between entity pairs is quadratic to entity pair size (n 2 × n 2 ). so it is difficult to mine useful global structure information in the lack of guidance. (2) noisy mining process. since the process of mining dependencies between entity pairs relies on initial entity representations and lacks direct supervision, global structure information learned by the model is likely to contain noise. mined noisy global structure knowledge by a model can in turn impair the performance of the model, especially when the model has low prediction accuracy in the early training stage. to this end, we propose a general global structure knowledge-guided visual relation extraction method named gose, which can efficiently and accurately capture dependencies between entity pairs. gose is plug-and-play, which can be flexibly equipped to existing pre-trained vrdu models. specifically, we first propose a global structure knowledge mining (gskm) module, which can mine global structure knowledge effectively and efficiently. the gskm module introduces a novel spatial prefix-guided self-attention mechanism, which takes the spatial layout of entity pairs as the attention prefix to progressively guide mining global structure knowledge in a local-global way. our intuition is the spatial layout of entity pairs in vrds may be a valuable clue to uncovering global structure knowledge. as shown in figure 1 (a), we can recognize crossover between entity pairs in 2d space by computing the spatial layout of linking lines. furthermore, in order to increase the robustness of gose and handle the noisy mining process, we introduce an iterative learning strategy to combine the process of entity representations learning and global structure mining. the integration of global structure knowledge can help refine entity embeddings, while better entity embeddings can help mine more accurate global structure knowledge. in summary, the contributions of our work are as follows: ## preliminaries in this section, we first formalize the visually-rich document relation extraction task and then briefly introduce how the task was approached in the past. ## methodology in this section, we describe the proposed framework named gose in detail. as shown in figure 2 , our method consists of three main modules, a base module, a relation feature generation (rfg) module, and a global structure knowledge mining (gskm) module. in section 3.1, we present how the base module can be combined with a pretrained vrdu model to generate initial key/value features. in section 3.2, we introduce the rfg module to generate relation features. then, in section 3.3, we elaborate on how the gskm module mines global structure knowledge in a local-global way. we further provide a theoretical analysis to better understand the efficient and effective features of the gskm module. finally, we show how to apply our iterative learning strategy to mine accurate global structure knowledge in section 3.4. ## experiments in this section, we perform detailed experiments to demonstrate the effectiveness of our proposed method gose among different settings. besides the standard setting of typical language-specific fine-tuning (section 4.2), we further consider more challenging settings to demonstrate the generalizability of gose such as cross-lingual zero-shot transfer learning (section 4.3) and few-shot learning (section 4.4). before discussing the results, we provide the details of the experimental setup below. ## discussion recently, the multimodal large language model (mllm) @xcite has emerged as a pioneering approach. mllms leverage powerful large language models (llms) as a brain to perform multimodal tasks. the surprising emergent capabilities of mllm, such as following zero-shot demonstrative instructions @xcite and ocr-free math reasoning @xcite , are rare in traditional methods. several studies @xcite have conducted comprehensive evaluations of publicly available large multimodal models. these investigations reveal that mllms still struggle with the vie task. in this paper, we introduce and empirically validate that global structural knowledge is useful for visually-rich document information extraction. our insights have the potential to shape the advancement of large model technology in the domain of visually-rich documents. ## conclusion in this paper, we present a general global structure knowledge-guided relation extraction method for visually-rich documents, which jointly and iteratively learns entity representations and mines global dependencies of entity pairs. to the best of our knowledge, gose is the first work leveraging global structure knowledge to guide visual relation extraction. concretely, we first use a base module that combines with a pre-trained model to obtain initial relation predictions. then, we further design a relation feature generation module that generates relation features and a global structure knowledge mining module. these two modules perform the "generate-capture-incorporate" process multiple times to mine and integrate accurate global structure knowledge. extensive experimental results on three different settings (e.g., standard fine-tuning, cross-lingual learning, low-resource setting) over eight datasets demonstrate the effectiveness and superiority of our gose.