Multilevel Hypernode Graphs for Effective and Efficient Entity Linking (2022)

Abstract: Information extraction on documents still remains a challenge, especially when dealing with unstructured documents with complex and variable layouts. Graph Neural Networks seem to be a promising approach to overcome these difficulties due to their flexible and sparse nature, but they have not been exploited yet. In this work, we present a multi-level graph-based model that performs entity building and linking on unstructured documents, purely based on GNNs, and extremely light (0.3 million parameters). We also propose a novel strategy for an optimal propagation of the information between the graph levels based on hypernodes. The conducted experiments on public and private datasets demonstrate that our model is suitable for solving the tasks, and that the proposed propagation strategy is optimal and outperforms other approaches.

## introduction information extraction (ie) from documents has become a hot research topic over the last few years @xcite @xcite . it is a challenging problem that requires combining computer vision (cv) and natural language processing (nlp) models in order to locate and parse the information segments, understand the document layout, and extract semantic relations between the segments. this problem becomes especially complex when dealing with unstructured documents, such as purchase receipts, where the layout of the documents can highly vary, making it hard for the models to learn how to extract semantic information. at this point, graph neural networks (gnns) seem to be a promising approach to overcome these difficulties and to solve the semantic information and relation extraction tasks, as they work over flexible graph-based representation capable of adapting to complex layouts, and they provide efficient and effective mechanisms for learning the relations between the segments @xcite @xcite @xcite @xcite . nevertheless, semantic ie still remains a challenging task. in fact, due to its complexity, it is usually split into three subtasks: thus, we can distinguish between three levels of information containers: for a solution based purely on gnns this leaves two options. one is trying to solve all the tasks using a single graph at segment level @xcite . the second option is splitting the problem into two graphs: one graph based on segment nodes for performing eb, and another one composed of entity nodes for performing @xcite . we believe that the second one is more effective for the following reasons: nevertheless, the multi-graph approach has more complexity, as it requires designing the way the output segment features and the entity features are related, and it has not yet been studied in depth. thus, in this work we focus on optimizing this propagation of information between the two stages using a novel approach within the ie field based on hypernodes. these are the main contributions: the model is purely based on gnns, using as inputs for each segment the bounding box and the entity category, and it is extremely light (0.3 million parameters). the conducted experiments demonstrate the effectiveness of the proposed method over highly unstructured documents in terms accuracy, processing time, and resource consumption. ## related work the growing interest in ie is patent in the number of recent publications. attending to the input data, most of the methods rely on the text and bounding boxes of an ocr engine for extracting the input features @xcite @xcite @xcite @xcite . other approaches enrich these ocr predictions with image features @xcite @xcite . however, the results reported in public ie benchmarks like funsd @xcite or cord @xcite suggest that the image features are not so relevant. finally, there are also a few models that purely rely on image features @xcite . the model proposed in this work extracts features from the ocr bounding boxes, but does not use the text, as it gathers the necessary information from the entity category input. attending to the model architecture, most of the methods are based on transformers @xcite and convolutional neural networks (cnns) @xcite @xcite @xcite @xcite @xcite @xcite @xcite . nevertheless, gnns are gaining importance thanks to their flexibility and capacity of adapting to complex layouts, along with their effective mechanisms for learning relationships between the nodes. in @xcite , the authors propose a two-stage gnn model. first, they generate a k-nearest neighbor (knn) graph to solve eb using text and bounding box features. then, the entity features are computed by aggregating the output features and processing them with a linear layer, and they are used to solve the et and el. in @xcite , the authors propose a single-stage gnn model: eb and et are solved via rel-s edges where each seed entity-type node links to its entity parts in sequence (solving also et as a consequence), el links the entities via rel-g edges, finally all mentioned edges are decoded at once. other gnn approaches solve only the et and el tasks, as they rely on the entity regions detected by the ocr engine @xcite @xcite , or by a previous cnn model @xcite . as it can be seen, there are few approaches based on gnn solving both eb and el. we aim at contributing to this line of research following an approach based on a two-stage gnn model, related to the one presented in @xcite , but with important modifications in the feature extraction, edge sampling, feature propagation, gnn architecture, and postprocessing. ## methodology we aim at solving the entity building (eb) and entity linking (el) tasks for a given list of documents. each document is composed of a list of semantic entities, that can be linked together to form entity groups. each entity can also be divided into smaller text segments. thus, given a list of text segments from an ocr engine, the goal is to group the text segments by their entity and then link together all the entities that belong to the same entity group. we propose to use gnns as the best approach: ## conclusions and future work in this work we have addressed the automation of information extraction on unstructured documents, given as inputs the predictions from an ocr engine and an entity tagging model, and focusing on two tasks, entity building and entity linking. we have justified the suitability of gnns for the considered use case and proposed a model based on this approach. this model tackles the problem in two stages that are strongly connected by using the concept of hypernodes. we have also proposed a novel strategy of propagating the features from the segment nodes to the entity nodes in an optimal way. the results of the conducted experiments demonstrate that the proposed model is suitable for solving the tasks, and that the proposed feature propagation strategy is optimal and outperforms other approaches. in addition, we have compared our model with other state-of-the-art methods that perform the eb and el tasks using the public benchmark cord and, although the models are not fully comparable, it can be observed that our model achieves state-of-the-art results with an extremely lower number of parameters. future work will focus on expanding the application of the model to address also the et task. to this end, new types of features could be considered, based on text or image, as we believe that the layout information is not enough to solve et task. in addition, we will keep enhancing the current capabilities of the model, exploring new ways of propagating the features, improving the postprocessing, and optimizing the gnn architectures.