Hypergraph based Understanding for Document Semantic Entity Recognition (2024)

Abstract: Semantic entity recognition is an important task in the field of visually-rich document understanding. It distinguishes the semantic types of text by analyzing the position relationship between text nodes and the relation between text content. The existing document understanding models mainly focus on entity categories while ignoring the extraction of entity boundaries. We build a novel hypergraph attention document semantic entity recognition framework, HGA, which uses hypergraph attention to focus on entity boundaries and entity categories at the same time. It can conduct a more detailed analysis of the document text representation analyzed by the upstream model and achieves a better performance of semantic information. We apply this method on the basis of GraphLayoutLM to construct a new semantic entity recognition model HGALayoutLM. Our experiment results on FUNSD, CORD, XFUND and SROIE show that our method can effectively improve the performance of semantic entity recognition tasks based on the original model. The results of HGALayoutLM on FUNSD and XFUND reach the new state-of-the-art results.

## introduction with the development of information technology, documents have become a main information carrier nowadays, which contains kinds of information type, such as text, table and image. manual recognition of these documents often requires plenty of manpower. ocr tools can only help us to identify the text, layout and other simple information in the document. to further understand documents, visually-rich document understanding (vrdu) @xcite ) is proposed to make use of visual, textual and other information for more in-depth analysis. semantic entity recognition (ser) is an important task in the field of vrdu. its purpose is to extract and classify the text with special semantic information in documents. different from text sequences in traditional natural language processing tasks, the information in documents is not onedimensional, single-modal and continuous, but twodimensional, multimodal and discrete. it is necessary to analyze not only text information, but also other modal information such as layout and vision in the document. figure 1 shows the difference between the traditional named entity recognition (ner) task on a single modal text and the semantic entity recognition task on a document. firstly, the text form of a single modal text task is a fixed text sequence, while the discrete text in a document is composed of text nodes in different locations. secondly, the named entity recognition task of a single modal text only needs to consider the semantic relationship between the tokens in the text sequence. however, the semantic entity recognition task on the document needs to consider not only the semantic relationship between nodes, but also the position relationship between nodes. finally, the span range of entity tags of ner task is flexible, while the range of task tags of semantic entity recognition task on document is affected by nodes. texts of the same node in the document share the same label in most cases. with the development of pre-training technology, document pre-training model has become popular. layoutlm @xcite is the first multi-modal pre-trained model to associate text with layout and vision, achieving leading results on multiple downstream document understanding tasks including semantic entity recognition. subsequently, more multi-mode pretraining models, such john smith traveled to new york city to visit his friend jane doe. they went to see the statue of liberty and had lunch at a famous restaurant in chinatown. later, they took a walk in central park and enjoyed the beautiful scenery. as layoutlmv2 @xcite , bros @xcite , ernie-layout @xcite and layoutlmv3 @xcite have been proposed successively. by integrating text, layout and visual information, they realize the understanding and information extraction of documents. so far, graphlayoutlm @xcite and geolayoutlm @xcite have the best performance in semantic entity recognition tasks. graphlayoutlm achieves the best f1 score of 94.39 and 93.56 on the funsd @xcite and xfund @xcite datasets. geo-layoutlm achieves the best f1 score of 97.97 on the cord @xcite datasets. however, these existing methods focus on the upstream document understanding part and pay little attention to the downstream task. geolayoutlm has studied the novel relational extraction head and achieves great improvement in the relational extraction task. but it has not done more research on the semantic entity recognition task. we study the problem of ignoring the downstream head and classification method in the semantic entity recognition task in the existing document intelligence work and propose a novel improvement scheme. traditional semantic entity recognition. the traditional document semantic entity recognition task process is shown in (a) of the figure 2 . in document understanding process, text nodes are spliced into text sequences and become text token sequences of documents after tokenization. these text nodes will be transformed to the highdimensional feature representations after the analysis of the document understanding model. to extract semantic information from document token features, linear layer or multilayer perceptron (mlp) will be used to convert high-dimensional features into label probabilities and the training objective is cross entropy loss. although this method can distinguish the node categories in the document, it ignores the characteristics of the document structure and it is difficult to make the classification layer pay attention to the node span. hypergraph semantic entity recognition. inspired by global pointer @xcite , we use the idea of hypergraph to extract the semantic information of documents and propose a hypergraph attention(hga) strategy for document semantic entity recognition. (b) of the figure 2 shows us the process of hypergraph semantic recognition. different from the traditional classification method, the semantic entity recognition idea of hga regard the document token features as graph nodes. the target entity is the set of nodes with the same hyperedge and the hyperedge type represents the entity label type. the process of hypergraph extraction is to establish hyperedges between token feature nodes. besides, we use the span hyperedge encoding to add the span information of text nodes. through the hypergraph and span position, the head can better focus on the entity boundary information and establish the relationship between the document discrete text span and the entity boundary. our main contributions are as follows: ## related work in recent years, self-supervised pre-training technology has become the mainstream trend in the fields of natural language processing (nlp) and ## conclusion in this work, we propose a semantic entity recognition method (hga) based on hypergraph attention. this method extracts semantic information from documents by establishing different hyperedges between feature nodes. on the basis of the hypergraph, we design span position encoding and balanced hyperedge loss to enhance the entity extraction capability of the hypergraph attention head. we use the hga method to build a novel semantic entity recognition model hgalayoutlm based on graphlayoutlm. this model has good performance in ser tasks. experiments show that our method achieves the state of art on semantic entity recognition tasks on the funsd and xfund datasets.