Merge and Recognize: A Geometry and 2 D Context Aware Graph Model for Named Entity Recognition from Visual Documents (2020)

Abstract: Named entity recognition (NER) from visual documents, such as invoices, receipts or business cards, is a critical task for visual document understanding. Most classical approaches use a sequence-based model (typically BiLSTM-CRF framework) without considering document structure. Recent work on graph-based model using graph convolutional networks to encode visual and textual features have achieved promising performance on the task. However, few attempts take geometry information of text segments (text in bounding box) in visual documents into account. Meanwhile, existing methods do not consider that related text segments which need to be merged to form a complete entity in many real-world situations. In this paper, we present GraphNEMR, a graph-based model that uses graph convolutional networks to jointly merge text segments and recognize named entities. By incorporating geometry information from visual documents into our model, richer 2D context information is generated to improve document representations. To merge text segments, we introduce a novel mechanism that captures both geometry information as well as semantic information based on pre-trained language model. Experimental results show that the proposed GraphNEMR model outperforms both sequence-based and graph-based SOTA methods significantly.

## introduction with the rapid progresses in natural language processing and computer vision, visual documents become a mainstream media for expressing abundant information. extracting the named entities from these visual documents is a critical step for further understanding. in the perspective of traditional natural language processing, the layout and the format information of documents is discarded. only plain text that simply consists of sequential words do the researchers focus on and are necessary to be extracted entities. however, visual documents consist of many discrete text segments with a large variety of layouts and formats as figure 1 shows. the combination of different text segments with different positions may represent different semantic information. without the layout structure and the 2d semantic context information in it, the named entity recognition in visual documents could be much harder. although the named entity recognition in visual documents is a newly proposed under-researched task. recently, in the attempt to make use of the structure information of visual documents, many works @xcite @xcite @xcite @xcite have designed nlp/cv/nlpcv based methods for visual-documents-related tasks. these approaches mostly focus on the coordinate of text segments to make features or learn embeddings. however, most of these methods do not consider the two problems: this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . 2. because of the layout design, a complete named entity may be separated into several segments and cannot represent its full meaning. meanwhile, some of text segments may lose the semantic information of their prefix ones and get incorrect tagging results. it is necessary and important to merge text segments into a complete named entity. specifically, for the first problem, as illustrated in figure 1 , it is hard to tell whether "会锦"("huijin") in text segment 1 is a named entity only according to its own plain text. while according to some of the nearest text segments, "秘书长"("secretary general") in text segment 2 and "创始人"("the founder") in text segment 3, human can help infer that text segment 1 is a person entity. for the second, as figure 1 shows, neither text segment 4 nor text segment 6 is a complete named entity. however, the two segments (segment 4 and segment 6) together represent a complete location entity that is 6/f, middle block, huijin world trade center, at the intersection of fengqi road and yan'an road, xihu district, hangzhou, zhejiang, china. apparently, the ability to merge text segments into a complete entity is important for ner from visual document. to solve the above two problems, in this paper, we propose graphnemr, a graph neural end-to-end joint model for named entity recognition and merging tokens into named entities from visual documents. graphnemr incorporates the geometric information with the semantics to automatically extract nonsequential context-aware hidden features for each text segments in the visual documents. specifically, we regard a visual document as a graph structure and all text segments in it are the graph nodes. the geometric information is represented by the adjacency matrix of the graph. in each text segment, a bilstm structure is used to sequentially encode tokens to represent semantic features. then the graph convolutional network (gcn) @xcite encoder integrates information between neighbor nodes to learn the final representation of each text segment. then the representations are used as the inputs of the merge module we proposed to decide the relation between text segments. after gcn encoder and the merge module, a lstm+crf decoder is used to get the named entity tagging. our main contributions of this paper are as follows: extensive experiments show that our model outperforms both the strong sequence-based baseline model (bilstm-crf) and the sota graph-based model (graphie) in visual document named entity recognition task. ## experiments we first introduce the datasets for evaluating our proposed model. then we describe baselines we compared with, the evaluation metrics and briefly explain our implementation details. next, we show the results for two datasets. finally, we demonstrate the improved effect of our model via ablation study. ## conclusions in this paper, we propose graphnemr, a graph-based model that uses graph convolutional networks to jointly merge text segments and recognize named entities from visual documents. we model the visual document as a graph and incorporate geometry information into graph convolution network to build richer 2d context. to address the problem that text segments need to be merged into a complete entity, we combine geometry features with semantic features for learning to merge and recognize named entities. we evaluate our model on relabeled sroie-vner dataset and a real-world bcd dataset. results show that our model outperforms sequence-based model (bilstm-crf) and graph-based model (graphie) for named entity recognition from visual documents.