Read Extensively, Focus Smartly: A Cross-document Semantic Enhancement Method for Visual Documents NER (2022)

Abstract: The introduction of multimodal information and pretraining technique significantly improves entity recognition from visually-rich documents. However, most of the existing methods pay unnecessary attention to irrelevant regions of the current document while ignoring the potentially valuable information in related documents. To deal with this problem, this work proposes a cross-document semantic enhancement method, which consists of two modules: 1) To prevent distractions from irrelevant regions in the current document, we design a learnable attention mask mechanism, which is used to adaptively filter redundant information in the current document. 2) To further enrich the entity-related context, we propose a cross-document information awareness technique, which enables the model to collect more evidence across documents to assist in prediction. The experimental results on two documents understanding benchmarks covering eight languages demonstrate that our method outperforms the SOTA methods.

## introduction visually-rich documents (vrds) are the most common information carriers in the real world, such as newspapers, resumes, tickets, etc. different from plain text data, the information in vrds is encoded by multiple modalities including text, vision, and layout. entity recognition from vrds, as a key step for document understanding, is of utmost practical interest for many downstream applications such as business analysis @xcite , intelligent education @xcite , digital library @xcite . the pioneering explorations approach this task by either computer vision @xcite or natural language processing @xcite figure 1 : redundant irrelevant information and insufficient entity-related information in current document make it difficult to extract entities accurately. we filter redundant information from irrelevant regions and expand the information source from a single to multiple related documents. thereby, entity-related information is efficiently integrated and the extraction is improved. however, the existing methods suffer from the following two main shortcomings. from intra-document perspective, most previous works model a document by encoding each token uniformly, ignoring the fact that the document is composed of several regions (e.g. paragraphs, tables, captions) that are not so closely related semantically @xcite . as shown in figure 1 , the type of covid-19 in the orange region does not depend on the context in other gray regions. unrestricted consideration of the whole document will not only distract the attention to local regions with stronger semantic associations @xcite but also increase the risk of fitting spurious features (also known as the shortcut phenomenon @xcite ). from the inter-document perspective, the observed information is limited to the single document context, which may not be enough to accurately recognize entities. as shown in figure 1 , we can't judge whether covid-19 is a piece of news or a virus only by virtue of "covid-19 spreads in the us". worse still, the insufficient context information can be further destroyed by the errors in character recognition when faced with low-quality documents (e.g. imaging blur, deformity) @xcite . in this work, we propose a cross-document semantic enhancement method to enable the model to focus and integrate entity-related information across documents. specifically, to prevent distractions from irrelevant regions in the current document, we design a learnable attention mask mechanism. cross-region attentions are masked with a higher probability, so the model tends to make predictions using more reliable features from the local region. we introduce gumbel softmax @xcite to solve the nondifferentiable problem of discrete mask variables. to embrace sufficient context to assist prediction, we propose a cross-document information awareness technique. inverted index @xcite is used to efficiently store and retrieve the contextualized representations of each token from each document. cross-document attention acts on entity token representations to collect sufficient evidence to improve prediction. our contribution is threefold: 1) we propose a cross-document semantic enhancement method to enable the model to focus on the entity-related information across documents. the method filters the redundant information while expanding the information source; 2) our method can be regarded as a plug-in. it can be added to any document understanding model to improve prediction. 3) the experimental results on two datasets covering eight languages demonstrate that the proposed method outperforms the state-of-the-art methods; 2 related work ## approach we propose a cross-document semantic enhancement method, which enables the model to make use of the cross-document context of the target entity and reduce the impact of noise from irrelevant regions of the current document. the problem settings in this paper are formally stated as follows. let @xmath0,...,n e denotes the predefined n e entity types of interest and @xmath1,...,nc denotes the entity label set derived by e according to the bio scheme. let @xmath2, b i , v i )} @xmath3,...,n d be a visually-rich document (vrds), where w i denotes token in the document. b @xmath4, y 0 i , x 1 i , y 1 i ) denotes the bounding box (i.e. the position in the document) and v i denotes image patch of w i . given a labeled set of vrds @xmath5,...,n s , we target at learning a mapping function f : @xmath6,...,n d . in other words, f assigns an entity label c w @xmath7. ## experimental setup in this section, we describe the datasets for training and evaluating the proposed method. we also detail the baseline models for comparison. finally, we clarify the implementation details and hyperparameter configuration of our method. ## results and analysis in this section, we present the experimental results on two well-known document understanding datasets to show the effectiveness of our method. ## conclusions in this work, we introduce a cross-document semantic enhancement method to improve entity recognition from visually-rich documents. the proposed learnable attention mask mechanism effectively filters redundant irrelevant information in the current document, which reduces the risk of overfitting spurious features. cross-document information awareness enriches sufficient entityrelated context to improve predictions. the proposed method can be regarded as a plug-in, which can be added to any existing document understanding model and improve prediction. experimental results show that the proposed method outperforms the existing state-of-the-art methods in documents of different languages and is more robust in real scenarios.