Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learning (2023)

Abstract: Extracting meaningful entities belonging to predefined categories from Visually-rich Form-like Documents (VFDs) is a challenging task. Visual and layout features such as font, background, color, and bounding box location and size provide important cues for identifying entities of the same type. However, existing models commonly train a visual encoder with weak cross-modal supervision signals, resulting in a limited capacity to capture these non-textual features and suboptimal performance. In this paper, we propose a novel Visually-Asymmetric coNsistenCy Learning (VANCL) approach that addresses the above limitation by enhancing the model’s ability to capture fine-grained visual and layout features through the incorporation of color priors. Experimental results on benchmark datasets show that our approach substantially outperforms the strong LayoutLM series baseline, demonstrating the effectiveness of our approach. Additionally, we investigate the effects of different color schemes on our approach, providing insights for optimizing model performance. We believe our work will inspire future research on multimodal information extraction.

## introduction information extraction (ie) for visually-rich formlike documents (vfds) aims to handle various types of business documents, such as invoices, receipts, and forms, that may be scanned or digitally generated. this task has attracted significant attention from the research and industrial communities @xcite @xcite . as shown in figure 1 , the goal of ie for vfds is to identify and extract meaningful semantic entities, such as company/person names, dates/times, and contact information, from serialized ocr (optical character recognition) output in the documents. since a single modality may not capture all the semantic information • shipper(发货人) there are two problems with the visual encoder in previous lmms. first, these models only impose coarse-grained cross-modal constraints during pretraining (e.g., text-image, word-patch, and layouttext alignment @xcite @xcite ) to enhance feature extraction from the visual channel, but this does not capture sufficient fine-grained visual features, leading to limited performance and underutilization of prior knowledge in vision. second, the visual encoders in previous lmms have weaker representational capabilities than those in the latest optical character recognition (ocr) engines because they do not consider the intermediate tasks such as text segment detection and bounding box regression, which are important for accurately localizing and extracting fine-grained visual features. to address the issues mentioned above, inspired by recent works on consistency learning @xcite @xcite @xcite , we propose a novel vision-enhanced training approach, called visually-asymmetric consistency learning (vancl). by incorporating color priors with category-wise colors as additional cues to capture visual and layout features, vancl can enhance the learning of unbiased multimodal representations in lmms. our approach aims to jointly consider the training objectives of the text segment (or bounding box) detection task and the entity type prediction objective, thereby bridging the preprocessing ocr stage with the downstream information extraction stage. vancl involves a standard learning flow and an extra vision-enhanced flow. the inputs are asymmetric in the visual modality, with the original document images input to the standard flow, while images to the vision-enhanced flow are the synthetic painted images. the vision-enhanced flow also can be detached when deploying the model. during training, we encourage the inner visual encoder to be as strong as the outer visual encoder via consistency learning. as a result, vancl outperforms existing methods while (1) extremely little manual effort to create synthetic painted images, (2) no need to train from scratch, whilst (3) no increase in the deployment model size. our contributions can be summarized as follows: ## problem formulation we treat the semantic entity recognition (ser) task as a sequential tagging problem. given a document d consisting of a scanned image i and a list of text segments within ocr bounding boxes @xmath0, . . . , b n }, we formulate the problem as finding an extraction function f ie (d : ⟨b, i⟩) → e that predicts the corresponding entity types for each token in d. the predicted sequence of labels e is obtained using the "bio" scheme -{begin, inside, other} and a pre-defined label set. we train our sequential tagging model based on pretrained lmms and perform viterbi decoding during testing to predict the token labels. each bounding box b i contains a set of m tokens transcribed by an ocr engine and coordinates defined as (x 1 i , x 2 i ) for the left and right horizontal coordinates and (y 1 i , y 2 i ) for the top and bottom vertical coordinates. ## approach our model architecture for visually-asymmetric consistency learning is illustrated in figure 2 . inspired by mutual learning @xcite , we start with a standard learning flow and an extra vision-enhanced flow, which are learned simultaneously to transfer knowledge from the visionenhanced flow to the standard learning flow. it is worth noting that the input images for the visionenhanced flow are colorful prompt paints, while the input images for the standard flow are original images. therefore, the information contained in the visual inputs to the vision-enhanced flow and the standard model is asymmetric. ## conclusion we present vancl, a novel consistency learning approach to enhance layout-aware pretrained multimodal models for visually-rich form-like document information extraction. experimental results show that vancl successfully learns prior knowledge during the training phase, surpassing existing state-of-the-art models on benchmark datasets of varying sizes and a large-scale real-world form document dataset. vancl exhibits superior performance against strong baselines with no increase in model size. we also provide recommendations for selecting color schemes to optimize performance.