PPTSER : A Plug-and-Play Tag-guided Method for Few-shot Semantic Entity Recognition on Visually-rich Documents (2024)

Abstract: Visually-rich document information extraction (VIE) is a vital aspect of document understanding, wherein Semantic Entity Recognition (SER) plays a significant role. However, few-shot SER on visually-rich documents remains relatively unexplored despite its considerable potential for practical applications. To address this issue, we propose a simple yet effective Plug-and-Play Tag-guided method for few-shot Semantic Entity Recognition (PPTSER) on visually-rich documents. PPTSER is built upon off-the-shelf multi-modal pre-trained models. It leverages the semantics of the tags to guide the SER task, reformulating SER into entity typing and span detection, handling both tasks simultaneously via cross-attention. Experimental results illustrate that PPTSER outperforms existing fine-tuning and few-shot methods, especially in low-data regimes. With full training data, PPTSER achieves comparable or superior performance to fine-tuning baseline. For instance, on the FUNSD benchmark, our method improves the performance of LayoutLMv3-base in 1-shot, 3-shot and 5-shot scenarios by 15.61%, 2.13%, and 2.01%, respectively. Overall, PPTSER demonstrates promising generalizability, effectiveness, and plug-and-play nature for few-shot SER on visually-rich documents. The codes will be available at [https://github.com/whlscut/PPTSER](https://github.com/whlscut/PPTSER).

## introduction information extraction from visually-rich documents (vie) is a process that concentrates on extracting pertinent information from various sources such as scanned images, documents, and pdf files. it effectively leverages layout and visual cues to decode the content enclosed within these documents @xcite . as an important part of vie, semantic entity recognition (ser) aims to extract entity spans from the visually-rich document. ser has been hailed as a significant advancement in the realm of document intelligence, and has found widespread applications in numerous sectors. the advent of multi-modal pre-trained models @xcite @xcite @xcite has ushered in a rapid evolution in ser methodologies. these models, pre-trained on a large corpus of scanned documents in a self-supervised manner, have significantly enhanced the comprehension ability of ser. despite the remarkable achievements of the multi-modal pre-trained models, they often rely on extensive data for fine-tuning. however, acquiring a large volume of well-annotated ser data poses significant challenges such as: (1) acquiring such data necessitates substantial financial resources and time. annotators are required to label a multitude of ocr detection boxes in the document, adhering to meticulously designed guidelines. identification of content within a box and accurately assigning labels to them are also tedious tasks. (2) the availability of data is often restricted due to privacy concerns. in scenarios involving sensitive information, such as invoices and insurance documents, data accessibility is severely limited due to the confidential nature of this information. despite the scarce research @xcite @xcite on fewshot semantic entity recognition for visually-rich documents (few-shot ser), results have shown limitations in terms of generality and performance, and were limited to the specific application scenario. this paper, inspired by the comprehension capabilities of pre-trained models and the selective focus nature of the attention mechanism, introduces a novel approach called pptser, a plug-and-play tag-guided method for few-shot semantic entity recognition on visually-rich documents. the underlying principle of pptser consists of three main components: (1) semantic understanding and alignment: words related to ser tags are used as a prompt and are concatenated with the document's text tokens. this combined input is then fed into a multi-modal pre-trained model. the motivation behind this is that the pre-trained model is expected to understand the semantics of both the document tokens and the tag-related prompt, thereby bringing the hidden states of the tokens and tag-related words for a specific entity type closer together. (2) decoupling of ser task: ser task is segmented into entity classification and entity boundary detection. this division aims to facilitate the resolution of boundary determination among adjacent entities of the same category within visually-rich documents. (3) efficient usage of multi-head attention: the attention weight obtained from the last attention block between the tag-related prompt and document tokens is directly used as the probability of tokens belonging to different tags. this mechanism, with different heads detecting various spans, is ideal for the ser task with numerous entity spans. by fully exploiting the weighted focus nature of the attention mechanism, the model eliminates the value transform layer, feed-forward layer in the last attention block, and omits a separate classifier layer compared to traditional fine-tuning methods (as depicted in figure 1 ), as a result, the total parameter is reduced. extensive experiments are conducted to show the pptser's effectiveness on commonly-used ser benchmarks, covering multiple languages, in fewshot to the full training set settings, and using different mainstream multi-modal pre-trained models. the main contributions of this paper can be summarized as follows: to the best of our knowledge, we are the first to propose a pluggable method that has shown effectiveness on various pre-trained models and languages. in this paper, we primarily focus on the in-label-space setting for few-shot ser. specifically, the pre-trained model is firstly fine-tuned on a small number of m annotated documents with label space c and then directly evaluated on the test set with the same label space c. this task presents a significant challenge as the model needs to learn the ser task with only limited training samples. it is notable that in the context of few-shot ser, the few-shot setting of n-way k-shot indicates that ## ablation study we have conducted extensive analyses of our pptser to ensure its effectiveness and rationality. for convenience, experiments are conducted on the cord dataset using pptser building upon lilt. ## conclusion in this paper, we present pptser, an innovative and efficient strategy for few-shot entity recognition on visually-rich documents using a plug-andplay, tag-guided approach. pptser redefines the ser task as a dual-function operation of entity typing and span detection, and utilizes the atten-tion weight between document tokens and prompts related to ser tags as the target probability distributions. our findings show that pptser is both effective and versatile in various data situations, from few-shot to full data scenarios. in the future, we plan to further investigate the capabilities of pptser across a range of vie tasks like entity linking. in addition, we aim to explore pptser's potential in other few-shot scenarios, particularly those outside of the in-label-space setting. it is our hope that our work will spark further research and advancements in the realm of few-shot ser.