Towards Few-shot Entity Recognition in Document Images: A Label-aware Sequence-to-Sequence Framework (2022)

Abstract: Entity recognition is a fundamental task in understanding document images. Traditional sequence labeling frameworks treat the entity types as class IDs and rely on extensive data and high-quality annotations to learn semantics which are typically expensive in practice. In this paper, we aim to build an entity recognition model requiring only a few shots of annotated document images. To overcome the data limitation, we propose to leverage the label surface names to better inform the model of the target entity type semantics and also embed the labels into the spatial embedding space to capture the spatial correspondence between regions and labels. Specifically, we go beyond sequence labeling and develop a novel label-aware seq2seq framework, LASER. The proposed model follows a new labeling scheme that generates the label surface names word-by-word explicitly after generating the entities. During training, LASER refines the label semantics by updating the label surface name representations and also strengthens the label-region correlation. In this way, LASER recognizes the entities from document images through both semantic and layout correspondence. Extensive experiments on two benchmark datasets demonstrate the superiority of LASER under the few-shot setting.

## introduction entity recognition lies in the foundation of document image understandings, which aims at extracting word spans that perform certain roles from the document images, such as header, question. distinct from the text-only named entity recognition task, the document images, such as forms, tables, receipts, and multi-columns, provide a perfect scenario to apply multi-modal techniques into practice where the rich layout formats in such document images serve as the new, complementary signals for entity recognition performance in addition to the existing textual data. recent methods @xcite @xcite follow the traditional sequence labeling framework to extract the word spans using the standard iobes tagging schemes @xcite in named entity recognition tasks. entity types are treated as class ids and the semantics of the label surface names are ignored. these methods also largely extend the label space by including combinations of the boundary identifiers (b, i, e, s) and entity types. for instance, when there are 3 target entity types, the extended label space would have 13 (i.e., 4×3+1) dimensions. as a result, they fail to learn from the data efficiently and require extensive datasets and high-quality annotations to create the connection between entities and their entity types. meanwhile, document images typically include various formats and have a high diversity of entities within each page. it is expensive or almost impossible to enumerate all required entity types and obtain enough annotated data for them. moreover, ethical concerns would arise when it comes to the receipts or consent forms, which makes it even harder to collect enough data. due to the inefficiency of traditional methods and the data limitation in real application scenarios, it is necessary to resort to few-shot learning for entity recognition in document images. we aim at exploiting the potential of a limited number of training pages and try to generalize our model on the much larger number of new pages for testing. in our method, we go beyond the sequence labeling framework and reformulate the entity recognition as a sequence-to-sequence task. specifically, we propose a new generative labeling scheme for entity recognition -the label surface name is explicitly generated right after each entity as a part of the target sequence. in this way, different entity types are no longer independent dimensions in the label space and models can leverage the semantic connect between the entities and entity types. to this end, we propose a label-aware sequenceto-sequence framework for entity recognition, laser. our implementation is based on pretrained language model layoutreader 1 @xcite , which is a layout-aware pre-trained sequence-to-sequence model. as shown in figure 1 , laser extends the architecture of layoutreader for our proposed generative labeling scheme to better solve the fewshot entity recognition task for document images. specifically, after generating certain word spans, the model can choose to generate either the following words in the source sequence or label surface names. the entity labels are explicitly inserted in the generated sequence so that the probability of the entity types conditioned on the entity, p (type|entity), can be maximized not only by the signals from the training data but also by the knowledge from the pre-training of the language models. we also embed the label surface names into the spatial embedding space, so the generation of labels is also aware of the correlation between labels and the regions in the page. benefit from the novel generative labeling scheme and the semantics of labels, laser is able to effectively recognize entities in document images with only a limited number of training samples. in contrast, the sequence labeling models use less efficient tagging scheme, thus requiring more data and failing in the few-shot settings. we validate laser using two benchmarks, funsd (guillaume jaume, 2019) and cord-lv1 @xcite . both datasets are from real scenarios and fully-annotated with textual contents and bounding boxes. we compare our model with strong baselines and study the label-entity semantic and spatial correlations. we summarize our contribution as follows. ## problem formulation the few-shot entity recognition in the document images is to take the text and layout inputs from a limited number of training samples to predict the boundary of each entity and classify the entity into categories. given a document image page p, the words within the page are annotated with their textual contents w and the bounding boxes @xmath0, y 0 , x 1 , y 1 ) (top-left and bottom-right corners) by human annotators or the ocr engines, and all the words and bounding boxes are listed in a sequence serving as the inputs from textual and layout modalities. in this way, the entities are spans of these words referring to precise concepts, which makes it possible to conduct entity recognition using sequence labeling or generative labeling scheme. we randomly select a small subset of training samples and evaluate the performance under the k-shot training, where k denotes the number of the training samples. ## our generative labeling scheme we propose our labeling scheme of entity recognition in the generative manner which generates the entity boundaries and the label surface names explicitly. specifically, given an entity @xmath1, w i+1 ..., w j ], we use the @xcite and [e] to denote the boundary of the entity and append the label surface name afterwards. overall, the generative formulation is to generate: according to the generative labeling scheme, the corresponding generated sequence is that: [b] sender [e] question [t] [b] charles duggan [e] answer [t]. 4 our laser framework in this section, we introduce our label-aware sequence-to-sequence framework for entity recog-[sos] w1 bpad b1 b2 b3 b4 b5 bpad b1 b [b] b2 b3 b4 w2 w3 w4 w5 [eos] w1 [b] w2 w3 w4 b [e] [e] b 𝜏 𝜏 b [t] [t] b5 w5 bpad [eos] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 positional embedding spatial embedding word embedding source embeddings target embeddings tgt src tgt allow to attend prevent from attending src laser label-aware sequence-to-sequence for entity recognition h 8 h 9 h 10 h 11 h12 h13 h14 @xmath2. target hidden states next token type classification next token prediction h 7 ∉ 𝐒𝐫𝐜. ∉ 𝐒𝐫𝐜. w1 w2 w3 w4 w5 src. words w4 [b] [e] [t] func. tokens [t] label surface names 𝝉 𝜏ʹ 𝜏ʹʹ 𝝉 uggan duggan [e] answer [t] generate label surface names after entities ... [b] sender [e] question answer header [t] insert spatial identifiers into spatial embedding space nition in document images. first, we introduce our method in a bird's eye view. then we dive into the details of each part including the multi-modal prefix language model, the label-aware generation. ## experiments in this section, we conduct experiments and ablation study on funsd (guillaume jaume, 2019) and cord-lv1 @xcite under few-shot settings. we replace the original label surface names with other tokens to study the importance of semantic meaning. we also plot the heatmaps of the similarity between the spatial identifiers and the spatial embeddings to interpret the spatial correspondence. case studies are also conducted. ## conclusions and future work in this paper, we present laser, a label-aware sequence-to-sequence framework for entity recog-nition in document images under few-shot settings. it benefits from the generative labeling scheme which reformulates the entity recognition task into the sequence-to-sequence setting. the label surface names are embedded into the generated sequence. compared with the sequence labeling methods, laser leverages the rich semantics of the label surface names and overcome the limitation of training data. moreover, we design spatial identifiers for each label and well insert them into the spatial embedding hyperspace. in this way, laser can inference the entity labels from the layout formats perspective and empirical experiments demonstrate our method can learn the layout formats though limited number of training samples. for further research, we will investigate the selection of label surface names and how to better leverage the semantics from the pre-trained sequence-to-sequence models. we also notice that such labeling scheme can cope with unknown categories. we will focus on the generalization of our method.