On Task-personalized Multimodal Few-shot Learning for Visually-rich Document Entity Retrieval (2023)

Abstract: Visually-rich document entity retrieval (VDER), which extracts key information (e.g. date, address) from document images like invoices and receipts, has become an important topic in industrial NLP applications. The emergence of new document types at a constant pace, each with its unique entity types, presents a unique challenge: many documents contain unseen entity types that occur only a couple of times. Addressing this challenge requires models to have the ability of learning entities in a few-shot manner. However, prior works for Few-shot VDER mainly address the problem at the document level with a predefined global entity space, which doesn’t account for the entity-level few-shot scenario: target entity types are locally personalized by each task and entity occurrences vary significantly among documents. To address this unexplored scenario, this paper studies a novel entity-level few-shot VDER task. The challenges lie in the uniqueness of the label space for each task and the increased complexity of out-of-distribution (OOD) contents. To tackle this novel task, we present a task-aware meta-learning based framework, with a central focus on achieving effective task personalization that distinguishes between in-task and out-of-task distribution. Specifically, we adopt a hierarchical decoder (HC) and employ contrastive learning (ContrastProtoNet) to achieve this goal. Furthermore, we introduce a new dataset, FewVEX, to boost future research in the field of entity-level few-shot VDER. Experimental results demonstrate our approaches significantly improve the robustness of popular meta-learning baselines.

## introduction visually-rich document understanding (vrdu) aims to analyze scanned documents composed of ⋆ work done when jiayi chen interned at google. ⋄ corresponding author structured and organized information. as a subproblem of vrdu, the goal of visually-rich document entity retrieval (vder) is to extract key information (e.g., date, address, signatures) from the document images such as invoices and receipts with complementary multimodal information @xcite @xcite . in real-world vder systems, new document types continuously emerge at a constant pace, each with its unique entity spaces (i.e., the set of entity categories that we are going to extract from the document). this poses a substantial challenge: a large amount of documents lack sufficient annotations for their unique entity types, which is referred to as few-shot entities. to tackle this, few-shot visually-rich document entity retrieval (fvder) has become a crucial research topic. despite the importance of fvder, there has been limited amount of prior works in this area. recent efforts have leveraged pre-trained language models @xcite or prompt mechanisms @xcite to obtain transferable knowledge from a source domain and apply it to a target domain, where a small number of document images are labeled for fine-tuning. these prior works address the few-shot problem in a granularity of document level, assuming a globally predefined entity space and balanced entity occurrences across documents. however, in certain real-world scenarios, the few-shot challenge can also manifest at the entity level-i.e., the number of entity occurrences in labeled documents are limited, assuming the situations where entity classes might be locally specialized by each user (task) and their occurrences maintain a significant imbalance across documents. in such scenarios, prior methods struggle to (1) efficiently address the model personalization on each task-specific label space and (2) effectively handle the increased complexity of out-of-distribution contents, with few-shot entity annotations. to provide a complementary research perspec-method instance granularity unseen entities? entity space entity occurrence @xcite document level yes globalized balanced @xcite document level no globalized balanced ours entity level yes personalized imbalanced then, to tackle the limitations of prior methods on this new task, we adopt a meta-learning based framework build upon pretrained language models, along with several proposed techniques for achieving task personalization and handling out-of-task distribution contents. with the help of the metalearning paradigm, (1) the learning experiences on some example tasks could be effectively utilized and (2) the domain gap between the pre-trained model and novel fvder tasks is largely reduced, promoting quicker and more effective fine-tuning on future novel entity types. yet we found popular meta-learning algorithms @xcite @xcite are still not robust to the entity-level n -way soft-k-shot vder tasks. the difficulty is that the background context that does not belong to the task-personalized entity types occupies most of the predictive efforts, and also, such noisy contextual information varies a lot across tasks and documents. to address this, we propose task-aware meta-learning techniques (contrastprotonet, anil+hc, etc.) to allow the meta-learners to be aware of those multi-mode contextual out-of-task distribution and achieve fast adaptation to the task-personalized entity types. furthermore, we present a new dataset, named fewvex, which comprises thousands of entitylevel n -way soft-k-shot vder tasks. we also introduce an automatic dataset generation algorithm, xdr, designed to facilitate future improvement of fewvex, such as the expansion of the number of document types and entity types. specifically, we set an upper bound for entity occurrences and sample across the training documents in a way that guarantees to return soft-balanced few-shot annotations. training documents are constructed in a way such that they cooperatively contain a certain number of entity occurrences per information type. our contributions are summarized as follows. ## entity-level few-shot vder setting general vder. a document image is processed through optical character recognition (ocr) @xcite to form a sequence of tokens @xmath0, x 2 , . . . , x l ], where l is the sequence length and each token x l is composed of multiple modalities n -way soft-k-shot vder in entity level. in real-world few-shot vder systems with label scarcity, individual users often show their personal interests in a small number of new entity types. such user-dependent task personalization gives rise to a novel entity-level task formulation of fewshot vder. formally, an entity-level n -way soft-k-shot vder task @xmath1, q, e} consists of a train (support) set s containing m s documents, a test (query) set q containing m q documents, and a target class set e containing n target entity types distribution over fvder tasks. based on the above formulation for a single fvder task, we further formulate a task distribution p (t ) over fvder tasks. assume there is a large pool of entity types c corresponding to the domain of p (t ). for any task t @xmath2, q i , e i } ∼ p (t ), its target entity types come from the class pool e i ⊂ c. ## methodology we propose a meta-learning (i.e., learning-to-learn) framework to solve the entity-level few-shot vder tasks. different from the recent advancements based on pre-training or prompts @xcite , meta-learning helps to significantly promote quick adaptation and improve model personalization on task-specific entity types. the proposed framework consists of three components: (1) a multimodal encoder (section 3.1) that encodes the document images within a task into a task-dependent embedding space (section 3.2); (2) a token labelling function (section 3.3); and (3) a meta-learner built upon the encoderdecoder model, where we propose two taskpersonalized meta-learning methods (section 3.4). figure 2 shows an overview of the framework. ## fewvex dataset there is no existing benchmark specifically designed for task-personalized entity-level n -way soft-k-shot vder. to facilitate future research on this problem, we create a new dataset, fewvex. source collection foot_0 : fewvex is built from two source datasets: funsd (jaume et al., 2019) contains images of forms annotated by the bounding boxes of 3 types of entities; cord @xcite ) contains scanned receipts annotated by 6 superclasses which are divided into 30 fine-grained subclasses. from them, we collect 1199 document images (d) annotated by 26 entity types (c). meta-learning tasks: we use d and c to construct fewvex, represented by d @xmath3, d tst meta } such that the testing tasks d tst meta focus on novel classes that are unseen in d trn meta during meta-training. to create this, we split c into two separate sets @xmath4, c base ∩ c @xmath5, where c base is used for meta-training and c novel for meta-testing. single task generation: following the definition in eq.( 1 ), each individual entity-level n -way soft-k-shot vder task @xmath6, q, e} in either d trn meta or d tst meta can be generated through the following steps. (1) task-personalized class sampling. the task's target classes e are generated by randomly sampling n entity types from either c base (for the training task) or c novel (for the testing task). (2) document sampling. given the n target classes, we then collect document images that satisfies the n -way, soft k-shot entity occurrences (as in appendix 1). (3) annotation conversion. a task only focuses on its specific n rarely-present entity types. the entities in the original annotated documents, whose class do not belong to e, are replaced with the background o class foot_1 . datasets meta training (from c base ) meta testing (from c novel ) range domains # entity types # tasks domains # entity types # tasks of n fewvex(s) cord 18 3000 cord 5 128 [1, 5] fewvex(m) cord+funsd 20 3000 cord+funsd 6 256 [1, 6] ## conclusions in this paper, we studied the multimodal few-shot learning problem for vder. we started by proposing a new formulation of the fvder problem to be an entity-level, n -way soft-k-shot learning under the framework of meta learning as well as a new dataset, fewvex, which is designed to reflect the practical problems. to solve the new task, we exploited both metric-based and gradient-based metalearning paradigms, along with a new technique we proposed to enhance task personalization via outof-task-distribution awareness. the experiments showed that the proposed methods achieve major improvements over the baselines for fvder. for future works, our approaches might be improved in the following directions: (1) a more robust algorithm that distinguishes between the otd and itd. (2) an advanced decoding process considering graphical structures or implicit correlations between entity instance within each task. (3) exploring the causal role of pretrained models.