Selective Labeling: How to Radically Lower Data-Labeling Costs for Document Extraction Models (2023)

Abstract: Building automatic extraction models for visually rich documents like invoices, receipts, bills, tax forms, etc. has received significant attention lately. A key bottleneck in developing extraction models for new document types is the cost of acquiring the several thousand high-quality labeled documents that are needed to train a model with acceptable accuracy. In this paper, we propose selective labeling as a solution to this problem. The key insight is to simplify the labeling task to provide “yes/no” labels for candidate extractions predicted by a model trained on partially labeled documents. We combine this with a custom active learning strategy to find the predictions that the model is most uncertain about. We show through experiments on document types drawn from 3 different domains that selective labeling can reduce the cost of acquiring labeled data by 10 \times with a negligible loss in accuracy.

## introduction visually rich documents such as invoices, receipts, paystubs, insurance statements, tax forms, etc. are pervasive in business workflows. the tedious and error-prone nature of these workflows has led to much recent research into machine learning methods for automatically extracting structured information from such documents @xcite @xcite @xcite . given a target document type with an associated set of fields of interest, as well as a set of humanannotated training documents, these systems learn to automatically extract the values for these fields from documents with unseen layouts. a critical hurdle in the development of highquality extraction systems is the large cost of acquiring and annotating training documents belonging to the target types. the human annotators often require training not only on the use of the annotation tools but also on the definitions and semantics of the target document type. the annotation task can be tedious and cognitively taxing, requiring the annotator to identify and draw bounding boxes around dozens of target fields in each document. this data efficiency requirement has not gone unnoticed in the research literature on this topic. however, even with model pre-training @xcite , transfer learning from an out-of-domain labeled corpus @xcite , and data-programming @xcite , empirical evidence suggests that performing well on a new target document type like procurement, banking, insurance, mortgage, etc. still requires thousands of annotated documents, amounting to hundreds of hours of human labor @xcite . the cost of acquiring high quality labeled data for hundreds of document types is prohibitively expensive and is currently a key bottleneck. we could apply active learning strategies to select a few but informative documents for human review @xcite , however the cost-reducing effect of this approach is limited, as it requires annotating the span in every selected document for every field. many of these annotations are repetitive, and often not very informative if a model can already extract those fields easily. in fact, our initial experiments with a document-level active learning approach yielded modest results that cut down the number of documents required to get to the same level of quality as random selection by approximately 20%. in this paper, we propose a technique called selective labeling that reduces this cost by 10×. the key insight is to combine two ideas: first, we redefine and simplify the task performed by the human annotators -rather than labeling every target field in every document by drawing bounding boxes around their values, we ask them to simply verify whether a proposed bounding box is correct. this binary "yes/no" annotation task is faster and imposes a lighter cognitive burden on the an-figure 1 : a classic annotation task: even labeling 9 fields in this toy invoice imposes a heavy cognitive burden on the annotator, while real-world documents are significantly more complicated (red). a "yes/no" annotation task: presenting a proposed span and asking the annotator to accept or reject the label is simpler, quicker, and less prone to errors (green). we interleave rounds of such human annotation with training a model that is capable of consuming partially labeled documents. in combination, our proposed approach dramatically improves the efficiency of the annotation workflow for this extraction task. in fact, through experiments on document types drawn from multiple domains, we show that selective labeling allows us to build models with 10× lower annotation cost while achieving nearly the same accuracy as a model trained on several thousand labeled documents. note that our goal in this paper is not to advance the state-of-the-art in active learning, nor to propose a more data-efficient model for extraction from layout-heavy documents. our main contribution is that we demonstrate that a novel combination of an existing active-learning strategy with an existing extraction model can be used to dramatically cut down the primary bottleneck in developing extraction models for visually rich documents. ## background we first describe how a typical annotation task is set up to acquire labeled documents. we point out two major deficiencies with this approach before outlining an alternative that takes advantage of the characteristics of this domain. we then describe the assumptions underlying our approach. ## selective labeling methodology we first provide an overview of the selective labeling framework before describing various uncertainty measures and ways to deal with the unique characteristics of this setting, such as varying difficulty for different fields. ## datasets and setups to evaluate the performance of our proposed methods, we use datasets belonging to three different domains, summarized in ## results in this section, we provide evidence to prove that selective labeling reduces the annotation cost by 10x in different domains and analysis to support the design choices including number of selection rounds, selection and sampling strategies. ## related work form extraction. there have been numerous recent studies on information extraction for form-like documents. existing approaches either individually categorize every text span in the document @xcite or formulate the task into a sequence modeling problem @xcite @xcite and encode texts, layouts, and visual patterns into feature space. while these approaches produce state-of-the-art extraction systems, they require large amounts of labeled training data to do so. we do not propose a new model architecture but instead, focus on the cost of acquiring labeled data for such extraction models. active learning. we refer to @xcite @xcite for an extensive review of the literature. two popular approaches for requesting annotation are uncertainty-based selection @xcite and committee-based selection @xcite @xcite . researchers seek to increase the diversity by forcing the selection to cover a more representative set of examples @xcite @xcite or incorporating discriminative learning @xcite . researchers have studied combining active learning with deep learning. most of the advanced strategies such as coreset @xcite , dropout @xcite , discriminative active learning @xcite . @xcite @xcite @xcite @xcite ) studies deep active learning on various nlp tasks. while we don't focus on inventing new active learning methods, we are the first to customize existing methods to reduce annotation costs in form-like document extraction. ## conclusion and future work we propose selective labeling that dramatically cuts down the primary dataset annotation bottleneck in developing extraction models for visually rich documents. there are several future avenues for investigation. first, we simplified the annotation task to a binary "yes/no" question. another approach is to allow the annotator to either accept the candidate annotation, or correct it -either by deleting it or by adjusting the bounding box. for certain text fields it can be valuable to adjust spans to include/exclude details like salutations from a name field ("mr.", "dr." etc.) or names from an address. the cost model for such an option is more complex than "yes/no", but can be used to build on the results in this paper. second, many recent approaches @xcite treat this as a sequence-labeling problem and use a layout-aware language model. adapting selec-tive labeling to a sequence-labeling model requires tackling several problems: a) getting uncertainty estimates for a given span from a sequence labeling model, b) training a sequence labeling model using partially labeled documents, and c) optionally, eschewing candidate-generators entirely and generating both candidate-spans and their uncertainty estimates form the sequence labeling model. we hope to explore the multiple ways to tackle each of these problems in future work.