D o SA : A System to Accelerate Annotations on Business Documents with Human-in-the-Loop (2022)

Abstract: Business documents come in a variety of structures, formats and information needs which makes information extraction a challenging task. Due to these variations, having a document generic model which can work well across all types of documents for all the use cases seems far-fetched. For document-specific models, we would need customized document-specific labels. We introduce DoSA (Document Specific Automated Annotations), which helps annotators in generating initial annotations automatically using our novel bootstrap approach by leveraging document generic datasets and models. These initial annotations can further be reviewed by a human for correctness. An initial document-specific model can be trained and its inference can be used as feedback for generating more automated annotations. These automated annotations can be reviewed by humanin-the-loop for the correctness and a new improved model can be trained using the current model as pre-trained model before going for the next iteration. In this paper, our scope is limited to Form like documents due to limited availability of generic annotated datasets, but this idea can be extended to a variety of other documents as more datasets are built. An opensource ready-to-use implementation is made available on GitHub.

## introduction with the recent advancements in technology and increased adoption of digitization, almost all organizations maintain and exchange business documents in digitized formats like pdfs, scans, faxes, images etc. these documents come in all shape, sizes and format like invoices, emails, medical reports, contracts, scientific papers and many more. the majority of the research has concentrated on documents present on the web that do not adequately capture the complexity of analysis or comprehension required for business documents. these documents require a multidisciplinary approach that includes understanding of layout and structure, computer vision, natural language processing. usually, organizations rely on humans to manually process these documents. the ability to read, understand and interpret these documents is referred as document intelligence (di) or document understanding. there have been recent advancements in this area specifically with deep learning where many architectures @xcite @xcite and annotated datasets @xcite @xcite have been published for various di tasks like document classification, document visual q&a, form understanding etc. these publicly annotated datasets mostly capture only few document types like receipts, scientific articles etc. it becomes challenging & tedious to have annotated public datasets available for various document types which therefore brings in the very need of having customized annotated datasets for specific use cases and document types. to limit our scope, in this paper we are focusing on information extraction from documents that follow a form-like structure. forms are documents that have information usually present in question-answer or key-value format as shown in figure 1 . documents like invoices, driving licenses, passports, medical records, financial statements, tax forms, quotations, payment cards, etc. fall under this category. there has been an effort in building a generic form dataset funsd foot_1 @xcite . funsd is a dataset for form understanding in noisy scanned documents that aim at extracting and structuring the textual content of forms. it proposes an idea where a generic document can be represented via generic information and labels like question (or key), answer (or value), header and others. a model can consume this kind of generic representation to extract generic information. in most of the scenarios, users are interested in document-specific meaningful labels like document number, document date, etc and extracting a subset of information. having a document generic labeling approach results in a noisy and verbose extraction. therefore a need for document-specific annotations arises. commercial solutions like microsoft form recognizer foot_2 and google document ai foot_3 mostly support specific document type prebuilt models and provide a facility to custom train a model for specific documents foot_4 which require new annotations. the sota models of document intelligence use multimodality of the document: text, position and image. due to a change in the format or layout of the document, these modalities might be affected and a new model needs to be trained which will need a new set of annotations. with these, manual annotation becomes repetitive, laborious, expensive and time-consuming. to reduce the time and human effort, we are proposing an active learning based automated annotation system dosa (document specific automated annotations), where the initial set of document-specific annotations are generated by the system which can be reviewed by human annotators for correctness. an initial model can be trained with these annotations and its inference can be taken by the system as feedback to generate annotations on new documents and improve the model incrementally with the human in the loop. the main contribution of this paper is a novel bootstrapping approach to generate automated document-specific labels. to the best of our knowledge, this is the first attempt at generating auto-mated annotations on business documents that contains visual and layout structure information along with the text. all other previous approaches have mostly focused on web or text @xcite or images @xcite . we have seen an attempt on research documents but the scope was limited to automatically annotating documents with topic @xcite . ## dosa system a high-level flow of the dosa system is shown in figure 2 , where initial document-specific annotations are generated by document-generic model which is later reviewed by a human for correctness. with these initial annotations, a document-specific model is trained and its inference is taken as feedback to annotate further documents. a human will again review and correct the annotations and the reviewed documents can be added back to training for further improving the model. as the model matures, eventually the user would end up correcting a minimal number of annotated fields/documents. the journey for the model to get more precise with less human feedback is achieved by employing active learning strategies like uncertainty-based sampling which improve the document-specific model performance in very few iterations. ## conclusion and future work in this work, we presented dosa, a system to generate document specific annotations from model built on document generic datasets. our scope was limited to form like document which can be further enhanced with the availability of new type of generic datasets. this system in current state can only take one type of document to generate one set of annotations. in case the users have multiple type of documents, they have to group the documents by type beforehand and use this system for individual groups. a layer can be added on top of dosa system to automatically classify the documents and use dosa for individual groups. as this work is still in progress, in this paper we focused on proposing this idea. we are planning to discuss the effectiveness of our proposed approaches and overall system in the near future.