D ocument N et: Bridging the Data Gap in Document Pre-training (2023)

Abstract: Document understanding tasks, in particular, Visually-rich Document Entity Retrieval (VDER), have gained significant attention in recent years thanks to their broad applications in enterprise AI. However, publicly available data have been scarce for these tasks due to strict privacy constraints and high annotation costs. To make things worse, the non-overlapping entity spaces from different datasets hinder the knowledge transfer between document types. In this paper, we propose a method to collect massive-scale and weakly labeled data from the web to benefit the training of VDER models. The collected dataset, named DocumentNet, does not depend on specific document types or entity sets, making it universally applicable to all VDER tasks. The current DocumentNet consists of 30M documents spanning nearly 400 document types organized in a four-level ontology. Experiments on a set of broadly adopted VDER tasks show significant improvements when DocumentNet is incorporated into the pre-training for both classic and few-shot learning settings. With the recent emergence of large language models (LLMs), DocumentNet provides a large data source to extend their multimodal capabilities for VDER.

## introduction document understanding is one of the most errorprone and tedious tasks many people have to handle every day. advancements in machine learning techniques have made it possible to automate such tasks. in a typical visually-rich document entity retrieval (vder) task, pieces of information are retrieved from the document based on a set of predefined entity types, known as the schema. for example, "amount", "date", and "item name" are major parts of an invoice schema. the current setup of vder tasks presents several unique challenges for acquiring sufficient training data. first, the availability of raw document images is greatly limited due to privacy constraints. real-world documents, such as a driver's license or a bank statement, often contain personally identifiable information and are subject to access controls. second, detailed annotation is costly and typically requires intensive training for experienced human annotators. e.g., it takes deep domain knowledge to correctly label different fields in complex tax forms. finally, knowledge sharing between various types of documents is constrained by inconsistent label spaces and contextual logic. for example, the entity sets (i.e., schema) could be mutually exclusive, or the same entity type could take different semantic meanings in different contexts. a number of models have been proposed for vder tasks with various success @xcite @xcite . to tackle the aforementioned challenges, most prior works initialize from a language model followed by bert-style @xcite pre-training on document datasets with additional layout and visual features. however, even the largest dataset currently in use, i.e. iit-cdip @xcite dataset, has a limited size and only reflects a subset of document types. in this paper, we introduce the method of building the documentnet dataset, which enables massive-scale pre-training for vder modeling. documentnet is collected over the internet using a pre-defined ontology, which spans hundreds of document types with a four-level hierarchy. experiments demonstrated that documentnet is the key to advancing the performance on the commonly used funsd @xcite , cord @xcite , and rvl-cdip @xcite benchmarks in both classic and few-shot setups. more recently, llms @xcite have shown great potential for vder tasks given their reasoning capabilities. documentnet provides massive-scale multimodal data to boost the performance of llms for document understanding. ## related work tab. 1 provides an overview of relevant document datasets, with more details in app. b.1. ## documentnet dataset blindly crawling the web for images may seem easy, but it is not a practical solution since most images on the web are not relevant to document types. we need a scalable pipeline to only select the concerned images. broadly, this is achievable via a nearest-neighbor search of relevant keywords in a text-image joint embedding space. first, we financial legal business education common web images o(100b) ontology o(100) keyword embedding semantic embedding retrieve top m nearest neighbors per keyword (m~o(10k)) design a set of query keywords in english, i.e., the document ontology, and encode them into the embedding space of general web images. further, a nearest-neighbor algorithm retrieves the top-k semantically closest images to each query keyword. finally, a deduplication step consolidates all retrieved images across all query keywords. fig. 1 illustrates several exemplar documents retrieved using our provided keywords. ontology creation. each text string in the ontology list serves as a seed to retrieve the most relevant images from the general web image pool. an ideal ontology list should therefore cover a broad spectrum of query keywords across and within the concerned downstream application domains. although algorithmic or generative approaches may exist, in this paper, we manually curated about 400 document-related query keywords that cover domains of finance, business, personal affairs, legal affairs, tax, education, etc. the full ontology hierarchy and keyword list are provided in app. d. image retrieval from ontology. to retrieve only the most relevant document images out of the hundreds of billions of general web images, we leverage a highly efficient nearest neighbor pipeline by defining the similarity metric as the dot product between the semantic feature vectors of the image and each of the target query keywords. here we refer to graph-rise @xcite for the semantic image embedding, and all query keywords are encoded into the same feature space as the images. empirically, we pick the top 10k nearest neighbors in english for each query keyword. note that the same image might be retrieved via multiple semantically similar keywords, so a de-duplication step is needed afterward. we summarize the main pipeline steps in fig. 2 . fig. 3 shows statistical insights of the retrieved 30m document images with the mean and standard deviation histogram over each of the query keywords. the majority of the retrieved images are with mean distance values greater than 0.8 and standard deviations no more than 0.03, indicating high relevance to the document ontology. ## uniformer model to take advantage of all the modalities available in documentnet, we build a lightweight transformer model named uniformer for document pretraining. ## experiments we pre-train uniformer on documentnet and evaluate on two settings: (1) the classic vder setting with the full split of train and test; ## conclusions in this paper, we proposed a method to use massive and noisy web data to benefit the training of vder models. our approach has the benefits of providing a large amount of document data with little cost compared to usual data collection processes in the vder domain. our experiments demonstrated significantly boosted performance in both the classic and the few-shot learning settings.