D oc Q uery N et: Value Retrieval with Arbitrary Queries for Form-like Documents (2022)

Abstract: We propose, DocQueryNet, a value retrieval method with arbitrary queries for form-like documents to reduce human effort of processing forms. Unlike previous methods that only address a fixed set of field items, our method predicts target value for an arbitrary query based on the understanding of the layout and semantics of a form. To further boost model performance, we propose a simple document language modeling (SimpleDLM) strategy to improve document understanding on large-scale model pre-training. Experimental results show that DocQueryNet outperforms previous designs significantly and the SimpleDLM further improves our performance on value retrieval by around 17% F1 score compared with the state-of-the-art pre-training method. Code is available here, https://github.com/salesforce/QVR-SimpleDLM .

## introduction form-like documents are very commonly used in business workflows. however, tremendous forms are still processed manually everyday. when humans need to extract some relevant information from a form-like document, they proceed as conducting a value retrieval task with some text queries. for example in figure 1 , when humans process a form, they usually have a description of the information (query) that they want to extract (e.g., total page number). then, they examine the form (usually an image or a pdf) carefully to locate the key (e.g., number of pages including cover sheet in figure 1 ) that is most semantically similar to the query and finally infer the target value based on the localized key. this manual process costs a large amount of human efforts as the number of forms and queries increases. automating information extraction from forms is important to alleviate this problem. existing methods formulate the problem as sequence labeling @xcite or field extraction @xcite , where they define a fixed set of items of interest (referred as fields) and train models that only extract values of the pre-defined fields. there are at least two limitations of this formulation. first, forms are very diverse and it is impossible to cover all the items of interest using a fixed set of fields. second, their models are very domain-specific and hard to be utilized for different form types. for example, an invoice field extractor may not be able to process resumes, since different fields are expected for these two form types. to handle diverse queries with a unified model, we formulate the problem as value retrieval with arbitrary queries for form-like documents. under such task formulation, users can extract values from a form by presenting variants of the corresponding keys as queries. we also set up a benchmark for the task by introducing a simple yet effective method, i.e., docquerynet. docquerynet takes an arbitrary query phrase and all the detected optical character recognition (ocr) words with their locations in the form as inputs. then, we model the interactions between the query and the detected words from the document using a transformer-based architecture. the training objective encourages the matching of the positive query-value pairs and discourages that of the negative ones. to further boost the performance, we present simpledlm, a simple document pre-training strategy that makes it more flexible to learn local geometric relations between words/tokens compared to existing pretrained models. experimental results show that docquerynet outperforms the baselines by a large margin under different settings. when initializing using simpledlm, our method is further improved largely by about 17% f1 score compared to initializing using a state-of-the-art (sota) pre-trained model, i.e., layoutlm @xcite ). 2 related work 2.1 information extraction from documents information extraction from documents is crucial for improving the efficiency of form processing and reducing human labor. information extraction is often formulated as a field extraction task. palm et al. 2019 propose an invoice field extractor by using an attend, copy, parse architecture. majumder et al. 2020 present a field-value pairing framework that learns the representations of fields and value candidates in the same feature space using metric learning. nguyen et al. 2021 propose a span extraction approach to extract the start and end of a value for each field. gao et al. 2022 introduce a field extraction system that can be trained with large-scale unlabeled documents. xue et al. 2021 propose form transformations to mimic the variations of forms for robustness evaluation. unlike previous methods that aim to extracting values for a pre-defined set of fields, our method targets at retrieving values for arbitrary queries. ## our approach problem formulation. the inputs to the system are an expected key phrase as the query, q i , and a document d i . d i is represented using a set of ocr words {w i1 , w i2 , ..., w in } and their bounding-box locations {b i1 , b i2 , ..., b in } in the document. the input query phrase is tokenized to q @xmath0, q i2 , ..., q im }. a value retrieval system reads the document and understands the layout and semantics. the goal is to pick a phrase from the ocr words as the value v i for the input query, q i . since the modeling is within one document, we will omit the subscript i for simplicity. ## experiments the following datasets are used in our experiments. iit-cdip @xcite is a large-scale unlabeled document dataset that contains more than 11 million scanned images. following prior works @xcite , our model is pre-trained using this dataset. funsd @xcite is a commonly used dataset for spatial layout analysis. it contains 199 scanned forms with 9,707 semantic entities annotated, where 149 samples are for training and 50 for testing. the semantic linking annotations for all the key-value pairs are provided in the dataset. inv-cdip @xcite is document dataset which contains 350 real invoices for testing. this dataset has key-value pair annotations for 7 commonly used invoice fields including invoice_number, purchase_order, invoice_date, due_date, amount_due, total_amount and total_tax. we evaluate our model using this test set. settings. by default the annotated key texts of each dataset is used as the queries. the location of the keys are not used. models are pre-trained on iit-cdip and fine-tuned on the train set of funsd. more implementation details are in the appendix. evaluation metric. we use f1 score to evaluate models. exact string matching between our pre if a query has multiple value answers, a prediction is counted as correct if it equals one of them. experimental results. the comparisons between our docquerynet and the baseline when pretrained using different approaches are shown in transferring ability to another dataset is important in real-world applications. we measure this ability by directly evaluating the trained models using funsd to the test set of inv-cdip in in practice, we may not assume the input queries match exactly with the actual keys shown in a form. here, we experiment using the field names directly as the queries. using field names is a more conve-nient way, since users don't need to design different queries that match keys for different forms. however, field names are more abstract, which makes the problem more challenging. when using the abstract field name as a query, our method achieves 20.5% f1 which is 14.8% better than our baseline. discussion. we use a dummy box to serve as the location of a query word, since there are no actual boxes for the query words in practice. this choice may make our model lose the sequential ordering information of a query phrase. to investigate this effect, we conduct an experiment and leverage the ordering information of query words by adding different positional ids to different words of a query in our model. the results show that this change decreases our performance from 60.7% to 60.3% in f1 score on funsd and from 49.0% to 47.6% in f1 score on inv-cdip. it may be because that queries with the same meaning might be represented with words in different orders. for example, total amount could be referred to as amount total. ignoring the ordering could help improve our model's generalization ability. ## conclusion we introduce docquerynet, a framework for value retrieval with arbitrary queries for form-like documents. it takes a query and the detected ocr words from a document as inputs, models their interactions and predicts the best value corresponding to the input query. we also present simpledlm as a pre-training strategy to boost performance. experimental results show that our method significantly outperforms previous designs in different settings.