Representation Learning for Information Extraction from Form-like Documents (2020)

Abstract: We propose a novel approach using representation learning for tackling the problem of extracting structured information from form-like document images. We propose an extraction system that uses knowledge of the types of the target fields to generate extraction candidates and a neural network architecture that learns a dense representation of each candidate based on neighboring words in the document. These learned representations are not only useful in solving the extraction task for unseen document templates from two different domains but are also interpretable, as we show using loss cases.

## introduction in this paper, we present a novel approach to the task of extracting structured information from formlike documents using a learned representation of an extraction candidate. form-like documents like invoices, purchase orders, tax forms and insurance quotes are common in day-to-day business workflows, but current techniques for processing them largely still employ either manual effort or brittle and error-prone heuristics for extraction. the research question motivating our work is the following: given a target set of fields for a particular domain -e.g., due date and total amount for invoices -along with a small set of manually-labeled examples, can we learn to extract these fields from unseen documents? take, for instance, the domain of invoices, a document type that large enterprises often receive and process thousands of times every week @xcite . invoices from different vendors often present the same types of information but with different layouts and positioning. figure 1 shows the headers of invoices from a few different vendors † work done during an internship at google research showing the invoice date (highlighted in green) and number in different layouts. furthermore, invoices from the same supplier even share similar presentation and differ only in specific values. we refer to this unit of visual pattern that is similar across a collection of documents as a template, and the fields of information that are common across templates in a domain as the schema. the schema consists of fields like invoice_date and total_amount, each associated with a type like date and currency. extracting values for these fields from a given document, particularly one belonging to an unseen template, is a challenging problem for many reasons. in contrast to most prior work on information extraction @xcite , templatic documents do not contain much prose. approaches that work well on natural text organized in sentences cannot be applied directly to such documents where spatial layout elements like tables and grid formatting are commonplace. understanding spatial relationships is critical for achieving good extraction performance on such documents. moreover, these documents are usually in pdf or scanned image formats, so these presentation hints are not explicitly available in a markup language. techniques that are successful on html documents such as web pages, including traditional wrapper induction approaches @xcite , are therefore not immediately applicable. recently, there has been a surge in research interest in solving this extraction task adapting techniques in natural language processing @xcite , computer vision @xcite , or combinations thereof @xcite . in contrast to this body of work, we propose an approach based on representation learning for this task. we first generate extraction candidates for each target field using its associated type (e.g., all dates as candidates for invoice_date). we then use a neural network model to learn a dense representation for each extraction candidate independent of the field to which it belongs. we also learn a separate representation for the field itself, and use the similarity between the candidate and field representations to score the candidate according to how likely it is to be the true extraction value for that field. the design of our extraction system rests on a few observations about how information is often laid out in form-like documents (see section 2). an advantage of our representation learning approach is that it allows us to encode certain priors we developed based on these observations into the architecture of the neural network and its input features (see section 4). in fact, our experiments show that our proposed neural architecture outperforms a more naive mlp baseline using the same input features by about 10 f1 points on the extraction task for two different domains (see section 6). furthermore, the learned candidate representations are also meaningful and lend themselves to interpretation, as we show by delving into some loss cases. ## observations about forms we make three key observations about form-like documents that inform our design. observation 1 each field often corresponds to a well-understood type. for example, the only likely extraction candidates for the invoice_date field in an invoice are instances of dates. a currency amount like $25.00 would clearly be incorrect. since there are orders of magnitude fewer dates on an invoice as there are text tokens, limiting the search space by type dramatically simplifies the problem. consequently, we use a library of detectors for several common types such as dates, currency amounts, integers, address portals, emails addresses, etc. to generate candidates. observation 2 each field instance is usually associated with a key phrase that bears an apparent visual relationship with it. consider the invoice excerpt in figure 1(c ). it contains two date instances, only one of which is the true invoice_date, as indicated by the word "date" next to it. similarly, in the bottom-right invoice excerpt, we are easily able to distinguish between the invoice number (indicated by "invoice #") and the purchase order number (indicated by "po #"). we call such indicative words key phrases. proximity is not the only criterion that defines a key phrase. for instance, the word "date" is not the nearest one to the true invoice_date instance in figure 1(c) ; the document number in the line above and the page number below are clearly closer. it is also not the case that the key phrase always occurs on the same line; figure 1(a) shows a case where the key phrase "date" occurs just above the true invoice_date. an effective solution needs to combine the spatial information along with the textual information. fortunately, in our experience, these spatial relationships exhibit only a small number of variations across templates, and these tend to generalize across fields and domains. observation 3 key phrases for a field are largely drawn from a small vocabulary of field-specific variants. in a corpus of invoices we collected, we observed that, as exemplified by the samples in figure 1 , about 93% of the nearly 8400 invoice date instances were associated with key phrases that included the words "date" or "dated" and about 30% included "invoice". only about 7% of invoice dates had neither of these words in their key phrases. similarly, 87% of the nearly 2800 due_date instances in our corpus had key phrases that contained the word "due" and 81% contained "date". we found similar patterns for all other fields we investigated. the fact that there are only a small number of field-specific key phrases suggests that this problem may be tractable with modest amounts of training data. while these observations are applicable to many fields across different document types, there are several exceptions which we plan to tackle in future work. ## extraction pipeline we leveraged the observations laid out in section 2 to build a system to solve the information extraction task for form-like documents. given a document and a target schema, we generate extraction candidates for each field from the document text using the field type. we then score each candidate independently using a neural scoring model. finally, we assign at most one scored candidate as an extraction result for each field. we discuss the stages of this pipeline here, and delve into the architecture of the scoring model in section 4. ## neural scoring model the scoring module takes as input the target field from the schema and the extraction candidate to produce a prediction @xmath0, 1]. while the downstream assignement module consumes the scores directly, the scorer is trained and evaluated as a binary classifier. the target label for a candidate is determined by whether the candidate matches the ground truth for that document and field. an important desideratum for us in the design of the scorer is that it learns a meaningful candidate representation. we propose an architecture where the model learns separate embeddings for the candidate and the field it belongs to, and where the similarity between the candidate and field embeddings determines the score. we believe that such an architecture allows a single model to learn candidate representations that generalize across fields and document templates. we can conceptualize the learned representation of a candidate as encoding what words in its neighborhood form its associated key phrase since, apropos observation 2, the spatial relationships between candidates and their key phrases are observed to generalize across fields. on the other hand, the embedding for a field can be conceptualized as encoding the key phrase variants that are usually indicative of it, apropos observation 3. ## datasets to analyze the performance of our model, we used datasets belonging to two different domains, summarized in invoices we collected two corpora of invoices from different sources. the first corpus, invoices1, contains 14,237 single-page invoices. each invoice was from a different vendor, so the documents do not share any common templates. documents from the same vendor are generated from the same template. the second corpus, invoices2, contains 595 documents belonging to different templates, with no templates in common with invoices1. in all of our experiments, we used a 60-40 split of templates in invoices1 as our training and validation sets, and all the templates in invoices2 as our test set. we asked human annotators to provide us ground truth extraction results for the fields shown in about 95% of documents and fields present the training set had at least one positive example produced by our candidate generators. the fieldlevel recall of our candidate generators varies from about 87% for invoice_id to about 99% for invoice_date. improving the recall of candidate generators is part of our ongoing effort. while the candidate generators have reasonably high recall, their precision varies dramatically from field to field. for common fields like invoice_date and total_amount that are present in nearly all documents, we generate fewer than ten negatives for each positive example. on the other hand, for rare fields like total_tax_amount as well as for fields with low-precision candidate generators such as the alphanum candidate generator for purchase_order, there can sometimes be dozens of negatives for each positive. overall, since the negatives far outnumber the positives, we found it helpful to randomly downsample negatives in the training set to keep at most 40 negatives for each positive per field. the negatives in the validation and test sets were not downsampled. we created a vocabulary of the 512 most frequent tokens, case-normalized, taken from the ocr text of the documents in invoices1. the vocabulary also includes special tokens for numbers ([number]), out-of-vocabulary tokens ([rare]) and padding @xcite ). despite the small size of this vocabulary, it covered at least 95% of words that occurred in key phrases across the entire corpus where excluded words were usually ocr errors. ## related work information extraction from plain text documents for tasks like named entity recognition and relation extraction have benefited from recent advances in deep learning @xcite . however, these techniques are not directly applicable to our task on form-like documents. @xcite attempts to use rnns to extract information from form-like documents. however, they treat each line as a vector of n-grams limiting the resulting accuracy. the importance of understanding visual layout was recognized even in the context of information extraction of webpages in recent work @xcite @xcite . the techniques developed by them are, however, not immediately applicable in our context since we do not have access to the source markup representation for the documents we deal with. a common approach to solving the problem of extracting information from form-like documents is to register templates in a system, match new documents to an existing template, and use an extractor learnt from said template @xcite . the learning problem we tackle in this paper is more ambitious; we seek to generalize to unseen templates. our work is most closely related to recent attempts to combine layout features with text signals. @xcite use a document graph and intro-duce a graph combination model to combine visual and textual signals in the document. @xcite represent a document as a two-dimensional grid of text tokens. @xcite show that using grid information can be useful for information extraction tasks. denk and reisswig (2019) combine the grid-based approach with bert-based text encodings. while an apples-to-apples comparison with these approaches is difficult without a shared benchmark, our system has several advantages: in contrast to the graph-based approaches @xcite we focus on the harder problem of generalizing to unseen templates rather than dealing with the variations within a template. since we are not starting with raw pixels, our approach is computationally less expensive than grid-based approaches. further, we do not require clever heuristics to construct a multi-scale grid that is required for the image-segmentation style abstraction to work well. to the best of our knowledge, our approach of using representation learning for this task is the first of its kind. we gain many of the well-known benefits of this approach @xcite , most notably interpretability. ## conclusion and future work in this paper, we presented a novel approach to the task of extracting structured information from templatic documents using representation learning. we showed that our extraction system using this approach not only has promising accuracy on unseen templates in two different domains, but also that the learned representations lend themselves to interpretation of loss cases. in this initial foray into this challenging problem, we limited our scope to fields with domain-agnostic types like dates and numbers, and which have only one true value in a document. in future work, we hope to tackle repeated fields and learn domainspecific candidate generators. we are also actively investigating how our learned candidate representations can be used for transfer learning to a new domain and, ultimately, in a few-shot setting.