End-to-End Extraction of Structured Information from Business Documents with Pointer-Generator Networks (2020)

Abstract: The predominant approaches for extracting key information from documents resort to classifiers predicting the information type of each word. However, the word level ground truth used for learning is expensive to obtain since it is not naturally produced by the extraction task. In this paper, we discuss a new method for training extraction models directly from the textual value of information. The extracted information of a document is represented as a sequence of tokens in the XML language. We learn to output this representation with a pointer-generator network that alternately copies the document words carrying information and generates the XML tags delimiting the types of information. The ability of our end-to-end method to retrieve structured information is assessed on a large set of business documents. We show that it performs competitively with a standard word classifier without requiring costly word level supervision.

## introduction companies and public administrations are daily confronted with an amount of incoming documents from which they want to extract key information as efficiently as possible. they often face known types of documents such as invoices or purchase orders, thus knowing what information types to extract. however, layouts are highly variable across document issuers as there are no widely adopted specifications constraining the positioning and textual representation of the information within documents. this makes information extraction a challenging task to automate. in addition to the incremental approaches based on layout identification @xcite , a number of recent works have proposed deep neural models to extract information in documents with yet unseen layouts. following @xcite , most of these layoutfree approaches resort to classifiers that predict the information type of each document word. yet, the information extraction task does not offer word level ground truth but rather the normalized textual values of each information type @xcite . the word labels can thus be obtained by matching these textual values with the document words but this process is either time-consuming if manually performed or prone to errors if algorithmically performed. indeed, extracted information may not appear verbatim in the document as its textual values are normalized. for example, the value "2020-03-30" for the document date field may be derived from the group of words "mar 30, 2020". this forces the development of domain specific parsers to retrieve the matching words. also, multiple document words can share the textual value of a extracted field while being semantically distinct, hence imposing additional heuristics for disambiguation. otherwise, a street number may be wrongly interpreted as a product quantity, inducing noise in the word labels. to the best of our knowledge, @xcite is the only related model that directly learns from naturally produced extraction results. however, the authors only tackle the recognition of independent and non-recurring fields such as the document date and omit the extraction of structured entities. such entities are structures composed of multiple field values. within documents, structured information is often contained in tables. for example, a product entity is usually described in a table row with its field values, such as price and quantity, being in different columns. our work is intended to remedy this lack by proposing end-to-end methods for processing structured information. as a first step towards full end-to-end extraction, we focus in this paper on the recognition of fields whose values always appears verbatim in the document, thus eliminating the need for normalization operations. as illustrated in figure 1 , extracted structured information can be represented in a markup language that describes both its content and its structure. among many others, we choose the xml language 1 for its simplicity. we define as many xml tag pairs as the number of entity and field types to extract. a pair of opening and closing field tags delimits a list of words constituting a field instance of the corresponding type. following successful applications of sequenceto-sequence models in many nlp tasks (otter et al., 2020), we employ a recurrent encoder-decoder architecture for outputting such xml representations. conditioned on the sequence of words from the document, the decoder emits one token at each time step: either a xml tag or a word belonging to a field value. since field values are often specific to a document or a issuer, extracted information cannot be generated from a fixed size vocabulary of words. rather, we make use of pointing abilities of neural models @xcite to copy words of the document that carry relevant information. specifically, we adapt the pointer-generator network (pgn) developed by @xcite for text summarization to our extraction needs. we evaluate the resulting model for extracting ordered products from purchase orders. we demonstrate that this end-to-end model performs competitively with a word classifier based model while avoiding 1 https://en.wikipedia.org/wiki/xml to create supervision at the word level. ## approach we assume that the text of a document is already transcribed before extracting its information. for scanned documents, we employ a commercial optical character recognition (ocr) engine for retrieving the text. the method we propose for extracting structured information from a document is depicted in figure 2. the model is derived from the pgn of @xcite proposed for summarization of news articles. the attention-based pointing mechanism allows to accurately reproduce factual information of articles by copying words that are not in the generator's vocabulary, e.g. rare proper nouns. similarly, we take advantage of its pointing ability to copy the words from the document which carry relevant information while allowing the generator to produce the xml tags which structure the extracted information. in the following subsections, we describe in details our model and highlight key differences with the original pgn. ## dataset we train and evaluate our extraction model on a dataset of real world business documents which unfortunately cannot be publicly released. it consists of 219,476 purchase orders emanated by 17,664 issuers between april 2017 and may 2018. the dataset is multilingual and multicultural even if the documents mainly originate from the u.s. the number of purchase orders per issuer is at least 3 and at most 31, ensuring diversity of document layouts. training, validation and test sets have distinct issuers to assess the ability of the model to generalize to unseen layouts. they have been constructed by randomly picking 70 %, 10 % and 20 % of the issuers, respectively. more detailed statistics of the dataset are given in the ## experiments our end-to-end model is compared on this dataset with a baseline extraction method based on a word classifier. this baseline encodes the document as the end-to-end model does, i.e. with the same operations for constructing the word representations r i and the encoder outputs h i . on top of the encoder, a dense layer with softmax activation is added with 5 output units. 4 of these units refer to the beginning and continuation of an instance for id number and quantity fields. the remaining unit is dedicated to the outside class, i.e. for the document words carrying information that we do not want to extract. the words with a predicted probability above 0.5 for one of the 4 field units are associ-ated with the corresponding class, otherwise we attribute the outside class. field instances are then constructed by merging words with beginning and continuing classes of the same field type. finally, each quantity instance is paired with an id number instance to form the product entities. to do so, the hungarian algorithm @xcite solves a linear sum assignment problem with the vertical distance on the document as the matching cost between two field instances. for our task, this pairing strategy is flawless if the field instances are perfectly extracted by the word classifier. the model hyperparameters are chosen according to the micro averaged gain on the validation set. the end-to-end model and baseline share the same hyperparameter values, except the number of blstm cells in each encoder layer that is fixed to 128 and 256 respectively, to ensure similar numbers of trainable parameters. the input character and word vocabularies are derived from the training set. we consider all observed characters while we follow the word vocabulary construction of @xcite designed for business documents. this results in vocabularies of respectively 5,592 and 25,677 elements. their embedding has a size of 16 and 32 and are trained from scratch. the blstm layer iterating over characters of document words has 32 cells. for all blstm layers, each direction has n/2 lstm cells and their output are concatenated to form n-dimensional vectors. the decoder layers have a size of 128 and are initialized by the last states of the encoding blstm layers. at inference time, we decode with a beam search of width 3 and we set the maximum length of the output sequence to the number of words in the document. this results in 1,400,908 and 1,515,733 trainable parameters for the pgn and the word classifier. to deal with exploding gradients, we apply gradient norm clipping @xcite with a clipping threshold of 5. the loss is minimized with the adam optimizer, its learning rate is fixed to 0.001 the first 2 epochs and then exponentially decreases by a factor of 0.8. we stop the training when the micro gain on the validation set has not improved in the last 3 epochs. as suggested in @xcite , the coverage loss is added to the minimized loss only at the end of training, for one additional epoch. we weight its contribution by setting λ = 0.1 as the original value of 1 makes the negative log-likelihood loss increase. the batch size is 8 if the model fits on gpu ram, 4 other-wise. the experiments are carried out on a single nvidia titan x gpu. model training takes from 3 to 10 days for 10 to 15 epochs. due to the computational burden, the hyperparameters values have not been optimized thoroughly. besides, we are not able to train the models on documents with more than 1800 words, which amounts to about 4 % of the training set being put aside. yet, we evaluate the models on all documents of the validation and test sets. the implementation is based on the seq2seq subpackage of tensorflow addons @xcite . ## discussion the main difficulty faced by both models is ambiguity in the ground truth as our dataset has been annotated by users from many distinct companies. some documents contain multiple valid values for a field of a unique product. for example, there may be the references from both recipient and issuer for the id number. the field value which is retained as ground truth then depends on further processing of the extracted information, e.g. integration into a enterprise resource planning (erp) system. this seriously prevents any extraction model from reaching the upper bound of post-processing gain metrics which is 1. besides, the id number field does not always have a dedicated physical column and rather appears within the description column, without keywords clearly introducing the field instances such as in figure 1 . also, its instances are constituted on average of more words than the quantity, making less likely the exact match between predicted and target instances. these additional complications explain the gap of model performances between the two fields. unlike the word classifier based approach, the pgn tends to repeat itself by duplicating some field instances and skipping others. this is especially observed for documents having a large number of products, therefore large output sequences. to mea-sure the impact of these repetitions on metrics, we split the test set into 3 subsets according to the number of products contained in the document: no more than 3, between 4 and 14 and at least 15 entities. the last subset gathers documents with output sequences of at least 122 tokens. we recompute the metrics for each subset and report the micro averaged gains in ## conclusion we discussed a novel method based on pointergenerator networks for extracting structured information from documents. we showed that learning directly from the textual value of information is a viable alternative to the costly word level supervision commonly used in information extraction. in this work, we focused on purchase orders but the approach could be used to extract any structured entity as long as its information type is known at training time. future work should aim to: i) reduce repetitions in the output sequences, ii) add parsing abilities into our encoder-decoder in order to transform the values of copied words. this will allow to process fields that need to be normalized when being extracted.