Key Information Extraction in Purchase Documents using Deep Learning and Rule-based Corrections (2022)

Abstract: Deep Learning (DL) is dominating the fields of Natural Language Processing (NLP) and Computer Vision (CV) in the recent times. However, DL commonly relies on the availability of large data annotations, so other alternative or complementary pattern-based techniques can help to improve results. In this paper, we build upon Key Information Extraction (KIE) in purchase documents using both DL and rule-based corrections. Our system initially trusts on Optical Character Recognition (OCR) and text understanding based on entity tagging to identify purchase facts of interest (e.g., product codes, descriptions, quantities, or prices). These facts are then linked to a same product group, which is recognized by means of line detection and some grouping heuristics. Once these DL approaches are processed, we contribute several mechanisms consisting of rule-based corrections for improving the baseline DL predictions. We prove the enhancements provided by these rule-based corrections over the baseline DL results in the presented experiments for purchase documents from public and NielsenIQ datasets.

## introduction the intersection between nlp and cv algorithms is a key factor in systems that require processing visual and textual features. there are several use cases in the retailing industry in which this combination is typically applied, such as automated item coding @xcite , classification of promotions in digital leaflets @xcite or the application of visual question answering (vqa) to store observation systems @xcite , among others. in general terms, a great part of these use cases rely on kie for automatically obtaining data of interest from varied sources related to images and documents. one of the main approaches based on kie for retail and consumer measurement is focused on the automated recognition of data from purchase documents, such as receipts and invoices. figure 1 : example of kie in purchase documents from the nielseniq dataset. we can see entities tagged as product codes in gray, descriptions in green, quantities in yellow and total prices in red. moreover, the regions corresponding to a same product jointly with their linked entities are marked in a blue to purple scale. the information of interest commonly acquired by purchase decoding systems is associated with the purchased products printed in the documents to be recognized at item level, as depicted in fig. 1 . here, there are several entities of text related to the characteristics of a purchased product that are generally extracted, such as the following ones: in this paper, we propose a novel approach for kie in purchase documents using dl and rule-based corrections. the presented dl architecture initially extracts the words contained in the document using ocr facilities. the recognized words jointly with the processed image are used as input of an entity tagging model based on transformers and convolutional neural networks (cnns), which is able to predict the facts associated with the purchased products. then, we compute a line detection method based on gnns (graph neural networks) with the aim of grouping all the text lines corresponding to a same product for entity linking. finally, our pipeline applies some rule-based refinements to the previously predicted tags to correct possible inconsistencies. we observed that standard dl proposals for entity tagging decrease their performance in complex purchase documents with low image quality, so false positives and false negatives commonly appear within the set of predictions. however, we know some specific business rules related to purchase documents for product descriptions, codes, quantities and prices that can be applied in postprocessing, with the aim of enhancing the global accuracy of the system. in this way, we can take advantage of the patterns learned by dl and combine them with the human knowledge about the rules related to the purchase document decoding use case, so both can be applied in a complementary way to improve the quality of our whole solution. in this regard, the main contributions derived from the research presented in this paper are the following ones: the contents of the paper are organized as follows: a review of the main state-of-the-art methods for kie in document decoding is presented in section 2. our approach for kie in purchase documents using dl and rule-based corrections is detailed in section 3. the main experiments and results related to our dl proposal and the accuracy improvements of rule-based corrections are discussed in section 4. the final conclusions associated with our research and some insights about future works are finally summarized in section 5. ## related work the rise of dl has provided powerful tools to the fields of nlp and cv, which are the base of kie systems focused on purchase documents decoding. this research line has also grown thanks to the proliferation of related public datasets, such as cord @xcite , sroie @xcite or funsd @xcite . moreover, companies working on retail intelligence also generate large amounts of data that are typically used for researching in real use cases, such as nielseniq. in the intersection between nlp and cv, ocr techniques commonly represent the starting point for recognizing the text contained in a document or image. methods based on ocr examine images pixel by pixel, looking for shapes that match the character traits. the state of the art in ocr comprises solutions that are open source and proprietary. tesseract ocr is one of the most effective open-source approaches @xcite . however, proprietary solutions such as amazon or google ocr are currently obtaining much better results in text recognition @xcite . once the text is recognized, purchase document decoding systems typically need to understand certain parts or extract specific information. in this regard, the recent popularization of transformers @xcite and architectures such as bert @xcite in the nlp community has provided new tools for achieving the desired results. among the different possibilities of text understanding, document decoding systems for purchase facts are commonly associated with entity tagging. this approach is applied with the aim of extracting specific product information in purchase documents, such as product description, code, quantity or price. entity tagging is another state-of-theart field that has been benefited by transformers. for instance, the architecture defined by pick @xcite is typically used in entity tagging systems, because its combination of transformers to get text embeddings and cnns to obtain image embeddings @xcite provides one of the top performances in the recent literature. another popular technique for entity tagging is layoutlm and its different versions @xcite @xcite , including an approach focused on multilingual capabilities @xcite . apart from entity tagging, entity linking is also typically required to match each purchase tag with each respective product in the purchase document. here, state-of-the-art techniques such as spade @xcite or bros @xcite propose end-to-end entity linking approaches. however, our proposal considers a method based on two stages, consisting of an initial entity tagging which is then linked by using line detection algorithms and rule-based heuristics for product lines grouping. for line detection, we take advantage of recent gnn proposals such as @xcite or @xcite . unfortunately, the state of the art does not consider a lot of solutions for correcting wrong entity tagging predictions, which is something typical in complex documents. for this reason, we propose novel rule-based corrections for improving automated entity tagging in purchase documents. ## our approach in this section, our whole pipeline proposed for kie in purchase documents using dl and rule-based corrections is described. the core architecture builds upon dl in order to obtain the initial predictions for recognizing the text in the documents and tag words into varied categories from the different purchased products characteristics. once these baseline predictions are provided by the dl pipeline, the implemented rule-based corrections are applied to refine the final output. we explain both dl and rule-based schemas with the aim of fully understanding the complementary solutions designed to extract the information of interest from purchase documents. ## conclusions and future work the dl era has provided outstanding tools to solve complex problems in fields such as nlp and cv. unfortunately, generalization and scalability to different use cases are still an open challenge. in kie for purchase documents, dl results can be degraded in cases associated with complex formats, low resolution or multilingual scenarios, among others. however, we have demostrated how rule-based corrections can complement dl architectures to enhance the performance of document decoding in the most difficult scenarios. we have presented a basic set of rules based on business knowledge for correcting some common purchase tags, but more general rules could be explored in the future. besides, further works could also include confidences application to rule-based corrections or similar post-processing rules for improving other kie tasks such as ocr extraction.