Original: Combining Deep Learning and Reasoning for Address Detection in Unstructured Text Documents
Revised: Combining Deep Learning and Reasoning for Address Detection in Unstructured Text Documents

Original: * 1 , * 1 , * 1 , *1 REDACTED 2 REDACTED *,*,*,*

Original: Abstract
Revised: Abstract

Original: Extracting information from unstructured text documents is a demanding task, since these documents can have a broad variety of different layouts and a non-trivial reading order, like it is the case for multi-column documents or nested tables.
Revised: Extracting information from unstructured text documents is a demanding task, since these documents can have a broad variety of different layouts and a non-trivial reading order, like it is the case for multi-column documents or nested tables.

Original: Additionally, many business documents are received in paper form, meaning that the textual contents need to be digitized before further analysis.
Revised: Additionally, many business documents are received in paper form, meaning that the textual contents need to be digitized before further analysis.

Original: Nonetheless, automatic detec- tion and capturing of crucial document information like the sender address would boost many companies’ processing ef- ficiency.
Revised: Nonetheless, automatic detec- tion and capturing of crucial document information like the sender address would boost many companies’ processing ef- ficiency.

Original: In this work we propose a hybrid approach that combines deep learning with reasoning for finding and extracting addresses from unstructured text documents.
Revised: In this work we propose a hybrid approach that combines deep learning with reasoning for finding and extracting addresses from unstructured text documents.

Original: We use a visual deep learning model to detect the boundaries of possible ad- dress regions on the scanned document images and validate these results by analyzing the containing text using domain knowledge represented as a rule based system.
Revised: We use a visual deep learning model to detect the boundaries of possible ad- dress regions on the scanned document images and validate these results by analyzing the containing text using domain knowledge represented as a rule based system.

Original: Introduction
Revised: Introduction

Original: Many businesses need to deal with scanned text documents, such as invoices or purchase orders, on a daily basis.
Revised: Many businesses need to deal with scanned text documents, such as invoices or purchase orders, on a daily basis.

Original: This continuing trend is reflected by the importance of OCR solutions worldwide (Grand View Research Inc. 2019).
Revised: This continuing trend is reflected by the importance of OCR solutions worldwide (Grand View Research Inc. 2019).

Original: Nowadays, many powerful tools like OCR engines or deep learning models, trained on large quantities of data, are freely available and can be used for the automated analysis of such documents.
Revised: Nowadays, many powerful tools like OCR engines or deep learning models, trained on large quantities of data, are freely available and can be used for the automated analysis of such documents.

Original: However, adoption of these generic tools for specific tasks is still error-prone, since human expert domain knowledge is required in many use cases.
Revised: However, adoption of these generic tools for specific tasks is still error-prone, since human expert domain knowledge is required in many use cases.

Original: This a priori domain knowledge is usually not well represented or integrated in modern AI models (Stevens et al. 2020).
Revised: This a priori domain knowledge is usually not well represented or integrated in modern AI models (Stevens et al. 2020).

Original: For instance, before continuing to process the contents of a document, we found that a first crucial step for businesses is usually a proper identification of the correct meta data, like sender and receiver address, because this information might influence the subsequent processing flow.
Revised: For instance, before continuing to process the contents of a document, we found that a first crucial step for businesses is usually a proper identification of the correct meta data, like sender and receiver address, because this information might influence the subsequent processing flow.

Original: Humans can solve this task easily, since they know about typical locations of address information on document headers as well as the internal structure of addresses that all entities have to follow.
Revised: Humans can solve this task easily, since they know about typical locations of address information on document headers as well as the internal structure of addresses that all entities have to follow.

Original: Our experience showed, that pure machine learning solutions can heavily profit from such human reasoning assessments, pro- vided these information can be represented and integrated in a learning system.
Revised: Our experience showed, that pure machine learning solutions can heavily profit from such human reasoning assessments, provided these information can be represented and integrated in a learning system.

Original: For tackling this problem of address detection, we built a processing pipeline, consisting of different datadriven and rule-based components, which was developed in cooperation with *REDACTED*.
Revised: For tackling this problem of address detection, we built a processing pipeline, consisting of different data-driven and rule-based components, which was developed in cooperation with the University of Leipzig.

Original: Many of the components originally come from the open source OCR-D project (Neudecker et al. 2019), which aims to provide means for analysis of scanned documents by enabling usage of other (heterogeneous) open source processing tools, like e.g. Tesseract OCR (Smith 2019), in a standardized way.
Revised: Many of the components originally come from the open source OCRD project (Neudecker et al. 2019), which aims to provide means for analysis of scanned documents by enabling usage of other (heterogeneous) open source processing tools, like e.g. Tesseract OCR (Smith 2019), in a standardized way.

Original: Our work is structured as follows: We first give a short overview of related work, followed by a comparison of data-driven and reasoning-based approaches in general.
Revised: Our work is structured as follows: We first give a short overview of related work, followed by a comparison of data-driven and reasoning-based approaches in general.

Original: This forms the basis for further technology decisions during the pipeline implementation described in the subsequent section.
Revised: This forms the basis for further technology decisions during the pipeline implementation described in the subsequent section.

Original: Thereafter, we describe the evaluation of our approach using a document test set and conclude our work with an outlook and future work.
Revised: Thereafter, we describe the evaluation of our approach using a document test set and conclude our work with an outlook and future work.

Original: Related Work
Revised: Related Work

Original: Extracting address information from unstructured texts is not a new use case: Long before the rise of modern AI and deep learning technology, rule-based approaches and algorithms aimed to detect post addresses in digitized texts, for example on web pages (Lin et al. 2005).

Original: With the upcoming of machine learning techniques, more data driven approaches were applied, for example using Conditional Random Fields (Chang and Li 2010).

Original: However, when dealing with complex document layouts or noisy scanned documents, the correct identification and boundary detection in terms of entity extraction remains error-prone.

Original: State-of-the-art document analysis models, like the multimodal
Revised: State-of-the-art document analysis models, like the multimodal

Original: LayoutLM architecture released by Microsoft, target this problem by combining both visual and text information in the model architecture, even for multiple languages (Xu et al. 2021).
Revised: LayoutLM architecture released by Microsoft, target this problem by combining both visual and text information in the model architecture, even for multiple languages (Xu et al. 2021).

Original: Nonetheless, results achieved by pure data driven approaches always retain some amount of uncertainty regarding plausibility and validity from the view of a human expert.
Revised: Nonetheless, results achieved by pure data driven approaches always retain some amount of uncertainty regarding plausibility and validity from the view of a human expert.

Original: Therefore, we propose an approach that merges machine intelligence and human reasoning for improved address detection.
Revised: Therefore, we propose an approach that merges machine intelligence and human reasoning for improved address detection.

Revised: Extracting address information from unstructured texts is not a new use case: Long before the rise of modern AI and deep learning technology, rule-based approaches and algorithms aimed to detect post addresses in digitized texts, for example on web pages (Lin et al. 2005).

Revised: With the upcoming of machine learning techniques, more data driven approaches were applied, for example using Conditional Random Fields (Chang and Li 2010).

Revised: However, when dealing

Original: [Table]
Revised: [Table]

Revised: or noisy scanned documents, the correct identification and boundary detection in terms of entity extraction remains error-prone.

Revised: A similar approach to our pipeline was developed by (Vishwanath et al. 2019), where an enterprise-based platform was built that attempts to populate a general relational hierarchy using document templates prior to information extraction and then mapping the found information and hierarchy into a database.

Revised: While the authors and our work share similar preprocessing approaches like noise-reduction techniques or information retrieval using OCR, we rely on capturing the relational and document structure information implicitly by a combination of prior domain knowledge or business rules and the deep learning model without templates.

Original:  and Reasoning for Address Extraction
Revised: The authors of (Sunder et al. 2019) propose a twofolded approach for finding specific text entities. In the first step, they use a pre-trained model to generate a relational database similar to (Vishwanath et al. 2019) and afterwards apply deductive reasoning to learn extraction programs. In this paper, we focus more on specific use-cases for information localization and extraction, as well as validation of ambiguous entities by extracting structural information without predefined templates using deep neural networks and incorporating prior domain knowledge or business rules, in the aforementioned document classes. Comparison of Deep Learning and Reasoning for Address Extraction

Original: Our work focuses on the extraction of sender or receiver address data in business correspondence, as they appear in invoices, purchase orders or customer letters.
Revised: Our work focuses on the extraction of sender or receiver address data in business correspondence, as they appear in invoices, purchase orders or customer letters.

Original: This use case was chosen based on the feedback of many project partners and customers:
Revised: This use case was chosen based on the feedback of many project partners and customers:

Original: Extracting address data and matching the sender or receiver of documents with a database is very often the first step in automated document processing.
Revised: Extracting address data and matching the sender or receiver of documents with a database is very often the first step in automated document processing.

Original: The restriction to business correspondence implies some specificity that shows the strengths and limits of sole deep learning and reasoning approaches listed in Table 1, which led to the hybrid approach showcased in this work.
Revised: The restriction to business correspondence implies some specificity that shows the strengths and limits of sole deep learning and reasoning approaches listed in Table 1, which led to the hybrid approach showcased in this work.

Original: Recent work has shown that approaches such as transfer learning may reduce the downsides of deep learning when little training data is available (Mart´ınek, Lenc, and
Revised: Recent work has shown that approaches such as transfer learning may reduce the downsides of deep learning when little training data is available (Mart´ınek, Lenc, and

Original: Kr´al 2020).
Revised: Kr´al 2020).

Original: The application of transfer learning, however, requires the availability of sufficiently good foundational models (Bommasani et al. 2021).
Revised: The application of transfer learning, however, requires the availability of sufficiently good foundational models (Bommasani et al. 2021).

Original: These may be available for historical – mostly anglophone – business correspondence where training data is available, like the RVL-CDIP data set (Harley, Ufkes, and Derpanis 2015).
Revised: These may be available for historical – mostly anglophone – business correspondence where training data is available, like the RVL-CDIP data set (Harley, Ufkes, and Derpanis 2015).

Original: To the best of our knowledge, however, no sufficient publicly available datasets exists for business correspondence in German.
Revised: To the best of our knowledge, however, no sufficient publicly available datasets exists for business correspondence in German.

Original: Overall, the comparison of both pure methods indicates, that a hybrid approach combining the generalization capability of deep learning to a variety of layouts and the reduction of false positives in typical reasoning, appears promising.
Revised: Overall, the comparison of both pure methods indicates, that a hybrid approach combining the generalization capability [Figure] Incoming [Figure]

Original:  Hybrid Method for Address Extraction Problem Definition
Revised: scanned) [Figure] of deep learning to a variety of layouts and the reduction of false positives in typical reasoning, appears promising. Hybrid Method for Address Extraction Problem Definition

Original: In our use case, the goal of address detection is achieved in two steps:
Revised: In our use case, the goal of address detection is achieved in two steps:

Original: Detection of possible address candidates on the (scanned) pages of business documents 2.
Revised: Detection of possible address candidates on the (scanned) pages of business documents 2.

Original: Assignment of the correct address class for each candidate ( sender , receiver , other )
Revised: Assignment of the correct address class for each candidate ( sender , receiver , other )

Original: Note that the found sender and receiver address are limited to one instance per document page, while the amount of other addresses remains unlimited.
Revised: Note that the found sender and receiver address are limited to one instance per document page, while the amount of other addresses remains unlimited.

Original: An address candidate is considered as detected correctly, if the text within its region boundaries contains at least the ZIP Code and the city name of the correct address.
Revised: An address candidate is considered as detected correctly, if the text within its region boundaries contains at least the ZIP Code and the city name of the correct address.

Original: Pipeline Realization
Revised: Pipeline Realization

Original: We built a prototype in form of the processing pipeline which utilizes a hybrid workflow as is shown in Figure 1, including both: deep learning and rule-based pattern matching.
Revised: We built a prototype in form of the processing pipeline which utilizes a hybrid workflow as is shown in Figure 1, including both: deep learning and rule-based pattern matching.

Original: The choice of technology for the individual pipeline steps was based on the considerations listed in Table 2.Pipeline Component Deep Learning Reasoning
Revised: The choice of technology for the individual pipeline steps was based on the considerations listed in Table 2.

Original: Consequence / Technology Choice

Original: BinarizationStandard

Original: [Table] processing [Table] Nowadays
Revised: In the following, we describe the realization of different processing steps in the pipeline.

Original: [Table]

Original: ing

Original: [Table]

Original: proach

Original: Address Region Detection

Original: [Table]

Original: suitable

Original: [Table]

Original: complex

Original: Visual

Original: [Table]

Original: approach was chosen. Address Text Detection Customized Named
Revised: For many of the components, modules from the OCR-D project were adapted to our specific use case of address detection 1 .

Original: [Table]

Original: thinkable,

Original: [Table] needed [Table] checks
Revised: In general, the pipeline is working on scanned documents like invoice or delivery documents or any other type of correspondence letters that might arrive in the form of PDFs without text layer or any image format.

Original: [Table]

Original: well [Table] - ture of German addresses At the pipeline entry, all incoming documents are converted to 300dpi TIF files and the pages of each document are segregated.
Revised: At the pipeline entry, all incoming documents are converted to 300dpi TIF files and the pages of each document are segregated.

Original: In the following step, the file is binarized with the tool Olena, included in the OCR-D project.
Revised: In the following step, the file is binarized with the tool Olena, included in the OCR-D project.

Original: Afterwards, a general visual document segmentation is performed using Tesseract and OCRopy, an open-source document analysis framework which allows identification of table regions and structures (Breuel 2017).
Revised: Afterwards, a general visual document segmentation is performed using Tesseract and OCRopy, an open-source document analysis framework which allows identification of table regions and structures (Breuel 2017).

Original: As a result of this step, we receive all text regions and lines that have been identified, together with their bounding box coordinates on the page.
Revised: As a result of this step, we receive all text regions and lines that have been identified, together with their bounding box coordinates on the page.

Original: The OCR of Tesseract 4.0 is applied for extracting the textual contents of the detected lines.
Revised: The OCR of Tesseract 4.0 is applied for extracting the textual contents of the detected lines.

Original: For identification of address region candidates, we send the OCR results to our reasoning-based Address-API component.
Revised: For identification of address region candidates, we send the OCR results to our reasoning-based Address-API component.

Original: The module recognizes address-like sections using human crafted rules.
Revised: The module recognizes address-like sections using human crafted rules.

Original: Specifically, the module detects individual address components like ZIP codes, cities, streets or names of persons and organizations by using regular expressions and reference lists. 
Revised: Specifically, two steps of processing are performed: Detection of address components : The module detects individual address components like ZIP codes, cities, streets or names of persons and organizations by using regular expressions and reference lists. For instance, ZIP codes are found using a regular term matching all string tokens containing exactly 5 digits.

Revised: These candidates are then examined using a list of valid German ZIP Codes and discarded in the following steps if they are found to be invalid.

Original:  The found  entities  are validated according to the rules of German address structures (e.g. street is followed by house number, followed by ZIP code with always 5 digits and city name, etc.).
Revised: Composition of complex address entities : The found base entities from the previous step are composed to complex address entities according to the rules [Table] [STREET and HOUSENUMBER] [ZIP CODE and CITY].

Original: This way, a prediction can be made, whether a given section is a proper an address.
Revised: This way, a prediction can be made, whether a given section is a proper an address.

Revised: In detail, a confidence score is computed depending on the individual validation scores of the sub entities and their plausibility in the whole address context.

Original:  Finally, the address prediction for each text line is combined with the visual information, like region image, bound- the REDACTED aries and location on the document page.
Revised: For instance, ZIP codes are matched against city names found close to them by lookups in the German ZIP-city registry. Finally, the address prediction for each text line is combined with the visual information, like region image, boundaries and location on the document page.

Original: This is done by feeding both visual and text classification information into a modified Deep Learning model for predicting the final address region boundaries together with their address label ( sender , receiver , other ).
Revised: This is done by feeding both visual and text classification information into a modified Deep Learning model for predicting the final address region boundaries together with their address label ( sender , receiver , other ).

Original: More precisely, we fine-tuned a
Revised: More precisely, we fine-tuned a

Original: Mask R-CNN model (Abdulla 2017) using both synthetically generated letters (with corresponding address labels) and about 300 annotated internal invoice documents.
Revised: Mask R-CNN model (Abdulla 2017) using both synthetically generated letters (with corresponding address labels) and about 300 annotated internal invoice documents of different layouts and scan quality.

Revised: Since these documents have been taken from our internal ordering processes, the often also contain additional noise like small portions of handwritten notes or company stamps that have been put on the pages, also sometimes covering some of the original text data.

Original: At the end of the pipeline, the textual contents of the predicted address regions for sender , receiver and other are again validated by the same reasoning-based approach already used in the previous step (Address-API) to make sure we really received a valid address region from the model. 
Revised: At the end of the pipeline, the textual contents of the predicted address regions for sender , receiver and other are again validated by the same reasoning-based approach already used in the previous step (Address-API) to make sure we really received a valid address region from the model. This also includes utilization of a geocoding service for normalizing and validating detect address information.

Original: Note that we have not implemented this final integration yet, thus the following evaluation is only based on the results predicted by the deep learning model mentioned in the last step.
Revised: Note that although the conception and implementation of these pipeline steps have mostly been done, our approach described here is still work in progress. Hence, the following evaluation is still based on the results predicted by the deep learning model mentioned in the last step and thus constituting only a first step towards indicating the potential of our hybrid approach for address detection.

Original: We expect the results to further improve by introducing this second level of (repeated) textual validation.
Revised: We expect the results to further improve by introducing this second level of (repeated) textual validation in our future work.

Original: Evaluation Test Data
Revised: Evaluation Test Data

Original: For evaluating the quality of our address extraction approach, we built a test set consisting of 64 scanned documents with a resolution of 300 dpi.
Revised: For evaluating the quality of our address extraction approach, we built a test set consisting of 64 scanned documents with a resolution of 300 dpi.

Original: The data set provides a total of 104 pages, of which 67 pages contain a sender and 71 a receiver address.
Revised: The data set provides a total of 104 pages, of which 67 pages contain a sender and 71 a receiver address.

Original: Additionally, 105 addresses of type other are distributed across 74 pages.
Revised: Additionally, 105 addresses of type other are distributed across 74 pages.

Original: There are 8 pages left without any addresses.
Revised: There are 8 pages left without any addresses.

Original: [Table]
Revised: [Table]

Original: Results and Discussion
Revised: Results and Discussion

Original: In our experiments, we consider an address as correctly detected and classified (TP) if the predicted region contains at least the core components of an address (postal code and city, and additionally street and house number if present in the ground truth) and is assigned the correct class label ( sender , receiver , other ).
Revised: In our experiments, we consider an address as correctly detected and classified (TP) if the predicted region contains at least the core components of an address (postal code and city, and additionally street and house number if present in the ground truth) and is assigned the correct class label ( sender , receiver , other ).

Original: Regions that were predicted as addresses and contain no address information are defined as false positives (FP).
Revised: Regions that were predicted as addresses and contain no address information are defined as false positives (FP).

Original: Existing address regions in the test set that were not recognized or got the wrong address type assigned are false negatives (FN).
Revised: Existing address regions in the test set that were not recognized or got the wrong address type assigned are false negatives (FN).

Original: The performance results on the whole test set are listed in Table 3.
Revised: The performance results on the whole test set are listed in Table 3.

Original: Note that we leave out the indication of true negatives (TN), since we would have to consider all other non-address text regions that were correctly not labeled as addresses as well.
Revised: Note that we leave out the indication of true negatives (TN), since we would have to consider all other non-address text regions that were correctly not labeled as addresses as well.

Original: The low occurrence of FP indicates that our hybrid approach is able to predict address regions with a high degree of reliability, which is due to our reasoning-based textual validation mechanism, whose result is directly used as secondary input to the visual deep learning model.
Revised: The low occurrence of FP indicates that our hybrid approach is able to predict address regions with a high degree of reliability, which is due to our reasoning-based textual validation mechanism, whose result is directly used as secondary input to the visual deep learning model.

Original: The high amount of false negatives shows that we are missing a lot of addresses.
Revised: The high amount of false negatives shows that we are missing a lot of addresses.

Original: In many cases, we were able to trace back the reason for this to insufficient quality of the OCR results, which leads to incorrect address validation predictions by the reasoning-based component.
Revised: In many cases, we were able to trace back the reason for this to insufficient quality of the OCR results, which leads to incorrect address validation predictions by the reasoning-based component.

Original: This is due to the strict address formatting rules enforced by the module.
Revised: This is due to the strict address formatting rules enforced by the module.

Original: Hence, if for example single characters in either the postal code, city or street name are incorrectly extracted by the OCR process, the match to our reference list will fail with high probability.
Revised: Hence, if for example single characters in either the postal code, city or street name are incorrectly extracted by the OCR process, the match to our reference list will fail with high probability.

Original: In future works, we aim to tackle this problem by improving the OCR component and allowing more fuzzy matching in the reasoning and rule-based parts of the pipeline.
Revised: In future works, we aim to tackle this problem by improving the OCR component and allowing more fuzzy matching in the reasoning and rule-based parts of the pipeline.

Original: Conclusion and Future Work
Revised: Conclusion and Future Work

Original: Our experiments have shown that combining the strengths of data-driven and reasoning-based approaches for the use case of address extraction (in form of domain knowledge about structure and contents of country-specific addresses) can improve stability and reliability of results.
Revised: Our experiments have shown that combining the strengths of data-driven and reasoning-based approaches for the use case of address extraction (in form of domain knowledge about structure and contents of country-specific addresses) can improve stability and reliability of results.

Original: For our future work, we plan to optimize each step of the pipeline.
Revised: For our future work, we plan to optimize each step of the pipeline.

Original: For the deep learning models, this means acquiring more training and fine tuning on bigger data sets.
Revised: For the deep learning models, this means acquiring more training and fine tuning on bigger data sets.

Original: For the reasoning-based parts, this implies to soften the strictness of rule validation by allowing a small amount of fuzziness.
Revised: For the reasoning-based parts, this implies to soften the strictness of rule validation by allowing a small amount of fuzziness.

Original: Additionally, we aim to extend our approach to other tasks of information extraction, in particular intelligent analysis of complex or nested table structures.
Revised: Additionally, we aim to extend our approach to other tasks of information extraction, in particular intelligent analysis of complex or nested table structures.

Revised: Finally, we are planning to perform bigger automated evaluations on the whole pipeline results as well as on the single outcomes of each pipeline component for measuring progress and success of our approach.