Capturing Logical Structure of Visually Structured Documents with Multimodal Transition Parser (2021)

Abstract: While many NLP pipelines assume raw, clean texts, many texts we encounter in the wild, including a vast majority of legal documents, are not so clean, with many of them being visually structured documents (VSDs) such as PDFs. Conventional preprocessing tools for VSDs mainly focused on word segmentation and coarse layout analysis, whereas fine-grained logical structure analysis (such as identifying paragraph boundaries and their hierarchies) of VSDs is underexplored. To that end, we proposed to formulate the task as prediction of “transition labels” between text fragments that maps the fragments to a tree, and developed a feature-based machine learning system that fuses visual, textual and semantic cues. Our system is easily customizable to different types of VSDs and it significantly outperformed baselines in identifying different structures in VSDs. For example, our system obtained a paragraph boundary detection F1 score of 0.953 which is significantly better than a popular PDF-to-text tool with an F1 score of 0.739.

## introduction despite recent motivation to utilize nlp for wider range of real world applications, most nlp papers, tasks and pipelines assume raw, clean texts. however, many texts we encounter in the wild, including a vast majority of legal documents (e.g., contracts and legal codes), are not so clean, with many of them being visually structured documents (vsds) such as pdfs. for example, of 7.3 million text documents found in panama papers (which arguably approximates the distribution of data one would see in the wild), approximately 30% were pdfs foot_0 . good preprocessing of vsds is crucial in order to apply recent advances in nlp to real world applications. thus far, the most micro and macro extremes of vsd preprocessing have been extensively studied, such as word segmentation and layout analysis (detecting figures, body texts, etc.; @xcite , respectively. while these two lines of studies allow extracting a sequence of words in the body of a document, neither of them accounts for local, logical structures such as paragraph boundaries and their hierarchies. these structures convey important information in any domain, but they are particulary important in the legal domain. for example, figure 1 (1) shows raw text extracted from a non-disclosure agreement (nda) in pdf format. an information extraction (ie) system must be aware of the hierarchical structure to successfully identify target information (e.g., extracting "definition of confidential information" requires understanding of hierarchy as in figure 1 (2)). furthermore, we must utilize the logical structures to remove debris that has slipped through layout analysis ("page 1 of 5" in this case) and other structural artifacts (such as semicolons and section numbers) for a generic nlp pipeline to work properly. yet, such logical structure analysis is difficult. even the best pdf-to-text tool with a word-related error rate as low as 1.0% suffers from 17.0% newline detection error @xcite that is arguably the easiest form of logical structure analysis. the goal of this study is to develop a fine-grained logical structure analysis system for vsds. we propose a transition parser-like formulation of logical structure analysis, where we predict a transition label between each consecutive pair of text fragments (e.g., two fragments are in a same paragraph, or in different paragraphs of different hierarchies). based on such formulation, we developed a feature-based machine learning system that fuses multimodal cues: visual (such as indentation and line spacing), textual (such as section numreceiving party or any of its affiliates, before, on or after the effective date. this includes: page 1 of 5 a) the terms of this agreement; 2.2. confidential information does not include information which: a) is independently developed by receiving party; or techniques, know-how, ....., trade secrets, whether in tangible or intangible form. b) all confidential or proprietary .... business, customers, plans, pricing, operations, ..... 2. confidential information⏎ 2.1. confidential information means all confidential information relating to the purpose which the disclosing party or any of its affiliates, discloses or makes available, to the receiving party or any of its affiliates, before, on or after the effective date. this includes a) the terms of this agreement, ; page 1 of 5 b) all confidential or proprietary information relating to: the business, customers, plans, pricing, operations, techniques, know-how, technical information, design, trade secrets, whether in tangible or intangible form.⏎ 2.2. confidential information does not include information which: a) is or subsequently becomes public knowledge or publicly available through no fault of the receiving party; or … removing debris based on logical structure analysis without logical structure analysis bering and punctuation), and semantic (such as language model coherence) cues. finally, we show that our system is easily customizable to different types of vsds and that it significantly outperforms baselines in identifying different structures in vsds. for example, our system obtained a paragraph boundary detection f1 score of 0.953 that is significantly better than pdfminer 2 , a popular pdf-to-text tool, with an f1 score of 0.739. we open-sourced our system and dataset 3 . ## problem setting and our formulation in this study, we concentrate on logical structure analysis of vsds. the input is a sequence of text blocks (figure 1 (3)) that can be obtained by utilizing existing coarse layout analysis and word-level preprocessing tools. we aim to extract paragraphs and identify their relationships. this is equivalent to creating a tree with each block as a node (figure 1(4) ). we propose to formulate this tree generation problem as identification of a transition label between each consecutive pair of blocks (figure 1 while down is well-defined (because we assume a tree), up can be ambiguous as to how many levels we should raise. to that end, we also introduce a pointer to each up block, which points at b j whose level b i belongs to (ptr @xmath0, where @xmath1; figure 1(8) ). ## dataset in this study, we target four types of vsds in different file formats and languages: contract pdf en english ndas in pdf format. law pdf en english executive orders from local authorities. contract txt en english ndas in visually structured plain text format. contract pdf ja japanese ndas in pdf format. examples of each type of vsds are shown in figure 2. for pdfs, we downloaded pdfs from google.com search result. since our focus is not on coarse layout analysis or word-level preprocessing, we selected single column documents 1. provide an opportunity for data recipient to cure the violation within 30 days, and terminate the agreement if data recipient does not cure or end the violation within the time specified by dohmh; (a) contract pdf en executive order in response to covid-19 (covid-19 executive order no. 8) whereas, i, jb pritzker, governor of illinois, declared all counties in the state of illinois as a disaster area on march 9, 2020 (gubernatorial disaster proclamation) in response to the outbreak of coronavirus disease 2019 (covid-19); and, whereas, in a short period of time, covid-19 has rapidly spread throughout illinois, necessitating updated and more stringent guidance from federal, state, and local public health officials; and, whereas, for the preservation of public health and safety throughout the entire state of illinois, and to ensure that our healthcare delivery system is capable of serving those who are sick, i find it necessary to take additional measures consistent with public health guidance to slow and stop the spread of covid-19; whereas, covid-19 has resulted in significant economic impact, including loss of income and wages, that threaten to undermine housing security and stability; whereas, the enforcement of eviction orders for residential premises is contrary to the interest of preserving public health and ensuring that individuals remain in their homes during this public health emergency; therefore, by the powers vested in me as the governor of the state of illinois, and pursuant to sections 7(1), 7(2), 7(8), 7(10), and 7(12) of the illinois emergency management agency act, 20 ilcs 3305, and consistent with the powers in public health laws, i hereby order the following, effective march 21, 2020 at 5:00 pm and for the remainder of the duration of the gubernatorial disaster proclamation, which currently extends through april 7, 2020: ## implementation and customization in this section, we briefly describe the implementation of our system that allows easy customization to different types of vsds. our system employs modular and customizable design and is implemented in python. a user may implement a new feature extractor simply by writing a new feature extractor class where each feature is implemented as its class function (figure 4 ). for example, @single_input_feature([1]) denotes that the subsequent function should be applied to the second block of each context (thus corresponding to feature v6). likewise, the features for pointer identification can be implemented by marking a function with @pointer_feature(), which takes a candidate block b j (tb1), a target block b i (tb2), the block next to the target block b i+1 (tb3) and b head(j) (head_tb) as an input. a feature extractor object is instantiated for each document where all feature functions are automatically aggregated to produce the feature vector. a new feature extractor can inherit from an existing feature extractor (e.g., feature extractors for contract pdf en and contract pdf ja both inherit from a base pdf feature extractor), which makes it easy to reuse implementations. ## experiments 6.1 evaluation metrics while we do report transition prediction accuracy, it is not a true task metric since it is rooted on our formulation of the task. looking back at our initial motivation in section 1, we introduce two sets of evaluation metrics. the first set of metrics is rooted on ie perspective. for ie, it is important to identify ancestordescendant and sibling relationships because it allows, for example, identifying a subject (in an ancestral block) and its objects (a decendant block and its siblings). thus, we evaluate f1 scores for identifying pairs of blocks in (1) same paragraph, (2) sibling, and (3) ancestor-descendant relationships, respectively (figure 5 ). note that we do not include cousin blocks in the sibling relationship, because it is not clear whether cousin blocks have any meaningful information in the context of ie. we use the second set of metrics to evaluate a system's efficacy as a preprocessing tool for more general nlp pipelines. we evaluate paragraph boundary identification metrics since paragraph boundaries can be used to determine appropriate chunks of text to be fed into the nlp pipelines. we also report accuracy for removing debris with omitted. we used five-folds cross validation for the evaluation. ## related works as discussed in section 1, previous works mainly focused on word segmentation and layout analysis, whereas fine-grained logical structure analysis of vsds is less addressed. nevertheless, there exist some studies that focus on similar goals. @xcite and @xcite have tried to deal with logical structure analysis by identifying specific structures in vsds such as subheadings. however, these studies are too coarse-grained and cannot handle paragraph-level logical structure, thus they are unable to satisfy the need we have discussed in section 1. finsbd-3 shared task @xcite is more fine-grained than those works and incorporates extraction of list items. however, its main focus is not on analysis of logical structures; it has only four static levels for list hierarchies and does not consider hierarchies in non-list paragraphs. @xcite proposed a rule-based system that purely relies on numberings. we compared our system against it in section 6 and showed that our system, which also incorporates textual and semantic cues, is superior to their method. @xcite proposed a paragraph boundary detection method for plain texts that purely relies on textual and semantic cues. while their method is not intended for vsds, some of their ideas could be incorporated to our work as additional features. we leave use of more advanced semantic cues for a future work. while the goal is different, our textual features have some similarity to those used in sentence boundary detection @xcite . since our goal is to predict structures as well as boundaries, we employ richer textual and visual features that they do not utilize. layoutlm @xcite incorporates multimodal self-supervised learning to utilize deep learning for form understanding. while it may alleviate the need for a large training dataset, it is not trivial to adopt the same method for logical structure analysis as text blocks would not fit onto the layoutlm's context. furthermore, it is easier to diagnose and to improve our system as it utilizes a combination of hand-crafted features, while deep learning systems tend to be completely black box. ## conclusions we proposed a transition parser-like formulation of the logical structure analysis of vsds and developed a feature-based machine learning system that fuses visual, textual and semantic cues. our system significantly outperformed baselines and an existing open-source software on different types of vsds. the experiment revealed that incorporating both the visual and textual cues is crucial in successfully conducting logical structure analysis of vsds. as a future work, we will incorporate typographic and more advanced semantic cues.