D oc S truct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding (2020)

Abstract: Form understanding depends on both textual contents and organizational structure. Although modern OCR performs well, it is still challenging to realize general form understanding because forms are commonly used and of various formats. The table detection and handcrafted features in previous works cannot apply to all forms because of their requirements on formats. Therefore, we concentrate on the most elementary components, the key-value pairs, and adopt multimodal methods to extract features. We consider the form structure as a tree-like or graph-like hierarchy of text fragments. The parent-child relation corresponds to the key-value pairs in forms. We utilize the state-of-the-art models and design targeted extraction modules to extract multimodal features from semantic contents, layout information, and visual images. A hybrid fusion method of concatenation and feature shifting is designed to fuse the heterogeneous features and provide an informative joint representation. We adopt an asymmetric algorithm and negative sampling in our model as well. We validate our method on two benchmarks, MedForm and FUNSD, and extensive experiments demonstrate the effectiveness of our method.

## introduction forms are a ubiquitous document format. numerous forms are used in finance, insurance and medical industry every day. although forms vary a lot, we consider it as a collection of key-value pairs and all these pairs establish a hierarchical structure within the page. our work in this paper focuses on utilizing multimodal information to extract the hierarchy from the forms. equipped with the hierarchy, it is oversimplified to further analyze the general forms and extract the structural data. division: division name: detroit north division name: detroit east division: division name: division name: detroit north detroit east (a) (b) check items result heart rate 72 heart rhythm regular [medical examination] (c) (d) [medical examination] check items result heart rhythm regular heart rate 72 for the computers to read the textual contents of form pages. the contents can be divided into several individual textual fragments. however, it is not enough to form-understanding tasks. the information is expressed not only through the textual data in each section, but also through the way in which the sections are organized. some of the fragments serve as headers, topics or questions of their counterparts. we consider the relation as key-value pairs in a hierarchy. figure 1 are two examples foot_0 . therefore, after preliminary processing with ocr, we need to extract the latent structure in a form page to convert the textual data into structured data. some related works build their models on the handcrafted features @xcite @xcite . they propose heuristic methods and use top-down or bottom-up techniques to build their model. however, the results only reveal coarse structure such as layouts or bounding boxes, which is not enough for actual usage. others propose table detection techniques @xcite . however, tables are a subset of forms. table detection doesn't work when there is no table lines or cells in the given form page. unlike these previous works, our method consider the structure as a hierarchy where parent-child relation corresponds to the key-value pairs. the key-value pairs are the most elementary components and independent of any formats. this ensures a wide range of application of our method and should be the promising direction for general form understanding. to comprehensively acquire the informative representation for each fragment and catch the reliable signals for the hierarchical structure, we leverage multimodal information from semantic, layout and visual aspects. we carefully design targeted extraction module for each modality. following many previous studies, we adopt the pretrained language model, e.g. bert @xcite , roberta @xcite , to extract semantic features. the layout and visual information are also useful. for example, nearer fragments should be more likely to be related and fragments with bold faces are more likely to be the title. therefore, we use multilayer perceptron and character detection algorithms @xcite to extract layout and visual features. to fuse multimodal features, we are enlightened by @xcite ; @xcite and propose a hybrid fusion method of concatenation and feature shifting. the features differ in meaning and dimension. we utilize the most informative features, semantic and layout features, through concatenation. then we take vision features as shifting feature to refine fused features. finally, we design an asymmetric relation prediction module and negative sampling to finish the whole pipeline. we validate our method, docstruct, on two benchmarks, medform and funsd @xcite . the first one is built by us and composed of medical examination reports, and the second is composed of various real, fully annotated, scanned forms. we summarize our contribution as follows: ## methodology in this section, we will first describe the preliminary processing step and introduce an overview of our method. then we propose the docstruct model which extracts and fuses multimodal features and predicts the hierarchical relation between text fragments. finally, the negative sampling training method is also introduced in this section. ## experiment in this section, we conduct experiments on two benchmarks, medform and funsd @xcite foot_1 , to validate the effectiveness of our proposed model for building the latent hierarchy in forms. we design two tasks, reconstruction and detection. the metric explanation is in appendix. reconstruction: given the labeled hierarchical structure in a document, we predict the superior counterpart for each text fragment so as to rebuild the hierarchy. the mean average precision (map) and mean rank (mrank) are used as metrics. detection: to test the detection ability of our model, we choose the counterpart with the highest probability as the prediction result and calculate the hit@1, hit@2 and hit@5 as metrics. ## conclusion in this paper, we proposed a multimodal method to extract key-value pairs and build the hierarchy in forms to improve the general form understanding. we leveraged advanced models, e.g. bert, resnet, lstm, to acquire features from multiple aspects: semantic, layout and vision. for the first time, heterogeneous features are combined to extract the hierarchical structure in forms. and the proposed hybrid fusion method of concatenation and feature shifting effectively obtains the joint feature and eliminates the interference. we also adopted negative sampling technique to train our model. furthermore, extensive experiments demonstrate the advantages of our method to build the form hierarchy. in the future, we will apply our method to other challenging benchmarks and strive to combine the idea of pretraining with our method.