F orm2 S eq : A Framework for Higher-Order Form Structure Extraction (2020)

Abstract: Document structure extraction has been a widely researched area for decades with recent works performing it as a semantic segmentation task over document images using fully-convolution networks. Such methods are limited by image resolution due to which they fail to disambiguate structures in dense regions which appear commonly in forms. To mitigate this, we propose Form2Seq, a novel sequence-to-sequence (Seq2Seq) inspired framework for structure extraction using text, with a specific focus on forms, which leverages relative spatial arrangement of structures. We discuss two tasks; 1) Classification of low-level constituent elements (TextBlock and empty fillable Widget) into ten types such as field captions, list items, and others; 2) Grouping lower-level elements into higher-order constructs, such as Text Fields, ChoiceFields and ChoiceGroups, used as information collection mechanism in forms. To achieve this, we arrange the constituent elements linearly in natural reading order, feed their spatial and textual representations to Seq2Seq framework, which sequentially outputs prediction of each element depending on the final task. We modify Seq2Seq for grouping task and discuss improvements obtained through cascaded end-to-end training of two tasks versus training in isolation. Experimental results show the effectiveness of our text-based approach achieving an accuracy of 90% on classification task and an F1 of 75.82, 86.01, 61.63 on groups discussed above respectively, outperforming segmentation baselines. Further we show our framework achieves state of the results for table structure recognition on ICDAR 2013 dataset.

## introduction various works @xcite @xcite have studied semantic structure extraction for documents. structure extraction is necessary for digitizing documents to make them re-flowable and index-able, which is useful in web-based services @xcite @xcite . in this work, we look at a complex class of documents i.e., forms that are used to capture user data by organizations across various domains such as government services, finance, administration, and healthcare. such industries that have been using paper or pdf forms would want to convert them into an appropriate digitized version @xcite (such as an html). once these forms are made re-flowable, they can be made available across devices with different form factors @xcite . this facilitates providing better form filling experiences and increases the ease of doing business since their users can interact with forms more conveniently and enables other capabilities like improved handling of filled data, applying validation checks on data filled in fields, consistent form design control foot_0 . to enable dynamic rendering of a form while re-flowing it, we need to extract its structure at multiple levels of hierarchy. we define textblock to be a logical block of self contained text. widgets are spaces provided to fill information. some low level elementary structures such as text and widgets can be extracted using auto-tagging capabilities of tools like acrobat from the form pdf. however, such pdfs do not contain data about higher-order structures such as text fields, choicegroups etc. document structure extraction has been studied extensively with recent works employing deep learning based fully convolution neural networks @xcite @xcite that perform semantic segmentation @xcite @xcite on document image. such methods perform well at extracting coarser structures but fail to extract closely spaced structures in form images (as discussed in the experiments section). with increase in image resolution, number of activations(forward pass) and gradients(backward pass) increase at each network layer which requires more gpu memory during training. since gpu memory is limited, they downscale the original image at the input layer which makes it difficult to disambiguate closely spaced structures, especially in dense regions(occurring commonly in forms) which leads to merging. figure 1 shows different types of textblocks, widgets and higher order groups. given text blocks and widgets as input, our form2seq framework classifies them between different type categories. we hypothesize that type classification of lower level elements can provide useful cues for extracting higher order constructs which are comprised of such smaller elements. we establish our hypothesis for the task of extracting choicegroups, text fields and choice fields. a text field is composed of textblock(textual caption) and associated widgets, as shown in figure 1 . a choice group is a collection of boolean fields called choice fields with an optional title text (choice group title) that describes instructions regarding filling it. we study fillable constructs as they are intrinsic and unique to forms and contain diverse elementary structures. the spatial arrangement of lower level elements with respect to other elements in a form are correlated according to the type of construct. for in-stance, a list item usually follows a bullet in the reading order; field widgets are located near the field caption. similarly, elements that are part of same higher-order group tend to be arranged in a spatially co-located manner. to leverage this in our form2seq framework, we perform a bottom up approach where we first classify lower level elements into different types. we arrange these elements in natural reading order to obtain a linear sequence. this sequence is fed to seq2seq @xcite where each element's text and spatial representation is passed through a bilstm. the output of bilstm for each element is sequentially given as input to an lstm @xcite based decoder which is trained to predict the category type. for grouping task, we modify the framework to predict id of the group each lower level element is part of. here the model is trained to predict same group id for elements that are part of same group. our contributions can be listed as: ## related work earlier works for document layout analysis have mostly been rule based relying on hand crafted features for extracting coarser structures such as graphics and text paragraphs @xcite . approaches like connected components and others, were also used for extracting text areas @xcite and physical layouts @xcite . these approaches can be classified into top-down @xcite or bottom-up @xcite . the bottom-up methods focus on extracting text-lines and aggregating them into paragraphs. top-down approaches detect layout by subdividing the page into blocks and columns. with the advancement in deep learning, recent approaches have mostly been fully convolution neural network (fcn) based that eliminate need of designing complex heuristics @xcite @xcite . fcns were successfully trained for semantic segmentation @xcite which has now become a common technique for page segmentation. the high level feature representations make fcn effective for pixel-wise prediction. fcn has been used to locate/recognize handwritten annotations, particularly in historical documents @xcite . @xcite proposed a model that jointly learns handwritten text detection and recognition using a region proposal network that detects text start positions and a line follow module which incrementally predicts the text line that should be subsequently used for reading. several methods have addressed regions in documents other than text such as tables, figures etc. initial deep learning work that achieved success in table detection relied on selecting table like regions on basis of loose rules which are subsequently filtered by a cnn @xcite . he et al. proposed multi-scale, multi-task fcn comprising of two branches to detect contours in addition to page segmentation output that included tables. they additionally use crf (conditional random field) to make the segmented output smoother. however, segmentation based methods fail to disambiguate closely spaced structures in form images due to resolution limitations as discussed in experiments section. @xcite the new task of recognising only useful entities in long documents on two new datasets. funsd @xcite is a small-scale dataset for form understanding comprising of 200 annotated forms. in comparison, our forms dataset is much larger having richer set of annotations. for task of figure extraction from scientific documents, @xcite introduced a large scale dataset comprising of 5.5 million document labels. they find bounding boxes for figures in pdf by training overfeat @xcite on image embeddings generated using resnet-101. few works have explored alternate input modalities such as text for other document related tasks. extracting pre-defined and commonly occurring named entities from invoices like documents(using text and box coordinates) has been the main focus for some prior works @xcite @xcite ). text and document layouts have been used for learning bert @xcite like representations through pre-training and then combined with image features for information extraction from documents @xcite . however, our work focuses on extracting a much more generic, diverse, complex, dense, and hierarchical document structure from forms. document classification is a partly related problem that has been studied using cnn-only approaches for document verification @xcite . yang et al. have designed han which hierarchically builds sentence embeddings and then document representation using multi-level attention mechanism. other works explored multi-modal approaches, using mobilenet @xcite and fasttext @xcite to extract visual and text features respectively, which are combined in different ways (such as concatenation) for document classification @xcite . in contrast, we tackle a different task of form layout extraction which requires recognising different structures. yang et al. also proposed a multimodal fcn (mfcn) to segment figures, tables, lists etc. in addition to paragraphs from documents. they concatenate a text embedding map to feature volume. we consider image based semantic segmentation approaches as baselines for the tasks proposed. we compare the performance of our approach with 1) their fcn based method and 2) deeplabv3+ @xcite , which is state of the art deep learning model for semantic segmentation. ## methodology the spatial arrangement of a lower element among its neighbouring elements is dependent on the class of element. for instance, a list item usually follows a bullet in the reading order. similarly, elements that are part of the same higher-order group tend to be arranged in a spatially co-located pattern. to leverage relative spatial arrangement of all elements in a form together, we arrange them according to a natural reading order (left to right and top to bottom arrangement), encode their context aware representations sequentially using text and spatial coordinates and use them for prediction. for each task, the decoder predicts the output for each element sequentially, conditioning it on the outputs of elements before it in the sequence in an auto-regressive manner (just like sentence generation in nlp). for group extraction task, our model assigns a group id to each element conditioning it on ids predicted for previous elements. this is essential to predict correct group id for current element (for instance, consider assigning same group id to elements that are part of same group). let a form be comprising of a list of textblocks (f t ) and list of widgets (f w ). we arrange f @xmath0. ## conclusion we present an nlp based form2seq framework for form document structure extraction. our proposed model uses only lower level elements -textblocks & widgets without using visual modality. we discuss two tasks -element type classification and grouping into larger constructs. we establish improvement in performance through text info and joint training of two tasks. we show that our model performs better compared to current semantic segmentation approaches. further we also perform table structure recognition (grouping texts present in a table into rows and columns) achieving state-of-the-art results. we are also releasing a part of our forms dataset to aid further research in this direction.