F orm N et: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction (2022)

Abstract: Sequence modeling has demonstrated state-of-the-art performance on natural language and document understanding tasks. However, it is challenging to correctly serialize tokens in form-like documents in practice due to their variety of layout patterns. We propose FormNet, a structure-aware sequence model to mitigate the suboptimal serialization of forms. First, we design Rich Attention that leverages the spatial relationship between tokens in a form for more precise attention score calculation. Second, we construct Super-Tokens for each word by embedding representations from their neighboring tokens through graph convolutions. FormNet therefore explicitly recovers local syntactic information that may have been lost during serialization. In experiments, FormNet outperforms existing methods with a more compact model size and less pre-training data, establishing new state-of-the-art performance on CORD, FUNSD and Payment benchmarks.

## introduction form-like document understanding is a surging research topic because of its practical applications in automating the process of extracting and organizing valuable text data sources such as marketing documents, advertisements and receipts. typical documents are represented using natural languages; understanding articles or web content @xcite @xcite has been studied extensively. however, form-like documents often have more complex layouts that contain structured objects, such as tables and columns. therefore, form documents have unique challenges compared to natural language documents stemming from their structural characteristics, and have been largely under-explored. in this work, we study critical information extraction from form documents, which is the fundamental subtask of form document understanding. following the success of sequence modeling in natural language understanding (nlu), a natural approach to tackle this problem is to first serialize the form documents and then apply state-of-theart sequence models to them. for example, @xcite use seq2seq @xcite with @xcite use transformers @xcite . however, interwoven columns, tables, and text blocks make serialization difficult, substantially limiting the performance of a strict serialization approach. to model the structural information present in documents, @xcite ; @xcite ; @xcite treat the documents as 2d image inputs and directly apply convolutional networks on them to preserve the spatial context during learning and inference. however, the performance is limited by the resolution of the 2d input grids. another approach is a two-step pipeline @xcite that leverages computer vision algorithms to first infer the layout structures of forms and then perform sequence information extraction. the methods are mostly demonstrated on plain text articles or documents @xcite but not on highly entangled form documents @xcite . in this work, we propose formnet, a structureaware sequence model to mitigate the suboptimal @xcite ) -the core of our system -is able to model long inputs by limiting attention to a local radius of serialized tokens. the proposed rich attention uses the spatial relationship between tokens to penalize unlikely attention edges. in this example, for the word 'white', rich attention increases the relative weight for spatially nearby tokens such as 'ks', '-' and 'tip', while decreasing it for others, resulting in spatially aware attention scores. (c) even though they belong to the same entity, kool and masked may not be visible to each other from within the local radius of etc after the left-to-right, top-to-bottom serialization step, which breaks the text group on the left into multiple text segments. our proposed super-tokens are generated by executing graph convolutional networks directly on 2d tokens before serialization. the edges of the graph leverage the inductive bias of the β-skeleton graph, allowing information propagation w.r.t. the structural layout of documents before text serialization introduces noise. note that the β-skeleton graph connects kool and masked in this example. in our experiments, formnet outperforms existing methods while using (1) smaller model sizes and (2) less pre-training data while (3) avoiding the need for vision features. in particular, formnet achieves new best f1 scores on cord and funsd (97.28% and 84.69%, respectively) while using a 64% sized model and 7.1x less pre-training data than the most recent docformer @xcite . ## related work document information extraction was first studied in handcrafted rule-based models @xcite @xcite . later @xcite ; @xcite ; @xcite ; @xcite ; @xcite use learningbased approaches with engineered features. these methods encode low-level raw pixels @xcite or assume form templates are known a priori @xcite , which limits their generalization to documents with specific layout structures. in addition to models with limited or no learning capabilities, neural models have also been studied. @xcite ; @xcite use an rnn for document information extraction, while @xcite ; @xcite ; denk and reisswig (2019) investigate convolutional models. there are also self-attention networks (transformers) for document information extraction, motivated by their success in conventional nlu tasks. majumder et al. (2020) extend bert to representation learning for form documents. @xcite modified the attention mechanism in roberta @xcite . @xcite ; @xcite ; @xcite are multimodal models that combine bert-like architectures @xcite and advanced computer vision models to extract visual content in images. similarly, spade @xcite is a graph decoder built upon the transformer models for better structure prediction compared to simple bio tagging. the proposed formnet is orthogonal to multimodal transformers and spade. compared with multimodal models, form-net focuses on modeling relations between words through graph convolutional learning as well as rich attention without using any visual modality; compared with spade, formnet uses a graph encoder to encode inductive biases in form input. a straightforward extension would be to combine formnet with either layout transformers or spade for capturing visual cues or better decoding, which we leave for future work. graph learning with sequence models has also been studied. on top of the encoded information through graph learning, @xcite al. (2019 ); liu et al. (2019a); yu et al. (2020) use rnn and crf while we study rich attention in formnet for decoding. peng et al. (2017); song et al. (2018) do not study document information extraction. ## formnet for information extraction problem formulation. given serialized foot_0 words of a form document, we formulate the problem as sequential tagging for tokenized words by predicting the corresponding key entity classes for each token. specifically, we use the "bioes" scheme -{begin, inside, outside, end, single} @xcite to mark the spans of entities in token sequences and then apply the viterbi algorithm. proposed approach. by treating the problem as a sequential tagging task after serialization, we can adopt any sequence model. to handle potentially long documents (e.g. multi-page documents), we select the long-sequence transformer extension etc @xcite as our backbone 2 . in practice, it is common to see an entity sequence cross multiple spans of a form document, demonstrating the difficulty of recovering from serialization errors. as illustrated in figure 2 (b), 9.1 is next to tip-, while ping masked belong to the same entity as tip-are distant from it under the imperfect serialization. our remedy is to encode the original 2d structural patterns of forms in addition to positions within the serialized sentences. we propose two novel components to enhance etc: rich attention and super-tokens (figure 2 ). rich attention captures not only the semantic relationship but also the spatial distance between every pair of tokens in etc's attention component. super-tokens are constructed by graph convolutional networks before being fed into etc. they model local relationships between pairs of tokens that might not be visible to each other or correctly inferred in an etc model after suboptimal serialization. figure 3 shows the overall system pipeline. we discuss the details of etc in sec. 3.1, rich attention in sec. 3.2, and super-token in sec. 3.3. ## evaluation we evaluate how the two proposed structural encoding components, rich attention and super-tokens, impact the overall performance of form-like document key information extraction. we perform extensive experiments on three standard benchmarks foot_7 ## conclusion we present a novel model architecture for key entity extraction for forms, formnet. we show that the proposed rich attention and super-token components help the etc transformer to excel at form understanding in spite of noisy serialization, as evidenced quantitatively by its state-of-the-art performance on three benchmarks and qualitatively by its more sensible attention patterns. in the future, we would like to explore multi-modality input such as images.