F orm N et V 2: Multimodal Graph Contrastive Learning for Form Document Information Extraction (2023)

Abstract: The recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. However, existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. The graph contrastive objective maximizes the agreement of multimodal representations, providing a natural interplay for all modalities without special customization. In addition, we extract image features within the bounding box that joins a pair of tokens connected by a graph edge, capturing more targeted visual cues without loading a sophisticated and separately pre-trained image embedder. FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size.

## introduction automated information extraction is essential for many practical applications, with form-like documents posing unique challenges compared to article-like documents, which have led to an abundance of recent research in the area. in particular, form-like documents often have complex layouts that contain structured objects like tables, columns, and fillable regions. layout-aware language modeling has been critical for many successes @xcite @xcite . to further boost the performance, many recent approaches adopt multiple modalities @xcite @xcite . specifically, the image modality adds more structural information and visual cues to the existing layout and text modalities. they therefore extend the masked language modeling (mlm) from text to masked image modeling (mim) for image and textimage alignment (tia) for cross-modal learning. the alignment objective may also help to prime the layout modality, though it does not directly involve text layouts or document structures. in this work, we propose formnetv2, a multimodal transformer model for form information extraction. unlike existing works -which may use the whole image as one representation @xcite , or image patches @xcite , or image features of token bounding boxes @xcite ) -we propose using image features extracted from the region bounded by a pair of tokens connected in the constructed graph. this allows us to capture a richer and more targeted visual component of the intra-and inter-entity information. furthermore, instead of using multiple self-supervised objectives for each individual modality, we introduce graph contrastive learning @xcite @xcite to learn multimodal embeddings jointly. these two additions to formnetv1 @xcite enable the graph convolutions to produce better super-tokens, resulting in both improved performance and a smaller model size. in experiments, formnetv2 outperforms its predecessor formnetv1 as well as the existing multimodal approaches on four standard benchmarks. in particular, compared with formnetv1, form-netv2 outperforms it by a large margin on funsd (86.35 v.s. 84.69) and @xcite ); compared with docformer @xcite , formnetv2 outperforms it on funsd and cord with nearly 2.5x less number of parameters. ## related work early works on form document information extraction are based on rule-based models or learningbased models with handcrafted features @xcite @xcite @xcite . later on, various deep neural models have been proposed, including methods based on recurrent nets @xcite , convolutional nets @xcite @xcite , and transformers @xcite @xcite . recently, in addition to the text, researchers have explored the layout attribute in form document modeling, such as the ocr word reading order @xcite ), text coordinates @xcite @xcite @xcite , layout grids @xcite , and layout graphs @xcite . the image attribute also provides essential visual cues such as fonts, colors, and sizes. other visual signals can be useful as well, including logos and separating lines from form tables. @xcite uses faster r-cnn @xcite to extract token image features; @xcite uses resnet50 @xcite to extract full document image features; @xcite use vit @xcite with fpn @xcite to extract non-overlapping patch image features. these sophisticated image embedders require a separate pre-training step using external image datasets (e.g. imagenet @xcite or publaynet @xcite ), and sometimes depend upon a visual codebook pre-trained by a discrete variational autoencoder (dvae). when multiple modalities come into play, different supervised or self-supervised multimodal pre-training techniques have been proposed. they include mask prediction, reconstruction, and matching for one or more modalities @xcite @xcite @xcite @xcite . next-word prediction @xcite or length prediction @xcite have been studied to bridge text and image modalities. direct and relative position predictions @xcite @xcite @xcite have been proposed to explore the underlying lay-out semantics of documents. nevertheless, these pre-training objectives require strong domain expertise, specialized designs, and multi-task tuning between involved modalities. in this work, our proposed graph contrastive learning performs multimodal pre-training in a centralized design, unifying the interplay between all involved modalities without the need for prior domain knowledge. ## formnetv2 we briefly review the backbone architecture form-netv1 @xcite in sec 3.1, introduce the multimodal input design in sec 3.2, and detail the multimodal graph contrastive learning in sec 3.3. ## conclusion formnetv2 augments an existing strong form-netv1 backbone with image features bounded by pairs of neighboring tokens and the graph contrastive objective that learns to differentiate between the multimodal token representations of two corrupted versions of an input graph. the centralized design sheds new light to the understanding of multimodal form understanding.