A li GAT r: Graph-based layout generation for form understanding (2024)

Abstract: Forms constitute a large portion of layout-rich documents that convey information through key-value pairs. Form understanding involves two main tasks, namely, the identification of keys and values (a.k.a Key Information Extraction or KIE) and the association of keys to corresponding values (a.k.a. Relation Extraction or RE). State of the art models for form understanding often rely on training paradigms that yield poorly calibrated output probabilities and low performance on RE. In this paper, we present AliGATr, a graph-based model that uses a generative objective to represent complex grid-like layouts that are often found in forms. Using a grid-based graph topology, our model learns to generate the layout of each page token by token in a data efficient manner. Despite using 30% fewer parameters than the smallest SotA, AliGATr performs on par with or better than SotA models on the KIE and RE tasks against four datasets. We also show that AliGATr’s output probabilities are better calibrated and do not exhibit the over-confident distributions of other SotA models.

## introduction visually complex forms pose a multimodal challenge to the task of document-grounded reasoning. state of the art approaches have achieved increasingly advanced performance, but their utility in downstream applications remains restricted by a few key challenges we face broadly as a research area, as described in the recent position paper by @xcite . in this paper, we introduce aligatr, a new form understanding model that addresses these challenges in order to improve the practical impact of research on document-grounded reasoning for visually complex forms, with potential impact on the broader fields of information extraction and multimodal document understanding. the two common tasks in form understanding that illustrate the challenges we address include key information extraction (kie) and relation extraction (re), both of which rely on identifying field names and field values, understanding tabular structures, and distinguishing between headings, main content, and other components, all of which require joint reasoning over the spatial and textual signal on each page. with a few exceptions, most sota researches focus on the task of kie, disregarding re, whereas in most applications kie and re need to be paired in order to identify semantically valid key-value pairs from forms. without re, key structural information about the document will not be captured @xcite , and openended key-value extraction will be difficult @xcite . despite this, re remains underexplored in the form understanding literature @xcite @xcite , posing major challenges to downstream applications @xcite . furthermore, most models require extensive pretraining data and infrastructure to perform at the sota level. as an example, the most popular pretraining dataset is the iit-cdip dataset @xcite , composed of 11 million images. lastly, the trade-off between grounding (i.e. providing bounding boxes for each output token such that it can be traced back to the input) and calibration (i.e. producing distributionally robust probabilities) is difficult to balance. small, efficient, robust, and well-calibrated models remain difficult to obtain for users with limited access to large-scale pre-training data or compute. in this paper, we introduce aligatr, a new form understanding model that addresses the above challenges by combining a graph-based representation and a layout-generation objective. by focusing on the generation of layout (as opposed to the joint generation of text and layout), our approach leads to a model that is more compact and converges using a smaller pre-training dataset. our proposed graph representation, which we name alignet, enables the model to cover both kie and re tasks, leading to sota performance on the former and exceeding sota performance on the latter. even though aligatr has a generative objective, it samples its output from input tokens, leading to logits that are well-grounded and well-calibrated. concretely, our study offers the following contributions to the literature on visually rich form understanding (vrfu): to the best of our knowledge, aligatr is the first graph-based model to use a generative objective for form understanding. ## related work research in visually rich document understanding has explored models in two architectural paradigms, namely transformer-based models and graph-based models. ## methodology in this section, we describe our proposed graph representation for documents (alignet), as well as our proposed model architecture (aligatr). figure 2 shows the overall flow of pre-training and finetuning steps. ## experiments in this section we describe the datasets and baselines used in our experiments. other experimental settings are described in appendix a. ## conclusion and future work in this paper, we presented aligatr, a layout generation technique for form understanding that is competitive with sota on key information extraction and relation extraction tasks, using 30% fewer parameters and 11x fewer training examples. we showed how, despite using the spatial and textual modalities alone, and relying on subsequence generation, the model produces better-calibrated probabilities. in future studies, we hope to investigate aligatr's effectiveness in other adjacent tasks that lend themselves to graph-based representations, such as document classification, page segmentation, and structure extraction.