3 MVRD : Multimodal Multi-task Multi-teacher Visually-Rich Form Document Understanding (2024)

Abstract: This paper presents a groundbreaking multimodal, multi-task, multi-teacher joint-grained knowledge distillation model for visually-rich form document understanding. The model is designed to leverage insights from both fine-grained and coarse-grained levels by facilitating a nuanced correlation between token and entity representations, addressing the complexities inherent in form documents. Additionally, we introduce new inter-grained and cross-grained loss functions to further refine diverse multi-teacher knowledge distillation transfer process, presenting distribution gaps and a harmonised understanding of form documents. Through a comprehensive evaluation across publicly available form document understanding datasets, our proposed model consistently outperforms existing baselines, showcasing its efficacy in handling the intricate structures and content of visually complex form documents.

## introduction understanding and extracting structural information from visually-rich documents (vrds), such as academic papers @xcite , receipts @xcite , and forms @xcite , holds immense value for natural language processing (nlp) tasks, particularly in information extraction and retrieval. while significant progress has been made in solving various vrd benchmark challenges, including layout analysis and table structure recognition, the task of form document understanding remains notably challenging. this complexity of the form document understanding arises from two main factors: 1) the involvement of two distinct authors in a form and 2) the integration of diverse visual cues. firstly, forms mainly involve two primary authors: form designers and users. form designers create a structured form to collect necessary information as a user interface. unfortunately, the form layouts, designed to collect varied information, often lead to complex logical relationships, causing confusion for form users and heightening the challenges in form document understanding. secondly, diverse authors in forms may encounter a combination of different document natures, such as digital, printed, or handwritten forms. users may submit forms in various formats, introducing noise such as low resolution, uneven scanning, and unclear handwriting. traditional document understanding models do not account for the diverse carriers of document versions and their associated noises, exacerbating challenges in understanding form structures and their components. most vrd understanding models inherently hold implicit multimodal document structure analysis (vision and text understanding) knowledge either at fine-grained @xcite or coarse-grained @xcite levels. the fine-grained only models mainly focus on learning detailed logical layout arrangement, which cannot handle complex relationships of multimodal components, while the coarse-grained models tend to omit significant words or phrases. hence, we introduce a novel joint-grained document understanding approach with multimodal multi-teacher knowledge distillation. it leverages knowledge from various taskbased teachers throughout the training process, intending to create more inclusive and representative multi-and joint-grained document representations. our contributions are summarised as follows: 1) we present a groundbreaking multimodal, multitask, multi-teacher joint-grained knowledge distillation model designed explicitly to understand visually-rich form documents. 2) our model outperforms publicly available form document datasets. ## related works visually rich document (vrd) understanding entails comprehending the structure and content of documents by capturing the underlying relations between textual and visual modalities. several downstream tasks, such as layout analysing @xcite , key information extraction (kie) @xcite , document classification (dc) @xcite , and visual question answering (vqa) @xcite , have contributed to raising the attention of the multimodal learning community as shown by multimodal approaches: they leverage both the recognised text and the spatial relations (denoted by t and s) between document components (e.g., lilt @xcite and bros @xcite ). the main goal is to complement raw content understanding with layout information. expanding upon this multimodal frame-work, models such as layoutlmv3 @xcite , docformerv2 @xcite , fast-structext @xcite , and, form-netv2 @xcite integrate the visual modality with text and layout information. these approaches are capable of capturing nuances in the document content hidden in prior works. to leverage multimodal relations, these models are typically pre-trained in a multi-task fashion, exploiting a curated set of token-or word-based pre-training tasks, such as masking or alignment. our approach aligns with the multimodal model paradigm, distinguishing itself by eschewing generic pre-training tasks reliant on masking, alignment, or ntp. instead, it leverages the direct extraction of knowledge from multiple teachers, each trained on downstream datasets, encompassing both entity and token levels of analysis with the proposed intra-grained and cross-grained losses. this enriches the depth of understanding in visual documents, capturing intricate relationships and semantic structures beyond individual tokens. ## methodology as previously noted, our paper focuses on interpreting visually rich documents, particularly form documents created and used collaboratively by multiple parties. to accomplish this objective, we introduce and employ two tiers of multimodal information: fine-grained and coarse-grained levels, which play a crucial role in understanding the structure and content of an input form page. note that existing pre-trained visual-language models, whether designed for generic documents, possess implicit knowledge on either fine-grained or coarse-grained aspects. hence, we propose an approach that harnesses knowledge from diverse pre-trained models throughout training. this strategy aims to generate more comprehensive and representative multi-and joint-grained document representations, ultimately enhancing the effectiveness of downstream tasks related to document understanding. ## conclusion we introduced a multimodal multi-task multi-teacher framework in visually-rich form documents. our model incorporates multi-teacher, multi-task, and multi-loss, and the results show the robustness in capturing implicit knowledge from multi-teachers for understanding diverse form document natures, such as scanned, printed, and handwritten. we hope our work provides valuable insights into leveraging multi-teacher and multi-loss strategies for document understanding research.