MGD oc: Pre-training with Multi-granular Hierarchy for Document Image Understanding (2022)

Abstract: Document images are a ubiquitous source of data where the text is organized in a complex hierarchical structure ranging from fine granularity (e.g., words), medium granularity (e.g., regions such as paragraphs or figures), to coarse granularity (e.g., the whole page). The spatial hierarchical relationships between content at different levels of granularity are crucial for document image understanding tasks. Existing methods learn features from either word-level or region-level but fail to consider both simultaneously. Word-level models are restricted by the fact that they originate from pure-text language models, which only encode the word-level context. In contrast, region-level models attempt to encode regions corresponding to paragraphs or text blocks into a single embedding, but they perform worse with additional word-level features. To deal with these issues, we propose MGDoc, a new multi-modal multi-granular pre-training framework that encodes page-level, region-level, and word-level information at the same time. MGDoc uses a unified text-visual encoder to obtain multi-modal features across different granularities, which makes it possible to project the multi-granular features into the same hyperspace. To model the region-word correlation, we design a cross-granular attention mechanism and specific pre-training tasks for our model to reinforce the model of learning the hierarchy between regions and words. Experiments demonstrate that our proposed model can learn better features that perform well across granularities and lead to improvements in downstream tasks.

## introduction document images are ubiquitous and are often used as a representation for forms, receipts, printed papers, etc. unlike plain text documents, document images express rich information via both textual content and heterogeneous layout patterns, which leads to barriers to the automatic processing of these document images. here, layout pattern refers to how the text is spatially arranged on the document page and involves information from multiple levels of granularity. specifically, a layout pattern divides the entire page into individual regions and, within each region, the fine-grained textual content is distributed following a certain format, such as paragraphs, columns, lists, as shown in figure 1 . the layout of a document provides important cues for interpreting the document through spatial structures such as alignment, proximity, and hierarchy between content at different levels of granularity. for example, a numeric text field is more likely to be the total price of a grocery receipt if it is located at the bottom right of a table region; a region is more likely to correspond to the title area of a form if there are a lot of bold types inside of the region. in these two examples, it is important to understand page-level information (e.g., that the document is a receipt or a form), region-level information (e.g., that a region is a table or title), and word/token-level information (e.g., the font style of a word, or that a token is a number), as well as how these relate to each other. therefore, to facilitate the automatic processing of such documents, it is essential to consider the features of multiple granularities and let the model learn the hierarchy between different levels to encode the multi-granular structure in the document images. however, existing methods in document image understanding formulate the document image understanding tasks either at the word-level or regionlevel and thus do not use both cues. they mostly follow language modeling methods designed for plain text settings, formulating document image understanding tasks using word-level information and augmenting semantic features with spatial and visual features to exploit the word-level context @xcite @xcite . recent works go beyond fine-grained wordlevel inputs and focus on regions instead of words to acquire useful signals @xcite . by encoding the regions corresponding to paragraphs or text blocks, these region-level models manage to save training resources and achieve good performance with rich locality features. however, these models fail to leverage the cross-region word-level correlation, which is also necessary to tackle fine-grained tasks. motivated by this observation, we propose mgdoc, a new multi-modal multi-granular pretraining framework that encodes document information at different levels of granularity and represents them using multi-modal features as highlighted in figure 2 . specifically, we use the ocr engine to decompose a document page into three granularities: page-level, region-level, and word-level. following previous works @xcite , our multi-modal features represent text, layout (represented by bounding boxes), and image modalities. the input consists of information at different levels of granularity, and can be organized into a hierarchy within the page, which means words are included in the corresponding regions and the page includes all of them. we leverage attention to learn the correlation between inputs from different levels of granularity and add special attention weights to encode the hierarchical structure and relative distances @xcite @xcite . we rely on pre-training to encourage the model to learn the alignment between regions at different levels of granularity. in addition, we use masked language modeling for the word-level inputs and extend this idea into the more coarse-grained inputs. we mask a proportion of regions and ask the model to minimize the difference between the masked contextual features and the input features corresponding to the selected region. we validate mgdoc on three public benchmarks, the funsd dataset @xcite for form understanding, the cord dataset @xcite for receipt extraction, and the rvl-cdip dataset @xcite for document image classification. extensive experiments demonstrate the effectiveness of our proposed approach with great improvements on fine-grained tasks and good results on coarse-grained tasks. we summarize our contribution as follows: ## conclusions and future work we present mgdoc, a multi-modal multi-granular pre-training framework, which goes beyond the existing region-level or word-level models and leverages the contents at multiple levels of granularity to understand the document pages better. existing models fail to use the informative multi-granular features in the document due to the restriction from the word-level model architecture, and lead to unsatisfactory results. we solve these issues with the new architecture design and tailored pretraining tasks. with a unified multi-modal encoder, we embed the features from pages, regions, and words into the same hyperspace, and design a multigranular attention mechanism and multi-granularity modeling task for mgdoc to learn the spatial hierarchical relation between them. experiments show that our proposed model can understand the spatial relation between the multi-granular features and lead to improvements in downstream tasks. as for future work, since we have not fully exploited the multi-granular information, we will go beyond the page level and investigate the possibility of encoding multiple pages. we are also interested in inputs that are more fine-grained than word level, such as pixels.