A Multi-Modal Multilingual Benchmark for Document Image Classification (2023)

Abstract: Document image classification is different from plain-text document classification and consists of classifying a document by understanding the content and structure of documents such as forms, emails, and other such documents. We show that the only existing dataset for this task (Lewis et al., 2006) has several limitations and we introduce two newly curated multilingual datasets WIKI-DOC and MULTIEURLEX-DOC that overcome these limitations. We further undertake a comprehensive study of popular visually-rich document understanding or Document AI models in previously untested setting in document image classification such as 1) multi-label classification, and 2) zero-shot cross-lingual transfer setup. Experimental results show limitations of multilingual Document AI models on cross-lingual transfer across typologically distant languages. Our datasets and findings open the door for future research into improving Document AI models.

## introduction visual document understanding aims to extract useful information from a variety of documents (e.g., forms, tables) beyond merely optical character recognition (ocr). unlike texts in traditional nlp tasks, this task is challenging since the content is laid out in a 2d structure where each word contains an (x, y) location in the document. prior work has shown that this task requires a model to process information occurring in multiple modalities including cues in images and text as well as spatial cues @xcite . visual document understanding consists of several sub-tasks including, but not limited to, document image classification @xcite , entity extraction (guillaume jaume, 2019) and labeling @xcite , and visual question answering @xcite . models for document understanding tasks have also benefited from large-scale unsupervised pretraining that infuses data from the various modalities @xcite . in this work, we focus on the task of document classification on visually-rich documents, which aims to classify a given input document (usually pdf or an image) into one or more classes. a popular benchmark dataset for this task is the rvl-cdip collection @xcite which consists of 16 different classes. however, this dataset is not designed to evaluate models on the document classification task with deeper understanding @xcite -it consists of documents in english only, the documents are relevant to a single domain, each document belongs to only one class (i.e., multi-class), and some class labels (e.g., "email", "resume") do not require rich semantic understanding of the contents of the document. given these limitations, we argue that it is paramount to expand the evaluation for document classification to gain further insights into existing approaches, as well as identify their limitations. we propose two new datasets, multieurlex-doc and wiki-doc, that complement rvl-cdip in different ways. multieurlex-doc consists of eu laws in 23 different languages and is a multilabel dataset where each document is assigned to one or more of different labels. additionally, wiki-doc is a multi-class dataset comprises of rendered wikipedia articles that covers non-european languages. these datasets are derived from prior work by @xcite and @xcite respectively. both datasets require a deeper understanding of the text and contents of the documents to arrive at the correct label(s). we use these new datasets to study the behavior of pre-trained visually-rich document understanding (or document ai) models with a focus on answering three specific questions in document image classification. first, focusing only on the english portions of the datasets, we examine if different pretrained models perform consistently across datasets. second, we ask whether multi-lingual document understanding models exhibit similar performances across different languages. finally, we focus on the cross-lingual generalization ability of these models, and ask whether multi-lingual document understanding models can be used to classify documents without having to be trained on that specific language. we use a variety of pre-trained models that consume different inputs -text-only @xcite , text+layout @xcite @xcite , multi-modal @xcite @xcite ) that fuse input from text, layout, and visual features, as well as image-only models @xcite . empirical results on the two new datasets reveal that image-only models have large improvement opportunities unlike what is reported on rvl-cdip @xcite , and multi-modal models have limited cross-lingual generalization ability. our main contributions are as follows: ## related work models designed for document image understanding tasks, often referred to as document ai models, appear in various different forms. we give a brief overview in this section and leave the details to the survey paper by @xcite . technically, any pretrained text models (e.g., bert or gpt-3) can be applied to handle document images after being processed by ocr tools. however, initial work by @xcite opened up the exploration of including additional modalities such as layouts and images for handling documents. since then, many layoutaware models and pretraining tasks have been proposed in the literature @xcite @xcite @xcite , followed by using images as additional input for better models that exploit a broader set of signals from the input @xcite @xcite @xcite . recently, document ai models using only images as inputs (or ocr-free models) have been attracting attention from researchers and practitioners @xcite @xcite . as a results of not being dependent on ocr, such models potentially avoid the propagation of ocr errors. furthermore, these models do not have a fixed vocabulary and can technically be applied to any language without out-of-vocabulary concerns. nevertheless, most models support only english, and only limited number of multilingual models have been explored in the literature. additionally, such models are not thoroughly benchmarked on multilingual datasets due to lack of appropriate datasets. we address this by curating new multilingual datasets for document classification. ## multilingual evaluation datasets for document ai models a common benchmark dataset for evaluating document ai models on document classification tasks is rvl-cdip @xcite . however, we argue that using a single benchmark dataset naturally inhibits our understanding of the task as well as the solutions built for it. specifically, we argue that rvl-cdip has the following limitations: 1. it only includes english documents, thus limiting our understanding of multilingual document understanding models @xcite and their cross-lingual generalization ability. 2. the labels assigned to documents focus on the type of the document (e.g., "emails", "invoices", "tax form"), which does not necessarily require a deeper understanding of the contents of the documents. @xcite provide an in-depth analysis on these limitations of rvl-cdip and recommend that new datasets for document image classification should be multi-label to handle the naturally occurring overlap across labels, large-scale with 100+ classes, and multilingual to test the ability of models to transfer across languages (besides being accurately labeled). we introduce newly curated datasets covering these desired characteristics to complement the limitations of rvl-cdip. ## experiments we now empirically explore both intra-and crosslingual generalization ability of document ai models for document classification for multiple languages. we ask the following research questions: rq1: do document ai models, specifically those that are multilingual in nature, perform equally accurately across languages? rq2: can document ai models classify documents in a cross-lingual transfer setting where we train on english and evaluate on other languages? we conduct experiments under two different settings to answer these questions: 1) intralingual ## conclusion in this paper, we curated two new multilingual document image classification datasets, multieurlex-doc and wiki-doc, to evaluate both the multilingual and cross-lingual generalization ability of document ai models. through benchmarking on the two newly curated datasets, we show strong intralingual results across languages of the multimodel model but also show the limited cross-lingual generalization ability of the multimodal model. furthermore, the ocr-free or image-only model showed the largest gap between the best performing models, showing large areas of improvement in datasets which require deeper content understanding from texts. future work in this space should investigate improvement strategies of multi-modal and ocr-free document ai models to enable them to achieve a deeper understanding of the text from document images. finally, the curated datasets still cover only a small subset of languages spoken in the world. future work should expand the datasets and experiments to more diverse set of language families.