XFUND : A Benchmark Dataset for Multilingual Visually Rich Form Understanding (2022)

Abstract: Multimodal pre-training with text, layout, and image has achieved SOTA performance for visually rich document understanding tasks recently, which demonstrates the great potential for joint learning across different modalities. However, the existed research work has focused only on the English domain while neglecting the importance of multilingual generalization. In this paper, we introduce a human-annotated multilingual form understanding benchmark dataset named XFUND, which includes form understanding samples in 7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese). Meanwhile, we present LayoutXLM, a multimodal pre-trained model for multilingual document understanding, which aims to bridge the language barriers for visually rich document understanding. Experimental results show that the LayoutXLM model has significantly outperformed the existing SOTA cross-lingual pre-trained models on the XFUND dataset. The XFUND dataset and the pre-trained LayoutXLM model have been publicly available at https://aka.ms/layoutxlm .

## introduction recently, multimodal pre-training for visually rich document understanding (vrdu) has achieved new sota performance on several public benchmarks @xcite , including form understanding @xcite , receipt understanding @xcite , complex layout understanding @xcite , document image classification @xcite and document vqa task @xcite , due to the advantage that text, layout and image information is jointly learned end-to-end in a single framework. however, since most evaluation benchmarks focus on english vrds, it is hard to explore the performance of a document understanding system on vrds in other languages. simply translating these documents automatically with machine translation services might help, but it is often not satisfactory due to the poor translation quality on document images @xcite . therefore, it is vital to explore the multilingual generalization ability of multimodal pre-training for vrdu tasks. multilingual pre-trained models such as mbert @xcite , xlm @xcite , xlm-roberta @xcite , mbart @xcite , and the recent infoxlm @xcite and mt5 @xcite have pushed many sota results on cross-lingual natural language understanding tasks by pre-training the transformer models on different languages. these models have successfully bridged the language barriers in a number of cross-lingual transfer benchmarks such as xnli @xcite and xtreme @xcite . although a large amount of multilingual text data has been used in these cross-lingual pre-trained models, text-only multilingual models cannot be easily used in the vrdu tasks because they are usually fragile in analyzing the documents due to the format/layout diversity of documents in different countries, and even different regions in the same country. hence, to accurately understand these visually rich documents in different languages, it is crucial to pre-train the multilingual models in a multimodal framework. meanwhile, it is vital to provide a human-labeled benchmark to further facilitate multilingual document understanding. to this end, we introduce a human-annotated multilingual form understanding benchmark dataset named xfund, which contains 7 languages, including chinese, japanese, spanish, french, italian, german, portuguese. in addition to the fully annotated data, we propose two subtasks with three different settings. the two subtasks are semantic entity recognition and relation extraction. and we introduce three different settings to explore the multilingual and complex layout generalization ability: (1) language-specific fine-tuning follows the typical paradigm of fine-tuning and testing on the same language. (2) zero-transfer learning means that the model is trained on english data only and then evaluated on each target language. moreover, we also present a multimodal pretrained model for multilingual vrdu tasks, aka layoutxlm, which is a multilingual extension of the recent layoutlmv2 model @xcite . to evaluate the multilingual generalization ability of this framework, we use the pre-training objectives of layoutlmv2, including masked visual-language model (mvlm), image-text matching (itm), and image-text alignment (ita). in addition, we pre-train the model with the iit-cdip dataset @xcite as well as a great number of publicly available digital-born multilingual pdf files from the internet, which helps the layoutxlm model to learn from real-world documents. in this way, the model obtains textual and visual signals from a variety of document templates/layouts/formats in different languages, thereby taking advantage of the local invariance property from both textual, visual and linguistic perspectives. experiment results show that the pre-trained layoutxlm outperforms several sota cross-lingual pre-trained models @xcite on the xfund benchmark dataset, which also demonstrates the potential of the multimodal pre-training strategy for multilingual document understanding. the contributions of this paper are summarized as follows: ## xfund as illustrated in figure 1 , we develop our xfund dataset in four steps including §2.1 template collection, §2.2 form creation, §2.3 key-value annotation, and §2.4 data finalization and statistics, spending around 1,500 hours of human labor in total. further details of ethic consideration are presented in §a ethical consideration. ## layoutxlm in this section, we present a powerful baseline model layoutxlm and introduce its model architecture, pre-training objectives, and pre-training dataset. we follow the layoutlmv2 @xcite architecture and transfer the model to largescale multilingual document datasets. ## key-value extraction with plms in this section, we present a simple yet efficient baseline framework based on pre-trained language models (plms) for our two sub-tasks. equipped with this framework, we integrate two existing popular cross-lingual pre-trained language models, xlm-roberta and infoxlm, and our proposed layoutxlm as the pre-trained language model backbones. in this framework, given a visually rich document d, we will pass discrete token set @xmath0, t 1 , ..., t n } into these backbone models to obtain the contextual representation of each tokens @xmath1, h 1 , ..., h n }. for different tasks, the representations will be processed with different modules to predict the required labels. ## conclusion in this paper, we introduce the multilingual form understanding benchmark xfund, which includes key-value labeled forms in 7 languages. meanwhile, we present layoutxlm, a multimodal pre-trained model for multilingual visually rich document understanding. we make xfund and layoutxlm publicly available to advance the document understanding research. for future research, we will further enlarge the multilingual training data to cover more languages as well as more document layouts and templates. in addition, as there are a great number of business documents with the same content but in different languages, we will also investigate how to leverage the contrastive learning of parallel documents for the multilingual pre-training. figure 5: language distribution of the data for pre-training layoutxlm. we also show the document counts per language for different sampling exponents α.