UR eader: Universal OCR -free Visually-situated Language Understanding with Multimodal Large Language Model (2023)

Abstract: Text is ubiquitous in our visual world, conveying crucial information, such as in documents, websites, and everyday photographs. In this work, we propose UReader, a first exploration of universal OCR-free visually-situated language understanding based on the Multimodal Large Language Model (MLLM). By leveraging the shallow text recognition ability of the MLLM, we only finetuned 1.2% parameters and the training cost is much lower than previous work following domain-specific pretraining and finetuning paradigms. Concretely, UReader is jointly finetuned on a wide range of Visually-situated Language Understanding tasks via a unified instruction format. To enhance the visual text and semantic understanding, we further apply two auxiliary tasks with the same format, namely text reading and key points generation tasks. We design a shape-adaptive cropping module before the encoder-decoder architecture of MLLM to leverage the frozen low-resolution vision encoder for processing high-resolution images. Without downstream finetuning, our single model achieves state-of-the-art ocr-free performance in 8 out of 10 visually-situated language understanding tasks, across 5 domains: documents, tables, charts, natural images, and webpage screenshots. Codes and instruction-tuning datasets will be released.

## introduction leveraging strong large language models as the language decoder, some recent works propose multimodal large language models (mllms) @xcite @xcite and achieve promising vision-andlanguage understanding performance. surprisingly, without in-domain training, these mllms exhibit shallow zero-shot visual text recognition ability when fed a low-resolution image with salient text information @xcite . however, due to the variety of image types and the wide range of image sizes, they are still far from universal visually-situated language understanding, such as extracting information from documents, reading texts from webpages, and visual question and answering on tables, as shown in figure 1 . existing works for visually-situated language understanding can be categorized into two-stage @xcite @xcite and end-to-end @xcite @xcite methods according to whether relying on an off-the-shelf ocr model or api. these works all follow a domain-specific pretraining and finetuning paradigm, thus leading to high training costs, e.g. end-to-end model donut @xcite costs more than 192 a100 days. inspired by the shallow text recognition ability of existing mllms, in this work, we propose ureader for universal ocr-free visually-situated language understanding, which leverages the multimodal large language model via low-cost instruction tuning @xcite . different from previous works, we forgo pretraining tasks by leveraging the existing mllm and directly finetune mllm by taking full advantage of various visually-situated language understanding datasets. to make the most of the strong language understanding ability of mllm, we convert all tasks into the visionlanguage instruction tuning format. besides, to enhance text recognition and semantic understanding ability across diverse domains, we design auxiliary text reading and key points generation tasks in the same instruction format. to utilize the lowresolution encoder of mllm for processing highresolution images and avoid blurry and distortion problems due to resizing, we propose a shapeadaptive cropping module to cut a high-resolution image into multiple local images. each image is firstly independently encoded with the frozen visual encoder and a trainable visual abstractor and then concatenated to feed into the language decoder. moreover, we add learnable crop position encoding to help the model correlate local images and add a resized global image to alleviate salient information loss due to cropping. our contributions in this work are four-fold: ## related work visually-situated language understanding aims to comprehend images containing rich text information. the image types are quite diverse, covering document @xcite @xcite @xcite , table @xcite , chart @xcite @xcite , natural image @xcite @xcite , webpage screenshot @xcite , etc. ## ureader the primary goal of ureader is to efficiently utilize existing mllms for visually-situated language understanding tasks. in this work, we utilize but are not limited to, the mplug-owl @xcite as our basic mllm. figure 2 presents an overall architecture of ureader. the input image is firstly pre-processed by a shape-adaptive cropping module (in section 3.1). the resulting sub-images are then simultaneously passed through the visual encoder and visual abstractor. to enable the large language model to correlate multiple cropped subimages, we apply a crop position encoding module to introduce spatial information across sub-images. (in section 3.2). ## instruction tuning for developing a universal visually-situated language understanding model that could process various types of images and perform different comprehension tasks, we conduct low-cost instruction tuning with a multimodal large language model. without introducing any large-scale pretraining datasets, we directly ensemble multiple downstream datasets and perform joint training. different downstream tasks are all reorganized to the unified instruction format @xcite . besides, we design auxiliary text reading and key points generation tasks to enhance text recognition and semantic understanding ability. ## conclusion we first propose to leverage existing multimodal large language models for universal ocr-free visually-situated language understanding through low-cost instruction tuning. all downstream tasks are reorganized into a unified instruction-tuning format. besides, we design the text reading task and key points generation task to enhance text recognition and vision-and-language semantic comprehension abilities. to utilize the pre-trained vision encoder for processing high-resolution images, we design a shape-adaptive cropping module, which cuts the image into multiple local images considering its raw aspect ratio and resolution. ureader achieve state-of-the-art ocr-free performance in 8 out of 10 datasets, ranging from documents, tables, charts, and natural images to webpage screenshots.