Transferring General Multimodal Pretrained Models to Text Recognition (2023)

Abstract: This paper proposes a new method, OFA-OCR, to transfer multimodal pretrained models to text recognition. Specifically, we recast text recognition as image captioning and directly transfer a unified vision-language pretrained model to the end task. Without pretraining on large-scale annotated or synthetic text recognition data, OFA-OCR outperforms the baselines and achieves state-of-the-art performance in the Chinese text recognition benchmark. Additionally, we construct an OCR pipeline with OFA-OCR, and we demonstrate that it can achieve competitive performance with the product-level API.

## introduction optical character recognition (ocr) plays an important role in the real-world applications. it helps users or developers extract text contents from different types of images, including photos, scanned documents, etc. in practice, building a tool for ocr needs a pipeline consisting of a text localization module and a text recognition module. in this work, we focus on improving the accuracy of text recognition. text recognition has often been regarded as a key challenge owing to the room for improvements in recognition accuracy. in the deep learning era, the classical methods are mostly based on cnn and rnn, which are responsible for visual feature extraction and sequence modeling, respectively @xcite @xcite . recently, with the rise of transformer @xcite , researchers applied the transformer encoder-decoder framework to text recognition and achieved outperforming results over the baselines @xcite . however, most methods are based on largescale pretraining on human-annotated or synthetic ocr data. it is hard for other researchers to collect or create such data for reproduction. furthermore, the methods often include complex model or objective designs, like detr-like decoder @xcite , ctc loss @xcite , etc. these components also might hinder reproduction as they increase the difficulty in training. therefore, we naturally raise a question: is there any way to achieve high recognition accuracy without complex designs on data and model? inspired by the recent progress in multimodal pretraining, we argue that the transfer of a unified multimodal pretrained model is a possible solution. multimodal pretraining has proved significant to the performance of downstream tasks, and thanks to the rise of unified multimodal pretrained models, they can perform both cross-modal understanding and generation and achieve state-of-theart performance @xcite . we therefore propose to transfer the unified multimodal pretrained model by finetuning the pretrained model on the text recognition datasets with the task of image captioning, which is essentially a simple sequence-to-sequence learning task with maximum likelihood estimation for optimization. to support the effectiveness of the proposed method, we have conducted extensive experiments on the chinese text recognition benchmark @xcite covering multiple scenarios, including scene, web, document, and handwriting. specifically, we finetune the open-source chinese multimodal pretrained model ofa @xcite ) on text recognition, and we name the model ofa-ocr. figure 1 demonstrates the results of methods with or without general-domain pretraining. it shows that multimodal pretraining on generaldomain vision-language data can effectively boost downstream performance in text recognition. to achieve the best performance, we apply the multitask + single-task finetuning to ofa-ocr, and it outperforms the previous state-of-the-art methods on the benchmark. furthermore, through the ablation studies, we demonstrate the effectiveness of our method designs, including multitask + singletask finetuning, data augmentation, etc. furthermore, to enable deployment for real-world applications, we construct a pipeline with both ofa-ocr and a simple text localization module. we find that this simple pipeline can provide high-quality ocr performance, competitive with a productlevel api. ## conclusion in this work, we propose a simple method called ofa-ocr, which leverages the unified multimodal pretrained model and transfers it to text recognition by image captioning. to be more specific, we utilize the chinese multimodal pretrained model ofa without pretraining on ocr data and transfer it to text recognition with multitask + single-task finetuning. through extensive experiments, we demonstrate that ofa-ocr can achieve state-of-the-art performance on the chinese text recognition benchmark. additionally, we build a pipeline of ocr by integrating an existent simple text detection module and ofa-ocr. the deployed pipeline achieves competitive performance in comparison with a product-level api. we hope that this research sheds light on the application of general-domain multimodal pretraining, and also helps ocr practitioners.