DUBLIN : Visual Document Understanding By Language-Image Network (2023)

Abstract: In this paper, we present DUBLIN, a pixel-based model for visual document understanding that does not rely on OCR. DUBLIN can process both images and texts in documents just by the pixels and handle diverse document types and tasks. DUBLIN is pretrained on a large corpus of document images with novel tasks that enhance its visual and linguistic abilities. We evaluate DUBLIN on various benchmarks and show that it achieves state-of-the-art performance on extractive tasks such as DocVQA, InfoVQA, AI2D, OCR-VQA, RefExp, and CORD, as well as strong performance on abstraction datasets such as VisualMRC and text captioning. Our model demonstrates the potential of OCR-free document processing and opens new avenues for applications and research.

## introduction humans have an incredible ability to process documents visually, interpreting the layout and extracting valuable information from images and texts simultaneously. document layouts, with strategically placed figures, tables, and other visual elements, are designed to cater to human perception and visual cognition biases @xcite . however, in most contemporary visual document processing models, on the other hand, ocr is commonly employed as a starting point @xcite @xcite for extracting the text, followed by a text-only processing scheme. despite its usefulness, ocr can introduce errors, which can be particularly problematic in scenarios involving non-latin scripts or handwritten content. more importantly, ocrbased methods fall short in capturing the rich visual context present in document images, making them less effective for various applications @xcite @xcite . ## related works the transformer architecture has become prevalent in document understanding, and the layoutlm family of models has extended transformer-based approaches like bert @xcite to handle document visuals. various features, such as 2d spatial positional information @xcite , visual tokens, spatially biased attention @xcite , and crossmodal alignment objective @xcite , have been integrated into these models. however, some evaluations of layoutlm models overlooked text recognition, an essential task. doc-former used only visual features near text tokens @xcite . ernie-layout used reading order prediction as a pretraining task @xcite . tilt trained generative language models on document data using generative objectives @xcite . recent advances in document understanding have focused on self-supervised learning and multimodal embeddings. udoc used multi-modal embeddings and self-supervised losses to learn joint representations for words and visual features from document images @xcite . selfdoc used coarse-grained multimodal inputs, cross-modal learning, and modality-adaptive attention to model document components @xcite . udop used a vision-text-layout transformer and a prompt-based sequence generation scheme to enable document understanding, generation, and editing across domains @xcite . the above-described models depend on off-theshelf ocr tools for text processing in documents, which limits their applications and increases computational costs. recent models like donut @xcite , dessurt @xcite , and pix2struct @xcite are end-to-end imageto-text models that do not need ocr at inference time. pix2struct is a pretrained image-to-text model for purely visual language understanding that can be fine-tuned on tasks containing visually-situated language @xcite . it was pretrained by learning to parse masked screenshots of web pages into simplified html and enables resolution flexibility to a variety of visual language domains. matcha proposed pretraining objectives to enhance the mathematical reasoning and chart derendering capability of visual language models @xcite . ## experiments and results we conduct comprehensive experiments on various types of documents, such as handwritten, typewritten, scanned, infographics, diagrams, tables, and webpages, and evaluate our model on various downstream tasks to assess the model's generalization capability. in this section, we describe the tasks, datasets, and the results. for each experiment, we finetune our pretrained model on a dataset and then report the performance. for each dataset, we use the publicly available train/development/test set splits, except for websrc where the test set is not released and hence we report performance on the development set. the hyperparameters used for finetuning are listed in appendix e. we also adopt the following two generic strategies for input formatting: first, inspired by the pix2struct, for all tasks, we append the question/key visually rendered onto the document image itself as can be seen in figure 3 . subsequently, we also utilize the question/key as a prefix for the text decoder. second, for accommodating diverse image size and aspect ratios, we employ a variable resolution finetuning strategy. @xcite addresses the issue of aspect ratio distortion by rescaling input images either up or down to ensure the extraction of the maximum number of patches that can fit within the designated sequence length. however, this resizing technique can lead to a potential loss of information due to under-utilization of maximum sequence length tokens. in contrast, we focus on preserving information by adopting a different strategy that resizes the image to an aspect ratio which is an even power of 2 (e.g., 1, 4, 16, 64, etc.) as depicted in figure 4 . by doing so, we maintain the desired aspect ratio while accommodating the maximum allowable number of patches (4096) within the given sequence length. as a result, we have two versions of dublin, one fixed resolution model and one variable resolution model, which we called dublin fixed_res and dublin variable_res respectively. ## conclusion we have presented dublin, a transformer-based encoder-decoder model for visual document understanding that can analyze both text and visual elements in document images. evaluation on diverse downstream tasks show that it achieves competitive or superior performance compared to the existing state-of-the-art models. our work shows that dublin is a versatile and robust model that does not rely on external ocr systems and can be finetuned in an end-to-end fashion. we also introduce a new evaluation setup on text-based datasets by rendering them as images. while this is an unfair comparison as text-based models are expected to perform better for these tasks, this also serves as a challenging baseline for benchmarking vdu models.