Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Models (2023)

Abstract: Recent advances in Large Language Models (LLMs) have stimulated a surge of research aimed at extending their applications to the visual domain. While these models exhibit promise in generating abstract image captions and facilitating natural conversations, their performance on text-rich images still requires improvement. In this paper, we introduce Contrastive Reading Model (Cream), a novel neural architecture designed to enhance the language-image understanding capability of LLMs by capturing intricate details that are often overlooked in existing methods. Cream combines vision and auxiliary encoders, fortified by a contrastive feature alignment technique, to achieve a more effective comprehension of language information in visually situated contexts within the images. Our approach bridges the gap between vision and language understanding, paving the way for the development of more sophisticated Document Intelligence Assistants. Through rigorous evaluations across diverse visually-situated language understanding tasks that demand reasoning capabilities, we demonstrate the compelling performance of Cream, positioning it as a prominent model in the field of visual document understanding. We provide our codebase and newly-generated datasets at https://github.com/naver-ai/cream.

## introduction recent advances in large language models (llms) @xcite @xcite have facilitated the development of numerous real-world applications. researchers are increasingly focusing on extending these unimodal llms to multimodal ˚core contributions. correspondence to geewook kim: gwkim.rsrch@gmail.com. ## method our goal is to develop a system that accurately answers natural language questions based on specific evidence in an input image. to achieve this, we propose a model that explicitly identifies feature evidence in the image, such as texts and objects. ## conclusion in this paper, we present cream, a novel approach overcoming the constraints of current lvlms for visual tasks on text-rich images. cream's robust architecture synergizes a vision encoder, auxiliary encoder, and sophisticated techniques, including contrastive feature alignment. our comprehensive evaluations confirm cream's promising languageimage understanding capabilities and robustness against ocr errors. the integration of cream with llms provides a solid foundation for future improvements in visually-situated language comprehension. we believe our findings can easily be extended to other domains/tasks regarding visuallysituated natural language understanding. evaluated easyocr 7 and paddleocr 8 . note that, paddleocr operates as a lightweight, cpu-based ocr engine, thus potentially acting as a preferred option for those seeking to decrease overall model deployment costs. figure 10 presents a comparative performance analysis of cream and udop on the text-rich document vqa benchmark, docvqa. as can be observed in the figure, cream exemplifies superior robustness. figure 11 , additionally, exhibits ablated models of cream, evaluating the influence of cl. these findings are aligned with the trends noted in the main manuscript, underscoring that our implementation of cl improves robustness to ocr errors. figure 11 shows the performance of a new ablated model, vision encoder and text decoder (vetd), which is derived from cream small by removing the auxiliary encoder in the architecture. we use vetd as a performance baseline; if cream's performance does not exceed that of vetd, it suggests that the addition of the auxiliary encoder may not be justified. as indicated in the figure, by incorporating the proposed cl technique, the overall enhancement in performance is observed, enabling cream small to outperform the baseline vetd, even with the deployment of a cpu-based lightweight ocr engine -pad-dleocr.