Scene-Text Aware Image and Text Retrieval with Dual-Encoder (2022)

Abstract: We tackle the tasks of image and text retrieval using a dual-encoder model in which images and text are encoded independently. This model has attracted attention as an approach that enables efficient offline inferences by connecting both vision and language in the same semantic space; however, whether an image encoder as part of a dual-encoder model can interpret scene-text (i.e., the textual information in images) is unclear. We propose pre-training methods that encourage a joint understanding of the scene-text and surrounding visual information. The experimental results demonstrate that our methods improve the retrieval performances of the dual-encoder models.

## introduction when pre-trained on a large-scale corpus of image and text pairs, vision and language models can obtain effective multi-modal representations that bridge the semantic gap between visual and textual information. in general, two approaches are used: 1) the cross-encoder approach, in which textual and visual information are jointly fed into a single transformer-based model @xcite , and 2) the dual-encoder approach, in which the textual and visual information are independently fed into two modality-specific encoders. crossencoder models use cross-modal attention, which facilitates the interpretation of the different modalities. however, such models are not suitable for image retrieval and other tasks requiring fast and large-scale inferences @xcite . in contrast, dual-encoder models can make quick inferences, but their interpretation of concomitant modalities is insufficient; in particular, such models have difficulty jointly interpreting scene-text and the surrounding visual information. given the above background, this paper investigates the effectiveness of incorporating scene-text into a dual-encoder. the contributions of this study are as follows. 1) we introduce pre-training methods for a dual-encoder to facilitate a joint interpretation of the textual information in the images and surrounding visual information (figure 1 ). the performance of the model is then evaluated for image and text retrieval tasks. 2) we experimentally show that, similar to cross-encoder approaches, the joint scene-text and semantic representations improve the retrieval performance of the dual-encoder. ## related work to make sense of visual and textual semantics, recent studies concerning vision and language pretraining, such as image captioning and text-aware vqa @xcite @xcite , incorporate concomitant textual information, such as scene-text and object tags, in terms of regions-of-interest to enable cross-modal interactions using self-attention in a transformer-based model (cross-encoder) @xcite @xcite @xcite . however, cross-encoders are not suitable for image retrieval or other tasks requiring fast and large-scale inferences. although cross-encoder models typically allow expressive token-wise interactions for an input pair of a query and retrieval target, the similarity score cannot be decomposed and is not indexable @xcite . therefore, such models are impractical for application in tasks with many queries requiring quick responses, such as retrieval tasks. in contrast, dual-encoder approaches @xcite @xcite can successfully perform downstream tasks, enabling efficient offline inferences of all preencoded image and text embeddings. however, the effectiveness of incorporating concomitant modalities, such as scene-text, in dual-encoder models has not been thoroughly investigated or demonstrated in the community. ## scene-text aware dual-encoder this paper proposes the incorporation of textual information in images into the dual-encoder architecture. we build our method based on the light-ningdot @xcite ) framework, a cuttingedge dual-encoder that encodes both object-wise and token-wise representations. we first briefly introduce lightningdot in its current use. we then describe the proposed method, including the learning objectives, to incorporate the textual information in the images into the image encoder. ## conclusion we proposed a framework that incorporates the textual information in images into the dual-encoder architecture. an evaluation on the textcaps dataset confirmed that modeling the scene-textaware cross-modal relationships benefited the dualencoder architecture. in future research, we will attempt a more robust exploration of scene-text modeling @xcite .