MATE : Meet At The Embedding - Connecting Images with Long Texts (2024)

Abstract: While advancements in Vision Language Models (VLMs) have significantly improved the alignment of visual and textual data, these models primarily focus on aligning images with short descriptive captions. This focus limits their ability to handle complex text interactions, particularly with longer texts such as lengthy captions or documents, which have not been extensively explored yet. In this paper, we introduce Meet At The Embedding (MATE), a novel approach that combines the capabilities of VLMs with Large Language Models (LLMs) to overcome this challenge without the need for additional image-long text pairs. Specifically, we replace the text encoder of the VLM with a pretrained LLM-based encoder that excels in understanding long texts. To bridge the gap between VLM and LLM, MATE incorporates a projection module that is trained in a multi-stage manner. It starts by aligning the embeddings from the VLM text encoder with those from the LLM using extensive text pairs. This module is then employed to seamlessly align image embeddings closely with LLM embeddings. We propose two new cross-modal retrieval benchmarks to assess the task of connecting images with long texts (lengthy captions / documents). Extensive experimental results demonstrate that MATE effectively connects images with long texts, uncovering diverse semantic relationships.

## introduction recent advancements in vision language models (vlms) such as clip @xcite and others @xcite @xcite have successfully connected visual and textual data by embedding them into a shared space. these models exhibit robust generalization across various visual domains, including medical imaging, art, and remote sensing @xcite my friend ben's first time playing with a frisbee... ## method in this section, we present our mate method, which aims to establish image-long text alignment by employing a vlm image encoder and a pretrained llm-based encoder. it should be noted that mate does not require additional image-long text pairs for training. the pre-trained clip @xcite and llm-based e5 @xcite are utilized as our baseline models. first, we investigate how these models are trained to distribute embeddings (in section 3.1) to assess the feasibility of connecting these models. next, we outline the multi-stage training strategy (in section 3.2) that efficiently achieves our goal. ## conclusion in this paper, we introduce mate, a novel method that effectively bridges the gap between images and extensive texts without paired data. mate integrates a pretrained llm-based text encoder with a vlm-based image encoder to efficiently align image embeddings with text embeddings. the process begins by aligning vlm text embeddings with llm embeddings using extensive text pairs, followed by aligning image embeddings with these llm embeddings. we also introduce new benchmarks to test image-long text retrieval tasks, demonstrating that mate effectively connects images with extensive texts. this work pioneers a new direction for research in cross-modal interactions.