Unified Embeddings for Multimodal Retrieval via Frozen LLM s (2024)

Abstract: In this work, We present Unified Embeddings for Multimodal Retrieval (UniMuR), a simple but effective approach that embeds multimodal inputs and retrieves visual and textual outputs via frozen Large Language Models (LLMs). Specifically, UniMuR jointly retrieves multimodal outputs via a unified multimodal embedding and applies dual alignment training to account for both visual and textual semantics. Thus, unlike previous approaches, UniMuR significantly reduces LLM’s modality bias towards generating text-only outputs. Meanwhile, the proposed unified multimodal embedding mitigates the inconsistency between visual and textual outputs and provides coherent multimodal outputs. Furthermore, benefiting from the joint training of visual and textual semantics, UniMuR also achieves strong image/text retrieval ability. Compared to existing approaches, UniMuR achieves better zero-shot multimodal response retrieval performance on MMDialog, improving the overall R@1 by 6.5% while boosting the image retrieval rate and having better cross-modal consistency on multimodal outputs. UniMuR also achieves 2.4% and 3.9% improvement on context-based image retrieval tasks on MMDialog and VisDial respectively when compared to previous approaches, validating its generalization ability across multiple tasks.

## introduction trained on massive text corpora sourced from the internet, large language models (llms) have showcased remarkable capabilities, ranging from generating human-like dialogue to answering complex queries posed by users @xcite @xcite @xcite . one limitation of most widely available state-of-the-art llms-with some exceptions-is that they focus on text-only interactions and do not utilize visual information. however, beyond language, visual information is a fundamental signal through which humans perceive what do pandas eat? these animals eat bamboo. ## methodology in this section, we present unimur, a general approach based on frozen llms and image-text pretrained models. the training pipeline of our proposed approach is illustrated in figure 2 . we propose two alternating steps to embed multimodal inputs and retrieve multimodal outputs via frzoen llms. as shown in the left part of figure 2 , in the image-to-text training step, we train a linear mapping layer that maps the image into llm's input space in order to access the multimodal understanding ability of the llm. in the dual alignment training step, we propose to match the visual and textual semantics with the unified multimodal embedding, shown in the right part of figure 2 . during inference, unimur jointly retrieves multimodal outputs via the trained unified multimodal embedding. below, we discuss the different components of our unimur approach in more detail. ## results in this section, we present the empirical results of our proposed approach -unimur. we evaluate unimur on 3 different multimodal retrieval tasks; multimodal response retrieval (section 5.1), contextual image retrieval (section 5.2), and dialogueto-image retrieval (section 5.3). ## analysis in this section, we further analyze unimur to understand the impact of different model design choices as well as to showcase its capabilities. ## conclusion we present unified embeddings for multimodal retrieval (unimur), a simple yet effective approach that retrieves visual and textual output via unified multimodal embeddings and significantly reduces the text-centric bias from the llm's output as compared to previous approaches. we empirically show that unimur achieves better zero-shot multimodal response retrieval than state-of-the-art approaches through its joint retrieval process that is capable of retrieving multimodal outputs with better cross-modal consistency. in addition, unimur improves dialogue-to-image retrieval and contextual image retrieval performance to demonstrate its improved performance across multiple tasks.