Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond (2024)

Abstract: The recent advancements in generative language models have demonstrated their ability to memorize knowledge from documents and recall knowledge to respond to user queries effectively. Building upon this capability, we propose to enable multimodal large language models (MLLMs) to memorize and recall images within their parameters. Given a user query for visual content, the MLLM is anticipated to “recall” the relevant image from its parameters as the response. Achieving this target presents notable challenges, including inbuilt visual memory and visual recall schemes within MLLMs. To address these challenges, we introduce a generative cross-modal retrieval framework, which assigns unique identifier strings to represent images and involves two training steps: learning to memorize and learning to retrieve. The first step focuses on training the MLLM to memorize the association between images and their respective identifiers. The latter step teaches the MLLM to generate the corresponding identifier of the target image, given the textual query input. By memorizing images in MLLMs, we introduce a new paradigm to cross-modal retrieval, distinct from previous discriminative approaches. The experiments demonstrate that the generative paradigm performs effectively and efficiently even with large-scale image candidate sets.

## introduction recently, we have witnessed the explosive development of generative large language models (llms), such as gpt series @xcite and llama @xcite . undergone extensive pretraining on document corpora and instruction tuning, these language models have demonstrated an impressive ability to memorize a lot of knowledge in their parameters and * corresponding authors. ## conclusion and future work in this paper, we delved into a novel memorization mechanism for the mllm to memorize images within its parameters. building upon inbuilt visual memory within mllm, we proposed a generative cross-modal retrieval framework, which introduces a fresh paradigm in cross-modal retrieval. this paradigm transforms the original matching problem into a generation problem, eliminating the need for negative samples during training and image indexing during inference. our experiments demonstrate that the generative paradigm performs effectively and efficiently even with large-scale image sizes. furthermore, we showcased the mllm's ability to interact (e.g., describe and qa) with memorized images, following specific instructions. moving forward, we aim to further develop this topic from the following perspectives. on the one hand, although our generative framework achieves comparable performance to previous cross-modal retrieval approaches, there are still challenges to address, such as the limitations of current identifiers. exploring more effective identifiers, like "visual tokens @xcite ", would help to enhance generative cross-modal retrieval further. on the other hand, since we have enabled mllms to memorize and interact with images, it opens up the possibility of injecting personalized visual experiences of humans into mllms for them to understand an individual's visual journey and accomplish more visual tasks. peter young, alice lai, micah hodosh, and julia hockenmaier. 2014. from image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. transactions of the association for computational linguistics, 2:67-78. yidan zhang, ting zhang, dong chen, yujing wang, qi chen, xing xie, hao sun, weiwei deng, qi zhang, fan yang, et al. 2023. irgen: generative modeling for image retrieval. arxiv preprint arxiv:2303.10126. zhedong zheng, liang zheng, michael garrett, yi yang, mingliang xu, and yi-dong shen. 2020. dual-path convolutional image-text embeddings with instance loss. acm transactions on multimedia computing, communications, and applications (tomm), 16(2):1-23. deyao zhu, jun chen, xiaoqian shen, xiang li, and mohamed elhoseiny. 2023. minigpt-4: enhancing vision-language understanding with advanced large language models. arxiv preprint arxiv:2304.10592.