Cross-modal Clustering-based Retrieval for Scalable and Robust Image Captioning (2025)

Abstract: Recent advances in retrieval-augmented generative image captioning (RAG-IC) have significantly improved caption quality by incorporating external knowledge and similar examples into language model-driven caption generators. However, these methods still encounter challenges when applied to real-world scenarios. First, many existing approaches rely on bimodal retrieval datastores that require large amounts of labeled data and substantial manual effort to construct, making them costly and time-consuming. Moreover, they simply retrieve the nearest samples to the input query from datastores, which leads to high redundancy in the retrieved content and subsequently degrades the quality of the generated captions. In this paper, we introduce a novel RAG-IC approach named (CoDiRet), which integrates a text-only unimodal retrieval module with our unique cluster-based retrieval mechanism. This proposal simultaneously enhances the scalability of the datastore, promotes diversity in retrieved content, and improves robustness against out-of-domain inputs, which eventually facilitates real-world applications. Experimental results demonstrate that our method, despite being exclusively trained on the COCO benchmark dataset, achieves competitive performance on the in-domain benchmark and generalizes robustly across different domains without additional training.

## introduction retrieval-augmented generative image captioning (rag-ic) combines information retrieval with language model-based caption generation @xcite @xcite to leverage external knowledge or contextually relevant information to the input image and produce more accurate and informative image descriptions. this technology mitigates overdependence on the internal knowledge encoded in ➢ a woman in black dress looking at cellphone on sidewalk ➢ two people on a city street with a cell phone ➢ a man looks at his phone as a woman stands nearby ➢ a man talking on a cellphone on the sidewalk ## methodology fig. 2 presents the overall architecture of our proposed codiret framework, which is built upon two primary strategies: a cross-modal alignment strategy and a cluster-based retrieval strategy. hereafter, we will present formal notations of variables and task definitions related to rag-ic in sec.3.1, and introduce each component subsequently in detail. ## conclusion we addressed several fundamental problems concerning rag-ic and proposed a joint learning framework called codiret, which trains a retriever by leveraging contrastive learning and clustering techniques to enhance cross-modal retrieval. the proposed model facilitates more semantically relevant retrieval results by minimizing the modality gap between image and text representations. meanwhile, by incorporating a cluster constraint, the model effectively reduces redundancy in the retrieved content, ensuring better adaptation to out-ofdomain scenarios. experimental results, including those of the analysis of retrieved contents, demonstrated the effectiveness of codiret.