Bridging Semantic and Modality Gaps in Zero-Shot Captioning via Retrieval from Synthetic Data (2025)

Abstract: Zero-shot image captioning, which aims to generate image descriptions without relying on annotated data, has recently attracted increasing research interest. Pre-trained text-to-image generation models enable the creation of synthetic pairs solely from text data, while existing methods fall short in mitigating the discrepancy caused by the inability of synthetic images to fully capture the semantics of the textual input, resulting in unreliable cross-modal correspondences. To address this, we propose a retrieval-based framework that leverages only existing synthetic image-text pairs as its search corpus to systematically bridge the gap when using synthetic data for captioning. For the semantic gap between a synthetic image and its input text, our framework retrieves supplementary visual features from similar synthetic examples and integrates them to refine the image embedding. Then, it extracts image-related textual descriptions to mitigate the modality gap during decoding. Moreover, we introduce a plug-and-play visual semantic module that detects visual entities, further facilitating the construction of semantic correspondences between images and text. Experimental results on benchmark datasets demonstrate that our method obtains state-of-the-art results.

## introduction supervised image captioning approaches rely on large-scale labeled image-text pairs, which are often costly to acquire @xcite . to address this data dependency limitation, zero-shot image captioning approaches have attracted growing research interest. previous methods @xcite ) adopt text-only training and subsequently perform inference using images. however, the mismatch between training and inference leads to a modality gap @xcite , which hinders the model's performance. most existing works use pre-trained multi-modal models @xcite to map texts and images into the same space, which mitigates the modality gap @xcite . since using text as a proxy for visual information cannot fully capture visual semantics, textonly methods, which cannot utilize images during training, still maintain a gap between modalities. to address the above problem, many recent studies have utilized image generative models to construct synthetic image-text pairs @xcite , using synthetic visual semantics for training. although these pre-trained generative models could produce rich visual information, they suffer from information loss during both the text-to-vector @xcite encoding process and the image generation stage @xcite due to inherent model limitations. this results in a semantic mismatch between the synthetic image and its corresponding text, commonly referred to as the semantic gap @xcite . some straightforward strategies (e.g., replacing images based on similarity) could alleviate the semantic gap @xcite , but existing works heavily rely on the capabilities of image generative models and directly utilize synthetic image-text pairs with semantic mismatches, as illustrated by figure 1(a ). as a result, they fail to handle the discrepancies arising from the inability of synthetic images to fully capture the semantics of textual inputs, resulting in unreliable cross-modal correspondences. this paper proposes a method, called roscap, which mitigates both the semantic and modality gaps in synthetic image-text pairs using existing information and reduces reliance on image generative models. since synthetic images are generated based on text, the corresponding original text preserves complete semantic information. we retrieve embeddings of synthetic images using the original text and construct a support domain to mitigate gaps in image-text pairs, thereby improving the se- ## methodology our proposed zero-shot image captioning method, roscap, is based on the retrieval of semantic information. the overall framework is illustrated in figure 2 . during the training process, we leverage only text and text-based synthetic images. first, in the selective projection optimization method (section 3.1), a support domain is constructed by retrieving synthetic image embeddings. the original synthetic image embeddings are then projected into this domain to achieve feature optimization. this process not only preserves the original information of the synthetic images but also introduces additional semantic information to help alleviate the semantic gap between the synthetic image-text pairs. these optimized synthetic image embeddings are used to retrieve the most relevant text embeddings from the text memory, which are then fed into the cross-modal fusion module (section 3.2) for feature fusion. it addresses the modality gap between image-text pairs and further bridges the semantic gap. during inference, to capture more accurate entity information, we employ the visual semantic module (section 3.3) to retrieve data simultaneously from both the synthetic image memory and the text memory, enabling the extraction of entity information from multiple modalities. ## conclusion we propose roscap, a retrieval-based zero-shot image captioning framework designed to bridge both the semantic and modality gaps between synthetic images and their corresponding texts while reducing reliance on image generative models. roscap employs a post-processing strategy that optimizes synthetic image embeddings and leverages cross-modal fusion to enhance image-text alignment. additionally, the visual semantic module enables the extraction of representative entities from multiple modalities, improving the generation of semantically consistent captions. extensive experiments on benchmark datasets demonstrate that roscap achieves state-of-the-art performance and exhibits a strong generalization capability.