IFC ap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning (2024)

Abstract: Recent advancements in image captioning have explored text-only training methods to overcome the limitations of paired image-text data. However, existing text-only training methods often overlook the modality gap between using text data during training and employing images during inference. To address this issue, we propose a novel approach called Image-like Retrieval, which aligns text features with visually relevant features to mitigate the modality gap. Our method further enhances the accuracy of generated captions by designing a fusion module that integrates retrieved captions with input features. Additionally, we introduce a Frequency-based Entity Filtering technique that significantly improves caption quality. We integrate these methods into a unified framework, which we refer to as IFCap (**I**mage-like Retrieval and **F**requency-based Entity Filtering for Zero-shot **Cap**tioning). Through extensive experimentation, our straightforward yet powerful approach has demonstrated its efficacy, outperforming the state-of-the-art methods by a significant margin in both image captioning and video captioning compared to zero-shot captioning based on text-only training.

## introduction the task of image captioning generates appropriate textual descriptions for images by combining computer vision (cv) and natural language processing (nlp). with the emergence of large language models (llms) and vision and language models (vlms), various works have studied efficient training methods for image captioning @xcite @xcite . these approaches develop effective captioning by using pre-trained models with few parameters or lightweight networks. however, these works rely on paired image-text data, which is costly @xcite . to overcome this, recent studies have explored text-only training methods for image captioning, aiming to solve the problem using only textual data @xcite @xcite @xcite @xcite . text-only training introduces a new direction in which models are trained solely using text data. recent existing works have studied what to use as extra cues, such as extracted nouns @xcite , generated synthetic images @xcite for training, and extracted tags from object detectors @xcite . however, existing methods that rely on object information are sensitive to incorrect data, and utilizing large external models (e.g., stable diffusion @xcite or object detectors @xcite incurs additional costs. thus, we aim to address the problem by acquiring diverse information costeffectively without additional models. the retrieval task involves finding relevant information in a database for a given query. initially rooted in nlp @xcite , the field has expanded into cv and into multi-modal retrieval. depending on the input data and database, various retrieval methods are possible, such as image-totext @xcite ) and text-to-text retrieval @xcite . in the existing text-only training study, there have been attempts to use the textto-text retrieval method. however, existing works can't address the modality gap inherent in text-only training settings, where training is performed with text and inference with images. in addition, such works rely too much on retrieved captions without considering visual information. this modality gap and the use of a narrow scope of information may lead to performance degradation. to verify this, we visualize the analysis result of the clip embedding feature of retrieved captions that the model uses in training via t-sne in fig. 2 . the analysis is done on the coco @xcite ., ## methods we propose a new text-only image captioning model, ifcap, which is illustrated in fig. 4 . during training, the model only utilizes text data, as is standard for text-only training models. first, we embed the input text using a text encoder. the text embeddings are then fed into a mapping network to close the gap between different modalities. finally, the processed embeddings go through a caption decoder to generate the output caption. our ifcap utilizes a simple yet powerful retrieval mechanism and addresses the modality gap between image and text with image-like retrieval (section 3.1). after performing image-like retrieval, we employ a fusion module (section 3.2) to merge input embeddings with the retrieved features. during inference, we use the retrieved captions from the image to find accurate and detailed entities with frequency-based entity filtering (section 3.3). ## conclusion in this paper, we propose a zero-shot captioning method, ifcap, through text-only training. ifcap performs image-like retrieval to address the gap between image-to-text retrieval and textto-text retrieval, fusion module for interaction between existing and additional representations, and frequency-based entity filtering during inference time to extract frequently occurring entities from the retrieved sentences. our method can be easily applied to various tasks and provides valuable guidance for retrieval-based methods in a text-only setting. it offers clear and precise information to llms without relying on a limited vocabulary. the simplicity and robustness of ifcap are demonstrated through state-of-the-art performance across various datasets in image and video captioning. the future direction of our method includes the extension of our method on more complex datasets, such as region-based captioning @xcite or visual question answering @xcite , which suffer from data issues.