M ulti C ap CLIP : Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning (2023)

Abstract: Supervised visual captioning models typically require a large scale of images or videos paired with descriptions in a specific language (i.e., the vision-caption pairs) for training. However, collecting and labeling large-scale datasets is time-consuming and expensive for many scenarios and languages. Therefore, sufficient labeled pairs are usually not available. To deal with the label shortage problem, we present a simple yet effective zero-shot approach MultiCapCLIP that can generate visual captions for different scenarios and languages without any labeled vision-caption pairs of downstream datasets. In the training stage, MultiCapCLIP only requires text data for input. Then it conducts two main steps: 1) retrieving concept prompts that preserve the corresponding domain knowledge of new scenarios; 2) auto-encoding the prompts to learn writing styles to output captions in a desired language. In the testing stage, MultiCapCLIP instead takes visual data as input directly to retrieve the concept prompts to generate the final visual descriptions. The extensive experiments on image and video captioning across four benchmarks and four languages (i.e., English, Chinese, German, and French) confirm the effectiveness of our approach. Compared with state-of-the-art zero-shot and weakly-supervised methods, our method achieves 4.8% and 21.5% absolute improvements in terms of BLEU@4 and CIDEr metrics. Our code is available at https://github.com/yangbang18/MultiCapCLIP .

## introduction visual captioning targets to first 1) understand the information of visual inputs, which are typically videos or images, and then 2) produces a corresponding textual sentence describing the visual objects/attributes/relationships. visual captioning has drawn remarkable attention from natural language processing and computer vision fields due to its wide applications, e.g., cross-modal retrieval @xcite and helping the visually impaired @xcite . currently, visual captioning models based on the encoderdecoder framework @xcite @xcite @xcite have achieved tremendous progress in advancing the state-of-the-art. these models are usually trained with full supervision and rely on large-scale humanly-annotated training data (i.e., vision-caption pairs), which needs expensive labeling work. in particular, when it comes to non-english caption systems, it is challenging to collect and label sufficient visioncaption pairs in a timely manner, which prevents such encoder-decoder models from rapid deployment in different scenarios and languages. to deal with the shortage of labeled pairs, we propose the multicapclip -a prompt-based natural language auto-encoder. as shown in figure 1 , multicapclip only requires textual input for training, and it can conduct zero-shot multilingual visual captioning, including image and video captioning. therefore, multicapclip can deal with the situation where the labeled vision-caption pairs are missing. multicapclip is particularly suitable for new scenarios and languages, improving the practical value of visual captioning. to implement multicapclip, we first adopt a pre-trained vision-language model, i.e., clip @xcite , as our backbone. clip has shown success in correlating the visual and textual modalities into the same latent space (vision-language embedding space) @xcite @xcite . we observe two critical issues for zero-shot visual captioning: the understanding of domain visual knowledge (e.g., objects, attributes, and relationships) and the generation of descriptive sentences in a specific writing style and language. therefore, we propose a prompt-based auto-encoder, which introduces the visual concept prompts p to preserve the corresponding domain knowledge and writing styles of zero-shot visual captioning. during training, given the text-only data, we train the model by reconstructing the caption s in the @xmath0. since the auto-encoding process reconstructs the same input sentence, the model training needs only unlabeled text data. in the reconstruction process, the model is able to preserve the necessary domain knowledge and the writing styles of visual captioning @xcite . during inference, we can directly take the vision input v as queries to retrieve the domain knowledge preserved in the visual concept prompts and finally rely on the learned writing styles in a specific language in the text decoder to generate visual descriptions in the @xmath1. meanwhile, to further bridge the modality gap between the visual and textual data @xcite , we introduce an augmentation method, including input augmentation and feature augmentation, which can boost the robustness of the model and in turn improve the performance of zeroshot visual captioning. the experiments on four benchmark datasets, i.e., ms-coco @xcite , msr-vtt @xcite , vatex @xcite , and multi30k @xcite , show that our approach can accurately and dataefficiently generate visual captions in english, chinese, german, and french. overall, our main contributions are as follows: ## approach in this section, we first give a brief review of clip, whose vision-language embedding space lays a foundation for our approach. next, we introduce the framework of the proposed multicapclip, followed by two key components: concept prompts and textual augmentations. ## main experiments in this section, we first introduce the datasets, metrics, settings of the experiments; then, we provide the out-of-domain and in-domain results of our approach for zero-shot visual captioning. ## analysis in this section, we conduct several analyses to better understand our approach. ## conclusions we have presented a data-efficient method dubbed multicapclip to re-purpose clip-like visionlanguage pre-trained models for zero-shot multilin-gual visual captioning. our approach reduces the reliance on labeled vision-caption pairs of downstream datasets by auto-encoding concept prompts on text-only data. extensive experiments on four datasets and four languages confirm the effectiveness of our approach, which can be a solid basis for visual captioning in low-data regimes and lowresource languages.