Text-Only Training for Image Captioning using Noise-Injected CLIP (2022)

Abstract: We consider the task of image-captioning using only the CLIP model and additional text data at training time and no additional captioned images. Our approach relies on the fact that CLIP is trained to make visual and textual embeddings similar. Therefore, we only need to learn how to translate CLIP textual embeddings back into text, and we can learn how to do this by learning a decoder for the frozen CLIP text encoder using only text. We argue that this intuition is “almost correct” because of a gap between the embedding spaces, and propose to rectify this via noise injection during training. We demonstrate the effectiveness of our approach by showing SOTA zero-shot image captioning across four benchmarks, including style transfer. Code, data, and models are available at https://github.com/DavidHuji/CapDec.

## introduction vision and language are closely intertwined, as they are two ways of describing the world. this raises the potential for developing models that map images and text into a shared semantic space. indeed, this approach has recently achieved great success with models like clip @xcite and align @xcite . these models use parallel image-text data to train a joint representation, where the embeddings of image-text pairs are similar. such models have been employed for various vision-language tasks. image captioning is a key task in vision-language perception. yet, training image captioning models typically requires large datasets of captioned images, and these are challenging to collect. furthermore, it is not clear how one could adapt a pretrained vision-language model to generate captions in new styles. in this work, we present an approach to captioning that only requires clip and text data, and generates styled captions using only unpaired textual examples from that style. this alleviates the need for paired text-image data, and also allows for simple style transfer. a first approach one could consider for this setting is to train a decoder model to reconstruct texts from their respective clip embeddings, and at inference use this decoder to decode image embeddings. however, we observed that this approach fails at inference, and we conjecture this is due to the known domain gap between the image and text modalities @xcite . we propose a simple approach to mitigate this, by injecting noise into the embedding during training. this has the effect of creating a ball in embedding space that will map to the same caption, and corresponding image-embedding is more likely to be inside this ball, as illustrated in fig. 1 .a. we evaluate our "captioning via decoding" (capdec) method extensively, showing that it works well on several image captioning tasks, including standard, cross-domain, and style-guided captioning. overall, our main contributions are as follows: 1) a simple and intuitive approach to learning a captioning model based on clip and additional text training data, but no images for training. 2) evaluation of capdec on image captioning tasks, including generating captions in various styles, shows it outperforms other methods which use the same supervision. ## related work image captioning methods @xcite @xcite @xcite typically extract visual features using a pre-trained network. these are passed to a textual decoder that produces the final captions. to bridge the gap between vision and language, other works employ pre-training to create a shared latent space of vision and text @xcite @xcite @xcite @xcite ; text encoder image encoder text encoder ε embedding space text decoder text decoder people standing next to an elephant. an elephant is walking in the jungle. an elephant is walking in the jungle. ## results we next evaluate capdec on several captioning tasks, demonstrating state-of-the-art results. see supplementary for additional details. image captioning. we compare capdec caption quality to several baselines with different supervision levels, as presented in tab. 1(a). here, all methods were trained end evaluated over the same dataset, using the commonly used ms-coco @xcite and flickr30k @xcite . we begin by evaluating fully supervised techniques: butd @xcite , univlp @xcite , clip-cap @xcite , oscar @xcite , and lemon @xcite . as expected, these achieve a better score than capdec, as they exploit the additional supervision of image-text pairs. nevertheless, compared to the unsupervised approaches of magic @xcite and zero-cap @xcite , capdec achieves superior scores. note that zerocap does not require any training data, while magic requires text data similar to our setting. cross-domain captioning. we test our generalization ability by training on one dataset while evaluating on another, as in @xcite . again, as can be seen in tab. 1(b), capdec outperforms magic @xcite , which uses the same supervision as capdec. style-guided captioning. several works @xcite have studied the task of adapting a captioning model to a new style, such as "romantic" or "humorous". since collecting paired examples for each style requires great effort, these consider the setting where the new style is only learned from text. this is easy to do in our setting, since we can train the decoder on any given style text. to further analyze our approach, we present our results without pretraining (i.e., training on styled data only), with a text-only pre-training over coco, and with textimage pre-training over coco (similar to @xcite ). as can be seen, we outperform @xcite even with considerably less supervision at pre-training. moreover, both other variations improve results, demonstrating that capdec can effectively use additional training data where available. ## noise injection analysis noise-injection is a well-known technique for improving generalization @xcite @xcite , and can be viewed as a data augmentation mechanism @xcite . in our case, the use of noise was also meant to address the modality-gap observed in @xcite . in order to examine the specific effect of noise, we perform additional evaluations on coco and show the results in fig. 4 . ## conclusion the image captioning task has been extensively studied, with considerable progress in recent years. however, the number of available training datasets, containing image-text pairs, is still rather limited. consequently, image captioning models inherit the limitations of their training data, such as biases @xcite or confinement to neutral style @xcite . in this work, we suggest a new paradigm, where a generic vision-language model (e.g., clip) is adapted to image captioning using a text-only dataset. furthermore, we demonstrate a simple and intuitive technique to overcome the inherent domain gap of clip @xcite . for future work, we plan to study textonly training for other tasks, such as visual question answering and visual scene graph generation.