Fine-grained Image Captioning with CLIP Reward (2022)

Abstract: Modern image captioning models are usually trained with text similarity objectives. However, since reference captions in public datasets often describe the most salient common objects, models trained with the text similarity objectives tend to ignore specific and detailed aspects of an image that distinguish it from others. Towards more descriptive and distinctive caption generation, we propose to use CLIP, a multimodal encoder trained on huge image-text pairs from the web, to calculate multi-modal similarity and use it as a reward function. We also propose a simple finetuning strategy of CLIP text encoder to improve grammar that does not require extra text annotation. This completely eliminates the need for reference captions during the reward computation. To comprehensively evaluate descriptive captions, we introduce FineCapEval, a new dataset for caption evaluation with fine-grained criteria: overall, background, object, relations. In our experiments on text-to-image retrieval and FineCapEval, the proposed CLIP-guided model generates more distinctive captions than the CIDEroptimized model. We also show that our unsupervised grammar finetuning of the CLIP text encoder alleviates the degeneration problem of the naive CLIP reward. Lastly, we show human analysis where the annotators strongly prefer CLIP reward to CIDEr and MLE objectives on diverse criteria.

## introduction describing an image with its detailed, distinguishing aspects is crucial for many applications, such as creating text keys for image search engine and accessibility for the visual impaired. the standard deep learning approaches train an imageconditioned language model by maximizing the textual similarity between generated and reference captions @xcite @xcite . however, the reference captions of public datasets often describe only the most salient objects in images. this makes models trained to maximize textual similarity with reference captions tend to generate less distinctive captions that ignore the fine detailed aspects of an image that distinguishes it from others. to alleviate the problem, we propose to use clip @xcite , a multi-modal encoder model trained on large image-text data (mostly english) collected from the web, by using its similarity scores @xcite as rewards (sec. 3.1). in addition, we propose a clip text encoder finetuning strategy with synthetic negative caption augmentation to improve the grammar of captioning model, without any extra text annotations (sec. 3.2). note that our approach completely eliminates the need for reference captions during reward computation. to comprehensively evaluate descriptive captions, we also introduce finecapeval, a new dataset that measures captioning in diverse aspects: overall, background, object, and relation between objects (sec. 4). in our experiments on ms coco @xcite ) dataset, we show that the captions from models trained with clip reward are more distinctive and contain more detailed information compared to the captions from cider @xcite optimized models. the clip-guided captions even achieve the higher text-to-image retrieval performance than reference captions that are originally paired with images. we also show that our text encoder finetuning significantly improves caption grammars by removing degeneration artifacts such as word repetition. in fine-grained caption evaluation with finecapeval and human analysis, we show our clip based rewards outperform text similarity objectives by a large margin on all categories. objectives for image captioning. standard deep learning based image captioning approaches train models with maximum likelihood estimation (mle) objective. @xcite point that mle suffers from exposure bias problem. 2 to tackle exposure bias, @xcite propose a curriculum learning strategy called scheduled sampling. @xcite propose to train models by directly maximizing the textual similarity between generated and reference captions with re-inforce @xcite . @xcite ; @xcite propose self-critical sequence training (scst) approach by normalizing rewards to stabilize its high variance. recent studies have observed that referencetrained captioning models often neglect important information from images @xcite . @xcite use an visual question answering model's accuracy as a reward, encouraging models to generate captions that in-clude information sufficient to answer a visual question. @xcite ; @xcite ; @xcite use image-text retrieval model's self-retrieval score as a reward and combine them with n-gram based metrics, encouraging captioning models to generate captions that are distinctive to each input image. note that these works require careful balancing between self-retrieval and text similarity objectives for stable training. in contrast, by finetuning clip text encoder (sec. 3.2), our approach removes the need of reference caption and text similarity metrics for reward computation. ## finecapeval: fine-grained caption evaluation dataset we introduce finecapeval, a new dataset for caption evaluation in four different aspects. to construct finecapeval, we collect 500 images from the ms coco @xcite test2015 split and conceptual caption @xcite val split, respectively. then, for each image, we ask 5 human annotators to write phrases of 1) background, 2) objects (and their attributes; i.e., color, shape, etc.), 3) relation between objects (i.e., spatial relation), and 4) a detailed caption that includes all three aspects. see details of data collection process in appendix. in total, finecapeval consists of 1,000 images with 5,000 annotations for each of the 4 criteria. in ## experiments we train clip-res50 transformer captioning model @xcite with different rewards: mle, cider, clip-s, cider+clip-s, clip-s+grammar. following previous works, we conduct experiment on ms coco @xcite english captioning dataset with karpathy split @xcite . we evaluate the model with n-gram based metrics, embedding based metrics, text-to-image retrieval scores, and finecapeval. we also conduct human evaluation with five criteria to understand the human preference of the generated captions in diverse aspects. model architecture and training. we use the clip-res50 transformer @xcite as our captioning model architecture. the model consists of clip-res50 for visual feature extraction and a transformer encoder-decoder for conditional language model. we resize images in 224x224 to extract 2048-dimensional visual features. the transformer consists of 6-layer encoder and 6-layer decoder. we train our the model with mle objective for 15 epochs and further train with different rewards for 25 epochs (total 40 epochs), which takes within 1 day with 8 v100 gpus. we use beam size 5 for beam search decoding. we implement a training pipeline with pytorch @xcite , pytorch ligthning foot_2 , and hugging-face transformers @xcite . ## conclusion and future directions we introduce a novel training strategy for image captioning models by maximizing multimodal similarity score of clip and finetuning its text encoder to improve grammar. the use of clip reward eliminates the need for reference captions and their bias for reward computation. we also introduce finecapeval, a dataset for fine-grained caption evaluation. we demonstrate the effectiveness of our proposed method based on improvements in text-to-image retrieval, finecapeval, and human evaluation on fine-grained criteria along with quali-tative examples. future works involve finetuning clip reward models with desired writing styles for different applications and improving the synthetic augmentation process by using external data suitable for grammars with advanced linguistics expertise.