CLIPS core: A Reference-free Evaluation Metric for Image Captioning (2021)

Abstract: Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans. This is in contrast to the reference-free manner in which humans assess caption quality. In this paper, we report the surprising empirical finding that CLIP (Radford et al., 2021), a cross-modal model pretrained on 400M image+caption pairs from the web, can be used for robust automatic evaluation of image captioning without the need for references. Experiments spanning several corpora demonstrate that our new reference-free metric, CLIPScore, achieves the highest correlation with human judgements, outperforming existing reference-based metrics like CIDEr and SPICE. Information gain experiments demonstrate that CLIPScore, with its tight focus on image-text compatibility, is complementary to existing reference-based metrics that emphasize text-text similarities. Thus, we also present a reference-augmented version, RefCLIPScore, which achieves even higher correlation. Beyond literal description tasks, several case studies reveal domains where CLIPScore performs well (clip-art images, alt-text rating), but also where it is relatively weaker in comparison to reference-based metrics, e.g., news captions that require richer contextual knowledge.

## introduction for most text generation tasks, reference-based ngram overlap methods are still the dominant means of automatic evaluation. for image caption generation, recent reference-based metrics have sought to transcend overlap by considering richer models of reference-candidate similarity: e.g., approximate scene graphs @xcite , allowing reference-based methods to incorporate the image @xcite . but, references can be expensive to collect and comparing reference captions -two dogs are running towards each other across the sand. a recent trend in machine translation serves as inspiration: there, a key hurdle for reference-free evaluation (sometimes called quality estimation) has been estimating cross-lingual similarity between source+candidate pairs @xcite @xcite . but recent work @xcite @xcite has improved correlation with human judgment not by gathering more monolingual references, but instead by utilizing cross-lingual representations learned by large-scale, pre-trained, multilingual models e.g., laser @xcite or m-bert @xcite . 2 we hypothesize that the relationships learned by pretrained vision+language models (e.g., align @xcite and clip @xcite ) could similarly support reference-free evaluation in the image captioning case. indeed, they can: we show that a relatively direct application of clip to (image, generated caption) pairs results in surprisingly high correlation with human judgments on a suite of standard image description benchmarks (e.g., mscoco @xcite ). we call this process clipscore (abbreviated to clip-s). beyond direct correlation with human judgments, an information gain analysis reveals that clip-s is complementary both to commonly reported metrics (like bleu-4, spice, and cider) and to newly proposed reference-based metrics (e.g., vilbertscore-f @xcite ). we additionally (1) propose a referenceaugmented version of clipscore, refclipscore, that achieves even higher human correlation, (2) verify that clip-s is sensitive to adversarially constructed image captions, where one noun-phrase has been swapped for a plausible (but incorrect) distractor; and (3) construct a corpus of images that have never been posted publicly online to verify that clip-s is able to reconstruct human judgments on never-before-seen images. finally, we assess clip-s in the context of four case studies that diverge from context-free, literal photograph description. in two cases, clip-s works well: it achieves high correlation with alt-text quality rating on twitter, and demonstrates surprising capacity to reason about clipart images+captions. for news caption generation, reference-based meth-ods correlate best with human judgments. and, for emotive captions inspired by language use on social media, even reference-based metrics fall short. ## related work reference-only image caption evaluation in general, image caption generation models are evaluated by a suite of 5 reference based metrics: bleu-4 @xcite (which measures a version of precision between a candidate and the references), rouge-l @xcite (which measures a version of recall), meteor @xcite (which computes a word-level alignment), cider @xcite (which combines n-gram tf-idf weighting and stemming) and spice @xcite (which applies a semantic parser to a set of references, and computes similarity using the predicted scene graph). 3 yi et al. (2020) give a method for re-weighting bertscore @xcite specifically tuned to the image caption generation domain (we refer to their method as bert-s++). reference+image caption evaluation recent metrics incorporate image-text grounding features in addition to references: tiger @xcite uses a pretrained scan model @xcite , and vilbertscore-f @xcite uses a pretrained vilbert model @xcite that is also fine-tuned on 12 downstream vision and language tasks @xcite . our work provides perspective on the next logical extension: instead of incorporating visual-textual interactions in addition to references, can we ignore references entirely? self-retrieval for image captioning prior works have proposed incorporating a self-retrieval loss into caption generation, with the intuition that good captions should be able to uniquely identify their images with high accuracy @xcite @xcite ; monitoring this type of loss can provide insight into how distinctive the captions are according to the model itself. clip-s is similar in spirit, but distinct for its utility as an extrinsic evaluation metric like bleu-4 or cider. reference-free evaluation in addition to the machine translation cases highlighted in the introduction, reference-free evaluations have been proposed for other generation tasks, including summarization @xcite @xcite and dialogue @xcite . these metrics can be supervised, relying on human judgments for quality estimation, or less-supervised, relying on pre-trained model representations. for image captioning, a version of vifidel @xcite was proposed for reference-free evaluation; however, vifidel, computed based on a list of detected objects in the image from a fixed object vocabulary, generally produces less correlation with human ratings vs. reference-based metrics. the model we use is the vit-b/32 version. foot_2 it represents images via a vision transformer @xcite , which forgoes convolutional filters in favor of selfattention maps computed between a 7 by 7 grid of image patches, which evenly divides a 224 by 224 pixel input image. this model has 12 transformer layers and 86m parameters. the text is similarly represented by a 12-layer transformer trained over a vocab of 49k bpe token types @xcite (and is more fully described in @xcite ). both the text and image networks output a single vector; these vectors aim to represent the content of an input caption or an image, respectively. in the case of vit-b/32, these vectors are 512-d. the model's weights are trained to maximize the scaled cosine similarity of truly corresponding image/caption pairs while simultaneously minimizing the similarity of mismatched image/caption pairs using infonce @xcite . we hold fixed this set of weights for our experiments. ## benchmark captioning evaluations we first evaluate on a set of literal description corpora. broadly, the captions in these corpora aim to identify and highlight the literal, salient objects/actions in a photographic image, presented without additional context. ## case studies using clipscore our results thus far have demonstrated that clip encodes information useful for evaluating literal image description tasks. but, reference-based metrics may a priori seem more adaptable versus clip-s. does clip-s correlate with human judgment beyond cases like mscoco and flickr8k? to address this question, we consider four case studies, exploring the correlation between clip-s and human judgment across "divergent" image description datasets. these corpora qualitatively differ from the more popular domains explored in §4, either because the images are not "everyday" images from flickr, or because the captions are not literal description (figure 3 illustrates). ## conclusion for literal image description tasks, clipscore achieves high correlation with human judgments of caption quality without references when used in an off-the-shelf fashion. additional experiments in divergent domains suggest that clip can also reason about non-photographic clip-art, and serves as a reasonable option for reference-free evaluation in the alt-text case. promising future work includes exploring 1) clip-s as a reinforcement learning reward for literal caption generators; and 2) whether a small amount of labelled human rating data could help clip-s adapt to domains where it struggles, e.g., engagingness prediction. we hope our work can contribute to the ongoing discussion about the role of pretrained models in generation evaluation. reference-free evaluation runs some risks. much like bertscore, model-based metrics like clip-s reflect the biases of the pre-training data. while we believe that using clip-s as an offline evaluation metric for literal caption quality accords with the recommendations of clip's model card foot_14 @xcite , @xcite 's study demonstrates that clip can make disproportionate incorrect classifications of people, e.g., "male images were misclassified into classes related to crime." exploring potential social biases of candidate generations (as in, e.g., hendricks et al. (2018)) remains paramount, particularly if a system is to be deployed. contemporaneous work while this work was under submission, two alternate reference-free evaluation metrics for image caption generation were introduced: faier @xcite (based on a pretrained object detector, and fine-tuned on mscoco) and umic @xcite (based on uniter @xcite ). umic, in particular, produces similar correlations with human judgment on the literal image description tasks ( §4) compared to clip-s, but with the complementary approach of fine-tuning on synthetic negative captions. future work would be well-suited to explore if the textual data augmentations proposed by @xcite (1) result in a metric that complements or overlaps with the non-finetuned clip-s ( §4.6); and (2) could be extended beyond cases of literal description ( §5). a evaluation and replication details @xcite introduced a set of corpora, metrics, and experimental settings for comparing image caption generation evaluation metrics. perhaps unwittingly, their introduced protocols have become the accepted standard for evaluation of new caption generation metrics. however, seemingly innocuous preprocessing+reporting choices can significantly impact correlations with human judgment on these corpora. in what follows, we detail our replication efforts. our goal was to make the experimental comparisons involving clipscore reported in the main paper as fair as possible. we hope it can be useful for researchers reporting metrics on this setup going forward.