An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics (2024)

Abstract: Recently, reference-free metrics such as CLIPScore (Hessel et al., 2021), UMIC (Lee et al., 2021), and PAC-S (Sarto et al., 2023) have been proposed for automatic reference-free evaluation of image captions. Our focus lies in evaluating the robustness of these metrics in scenarios that require distinguishing between two captions with high lexical overlap but very different meanings. Our findings reveal that despite their high correlation with human judgments, CLIPScore, UMIC, and PAC-S struggle to identify fine-grained errors. While all metrics exhibit strong sensitivity to visual grounding errors, their sensitivity to caption implausibility errors is limited. Furthermore, we found that all metrics are sensitive to variations in the size of image-relevant objects mentioned in the caption, while CLIPScore and PAC-S are also sensitive to the number of mentions of image-relevant objects in the caption. Regarding linguistic aspects of a caption, all metrics show weak comprehension of negation, and CLIPScore and PAC-S are insensitive to the structure of the caption to a great extent. We hope our findings will guide further improvements in reference-free evaluation of image captioning.

## introduction image caption quality has been traditionally evaluated using a reference-based approach, with metrics like bleu @xcite , rouge @xcite , meteor @xcite , and cider @xcite assessing the lexical overlap between generated and reference captions. however, this approach is restrictive as the set of references may not capture the full range of valid captions, and furthermore, lexical overlapbased metrics tend to favor captions with similar vocabulary but different meanings. to address these limitations, recent studies like clipscore @xcite figure 1 : recently proposed reference-free image captioning evaluation metrics such as clipscore, umic, and pac-s are far from perfect. this figure shows how these metrics cannot tell apart an incorrect caption (shown in red) from a correct caption when there is a high lexical overlap between them. @xcite , umic @xcite and pac-s @xcite have proposed reference-free approaches for evaluating image caption quality, which more closely aligns with human judgments. these approaches leverage large pretrained imagetext matching models to measure the similarity between a given image and a candidate caption. however, the evaluation benchmarks for these metrics do not necessarily involve differentiating between captions with significant lexical overlap but vastly different meanings (fig. 1 ). in this work, we evaluate the robustness of these reference-free metrics in scenarios where the correct and incorrect captions have high lexical overlap. to our surprise, we found that all metrics fail to distinguish between correct and incorrect captions ∼46% of the time. in a pursuit to identify what aspects of a caption (e.g., plausibility, visual grounding, number and size of objects mentioned in the caption, negation and sentence structure) these metrics are most sensitive to, we conduct several controlled experiments, varying one aspect at a time. we found that: our primary contribution is highlighting specific areas where reference-free metrics exhibit limitations so that caution can be exercised when using these metrics for image captioning evaluation. we hope our findings will guide further improvements in reference-free evaluation of image captioning. ## experiments and results preliminary experiment: first, we describe our preliminary experiment that served as a motivation for the rest of the study. we were interested in examining how different the scores assigned by reference-free image captioning metrics are for correct/incorrect captions created by converting questions and correct/incorrect answers from the vqav2 dataset to caption-like sentences. captions generated in this way are unique in that even for incorrect captions, a significant portion of it (corresponding to the question part) is still correct. thus, such a dataset of captions serves as a good stress test dataset for examining the robustness of reference-free image captioning metrics. to obtain correct and incorrect answers, we obtained predictions from the albef @xcite visual question answering model on the validation splits of the vqav2 @xcite dataset. we fine-tuned albef on this dataset and conducted iid evaluation. we then converted each question and its corresponding albef answer into a caption-like sentence as described in section 3. we only use answers that match with either three or more human answers (and we classify them as correct answers) or that do not match with any human answers (and we classify them as incorrect answers), resulting in a total of 179,297 answers (43389 incorrect and 135908 correct). the histograms of results for the vqav2 dataset are presented in figure 3 . we see a significant overlap between the distributions of scores for correct and incorrect captions for all metrics, highlighting the ## conclusion and discussion in conclusion, recently proposed reference-free image captioning evaluation metrics are far from perfect; they cannot distinguish an incorrect caption from a correct caption when the difference between them is fine-grained. the sensitivity of clipscore, umic, and pac-s varies across different error types: they are less affected by plausibility errors yet more by visual grounding errors. all metrics struggle with understanding negation. all metrics are influenced by the size of the relevant objects mentioned in the caption, and clipscore and pac-s also responds to the number of object mentions. umic is responsive to sentence structure, while clipscore and pac-s disregards it often. moreover, umic prioritizes sentence structure over the number and size of objects mentioned in the caption; in contrast clipscore and pac-s prioritize the object size and number of object mentions over sentence structure. our primary contribution is highlighting specific areas where reference-free metrics exhibit limitations. the root cause of these limitations is traced to the insufficient fine-grained understanding of the clip and uniter models upon which these reference-free metrics rely. in order to improve the reference-free metrics, we believe that underlying models need to become better at fine-grained understanding of objects, attributes, relationships etc., so that they can better distinguish fine-grained differences between captions. promising avenues for enhancing this understanding include exploring object-centric representations @xcite @xcite and incorporating training with hard negatives @xcite @xcite , allowing the model to learn to discern fine-grained differences and errors. given the restricted fine-grained understanding of the underlying models shaping these metrics, caution is advised when employing them as evaluation metrics for image captioning.