I nfo M et IC : An Informative Metric for Reference-free Image Caption Evaluation (2023)

Abstract: Automatic image captioning evaluation is critical for benchmarking and promoting advances in image captioning research. Existing metrics only provide a single score to measure caption qualities, which are less explainable and informative. Instead, we humans can easily identify the problems of captions in details, e.g., which words are inaccurate and which salient objects are not described, and then rate the caption quality. To support such informative feedback, we propose an Informative Metric for Reference-free Image Caption evaluation (InfoMetIC). Given an image and a caption, InfoMetIC is able to report incorrect words and unmentioned image regions at fine-grained level, and also provide a text precision score, a vision recall score and an overall quality score at coarse-grained level. The coarse-grained score of InfoMetIC achieves significantly better correlation with human judgements than existing metrics on multiple benchmarks. We also construct a token-level evaluation dataset and demonstrate the effectiveness of InfoMetIC in fine-grained evaluation. Our code and datasets are publicly available at https://github.com/HAWLYQ/InfoMetIC .

## introduction image captioning aims to automatically generate natural language sentences to describe image contents. recently, there are significant breakthroughs in image captioning such as attentionbased model architectures @xcite @xcite and visionand-language pretraining (vlp) @xcite @xcite @xcite . however, as groundtruth image descriptions are extremely diverse and subjective, evaluating the image captioning performance remains a considerable challenge. the most widely used image captioning metrics such as meteor @xcite , cider @xcite and spice @xcite utilize human-written descriptions of images as references and measure similarities between generated captions and references for evaluation. such reference-based approaches suffer from two major limitations. firstly, these metrics mainly evaluate caption quality by n-gram overlaps which fail to measure genuine semantic similarities. secondly, references require time-consuming annotations and thus there are only a few annotated captions (typically 5) for each image. the limited number of references cannot fully capture image contents, resulting in incorrect penalties when generated captions describe correct novel things that are not mentioned in the references. to alleviate the above limitations, recent works are more focusing on reference-free metrics, which directly use images instead of reference captions in evaluation. benefited from the success of vlp on large-scale web data, umic @xcite and clip-s @xcite leverage vlp models uniter @xcite and clip @xcite respectively to calculate relevance scores between generated captions and images. although they have achieved promising correlations with human judgments, they can only produce an overall score as quality measurement. we humans instead tend to evaluate captions considering two aspects: 1) whether the caption correctly describes the image content (named text precision); and 2) whether the image content is comprehensively described in the caption (named vision recall). for example, as shown figure 1 , we can easily tell the "hat" in the second candidate is incorrect, and some salient contents such as "the bag" are not mentioned, and thus form our final evaluation to the caption. for the purpose of providing explainable and detailed feedbacks, we propose a informative metric for reference-free image caption evaluation (in-fometic). it is built on top of pretrained vlp models to measure fine-grained cross-modal similarities. infometic is able to point out incorrect semantic words in the caption and unmentioned regions in the image. based on fine-grained evaluation, it derives text precision and vision recall scores to measure captioning accuracy and completeness respectively. we take a summation of the two scores to rate overall quality of the caption. our contributions in this work are three-fold: ## related work reference-only caption evaluation. this type of evaluation only employs human-written captions as references and measures text similarity as the evaluation score. most widely used metrics such as bleu-4 @xcite , rouge-l @xcite , meteor @xcite , cider @xcite and spice @xcite all fall into this category. bleu-4 calculates the precision of n-gram matches; rouge-l measures the recall of the longest common subsequence; meteor utilizes wordnet-based synonym matching to relieve the shortage of exact word matching; cider introduces tf-idf to re-weight the importance of different n-grams; spice converts captions into scene graphs for similarity comparison. one major limitation of the above metrics is that they cannot properly count synonym matches. to overcome this deficiency, bert-s @xcite leverages learned embeddings from a pretrained language model bert @xcite to better measure semantic similarities. bert-s++ @xcite further improves bert-s by taking into account the variance of multiple references. ## method we first introduce our model architecture in sec 3.1 and then describe the training and inference ap- clip text encoder … [s] a man riding on the a motorcycle. [e] [s]a[e] [s]man[e] [s].[e] … … … … vision intra-modal encoder text intra-modal encoder inter-modal encoder vision recall score text precision score … … clip vision encoder 𝑡 ! 𝑣 ! t ! # 𝑣 ! 𝑣 "#$@xmath0 … 𝛼 " % 𝛼 # % 𝛼 @xmath1 text-filtered vision encoder proaches in sec 3.2 and sec 3.3 respectively.