Improving Image Captioning Evaluation by Considering Inter References Variance (2020)

Abstract: Evaluating image captions is very challenging partially due to the fact that there are multiple correct captions for every single image. Most of the existing one-to-one metrics operate by penalizing mismatches between reference and generative caption without considering the intrinsic variance between ground truth captions. It usually leads to over-penalization and thus a bad correlation to human judgment. Recently, the latest one-to-one metric BERTScore can achieve high human correlation in system-level tasks while some issues can be fixed for better performance. In this paper, we propose a novel metric based on BERTScore that could handle such a challenge and extend BERTScore with a few new features appropriately for image captioning evaluation. The experimental results show that our metric achieves state-of-the-art human judgment correlation.

## introduction image captioning is one of the key visual-linguistic tasks that asks for generated captions with specific images. researchers look forward to inexpensive evaluation metrics that closely resemble human judgment, which remains a challenging task since most of the metrics can hardly get close to human judgment. image captioning is a one-to-many task since each image can correspond to many possible captions. different captions may focus on different parts of the image; this not only creates a challenge for generating the captions @xcite , but also for evaluating them. most of the existing one-to-one evaluation metrics, however, overlook such a challenge. these oneto-one metrics @xcite @xcite ignore other reference captions since the score is computed by comparing the candidate capture with one single reference caption. figure 1 : intrinsic variance exists in a set of ground truth captions for an image. differences between two references are commonly caused by two reasons: different concerns or different descriptions. different concerns mean different expressions between references are caused by different regions of interest in an image, while different descriptions mean references focus on the same part but use different ways to explain it. oneto-one metrics can hardly deal with the cases caused by different concerns. for example, they may regard cand as a good caption compared with ref1; while regard cand as a bad caption compared with ref2 or ref3. when there are multiple reference captions, prior works compute individual scores for each reference caption and pool these scores together afterward. intrinsic variance exists in a set of ground truth captions for an image, since different captions may have different concerns or descriptions. it's challenging to find a remedy for such over-penalization if the metric looks at only one single reference caption. bertscore @xcite is the latest one-to-one metric that computes token-level cosine similarity between two sentences by contextual embeddings of pre-trained models, and greedily picks and adds up cosine values as a score. it reaches high performance in machine translation tasks and a system-level image captioning evaluation task. in one-to-one evaluation, although it is hard to consider all references directly, it is possible to combine references into a single one using contextual embedding from the pre-trained language model. in this work, we propose a metric where all of the references are combined as a new comprehensive embedding by detecting the mismatches between two contextual embeddings. to achieve this goal, we add the concept of mismatch into cosine similarity by a threshold for mismatch detection and proper penalization. also, our metric considers the importance of different words, and our research shows that adding a stop word list is an efficient way. using various image captioning evaluation datasets with human annotations like microsoft coco @xcite , flickr8k @xcite , composite @xcite and pascal-50s @xcite , the experimental results show that our metric achieves stateof-the-art correlation in several tasks, especially in caption-level tasks. our main contribution is a novel metric that can detect mismatches among captions, build a combined caption with multireferences, and achieve high human correlation in image captioning evaluation tasks. the code for our metric is released at here foot_0 . ## method proper scoring for generated captions should consider the information about multi-references and avoid the wrong penalization. in this section, we provide the idea about references combination and fix some under or over penalization issues for cosine similarity-based metrics. ## experiments the most convincing way for metric evaluation is the human correlation in caption-level and systemlevel tasks. in this section, we evaluate our metric in four typical image captioning evaluation datasets with standard metrics. we also consider the impact of each part in our metric by ablation experiment and key part replacements. ## more model results on our metric in ## conclusion in this work, we study the intrinsic variance among ground truth captions in image captioning evaluation. we propose an improved matching metrics based on bertscore, which can combine all of the references for taking full advantage of multireferences. our metric also benefits from stop word removal by reducing the impact of stop words. the experimental results show that our metric can reach state-of-the-art human correlation in several evaluation tasks.