UMIC : An Unreferenced Metric for Image Captioning via Contrastive Learning (2021)

Abstract: Despite the success of various text generation metrics such as BERTScore, it is still difficult to evaluate the image captions without enough reference captions due to the diversity of the descriptions. In this paper, we introduce a new metric UMIC, an Unreferenced Metric for Image Captioning which does not require reference captions to evaluate image captions. Based on Vision-and-Language BERT, we train UMIC to discriminate negative captions via contrastive learning. Also, we observe critical problems of the previous benchmark dataset (i.e., human annotations) on image captioning metric, and introduce a new collection of human annotations on the generated captions. We validate UMIC on four datasets, including our new dataset, and show that UMIC has a higher correlation than all previous metrics that require multiple references.

## introduction image captioning is a task that aims to generate a description that explains the given image in a natural language. while there have been many advances for caption generation algorithms @xcite and target datasets @xcite , few studies @xcite @xcite have focused on assessing the quality of the generated captions. especially, most of the evaluation metrics only use reference captions to evaluate the caption although the main context is an image. however, as shown in figure 1 , since there are many possible reference captions for a single image, a candidate caption can receive completely different scores depending on the type of reference (yi figure 1 : an example where the metric score for a given candidate caption varies significantly depending on the reference type. et al., 2020). because of this diverse nature of image captions, reference-based metrics usually use multiple references which are difficult to obtain. to overcome this limitation, we propose umic, an unreference metric for image captioning, which is not dependent on the reference captions and use an image-caption pair to evaluate a caption. we develop umic upon uniter @xcite which is a state-of-the-arts pre-trained representation for vision-and-language tasks. since uniter is pre-trained to predict the alignment for large amounts of image-text pairs, we consider that uniter can be a strong baseline for developing an unreferenced metric. we fine-tune uniter via contrastive learning, where the model is trained to compare and discriminate the ground-truth captions and diverse synthetic negative samples. we carefully prepare the negative samples that can represent most of the undesirable cases in captioning, such as grammatically incorrect, irrelevant to the image, or relevant but have wrong keyword. when evaluating the metric's performance, it is required to compare the correlations between human judgments and the metric's evaluation score for given datasets. we choose three standard benchmark datasets (i.e., @xcite , flickr8k @xcite , pascal-50s @xcite ) and further analyze the quality of the dataset. interestingly, we found that there exist critical issues in the benchmark datasets, such as poor-label or polarized-label. to perform a rigorous evaluation as well as stimulate the research in this area, we collect new 1,000 human judgments for the model-generated caption. finally, we evaluate our proposed metric on four benchmark datasets, including our new dataset. experimental results show that our proposed unreferenced metric is highly correlated with human judgments than all of the previous metrics that use reference captions. ## related work image captioning metrics following other text generation tasks such as dialogue systems and machine translation, n-gram similarity metrics such as bleu @xcite , rouge @xcite and meteor @xcite are widely used to evaluate an image caption. especially, cider @xcite , which weights each n-gram using tf-idf, is widely used. spice @xcite is a captioning metric based on scene graph. bertscore @xcite , which computes the similarity of the contextualized embeddings, are also used. bert-tbr @xcite focuses on the variance in multiple hypothesis and vilbertscore (vbtscore) @xcite utilizes vilbert @xcite to improve bertscore. different from these metrics, vifidel (madhyastha et al., 2019) computes the word mover distance @xcite between the object labels in the image and the candidate captions, and it does not require reference captions. similar to vifidel, our proposed umic does not utilize the reference captions. however, umic directly uses image features and evaluates a caption in various perspectives compared to vifidel. quality estimation quality estimation (qe) is a task that estimates the quality of the generated text without using the human references and this task is same as developing an unreferenced metric. qe is widely established in machine translation (mt) tasks @xcite @xcite . recently, @xcite introduces a large scale human ratings on image-caption pairs for training qe models in image captioning tasks. our work also trains caption qe model, (i.e. unreferenced captioning metric) but we do not use human ratings to train the metric. instead, we create diverse synthetic negative samples and train the metric with these samples via ranking loss. figure 2 : overall training procedure of umic. given an image i, a positive caption x and a negative caption x, we compute the score of each image-caption pair s x and s x using uniter respectively. then, we fine-tune uniter using raking loss that s x is higher than s x. ## umic we propose umic, an unreferenced metric for image captioning using uniter. we construct negative captions using the reference captions through the pre-defined rules. then, we fine-tune uniter to distinguish the reference captions and these synthetic negative captions to develop umic. ## dataset we briefly explain the previous benchmark datasets for captioning metrics and analyze the problems for two of these datasets, flickr8k and composite. also, we introduce a new benchmark dataset to alleviate the addressed problems. ## conclusion in this paper, we propose umic, an unreferened metric that does not require any reference captions for image captioning task through contrastive learning in uniter. also, we propose a new benchmark dataset for image captioning that relieve the issues in previous datasets. experimental results on four benchmark datasets, including our new dataset, show that umic outperforms previous metrics.