V i LBERTS core: Evaluating Image Caption Using Vision-and-Language BERT (2020)

Abstract: In this paper, we propose an evaluation metric for image captioning systems using both image and text information. Unlike the previous methods that rely on textual representations in evaluating the caption, our approach uses visiolinguistic representations. The proposed method generates image-conditioned embeddings for each token using ViLBERT from both generated and reference texts. Then, these contextual embeddings from each of the two sentence-pair are compared to compute the similarity score. Experimental results on three benchmark datasets show that our method correlates significantly better with human judgments than all existing metrics.

## introduction image captioning is a task that aims to generate a text that describes a given image. while there have been many advances for caption generation algorithms @xcite and target datasets @xcite , few studies have focused on assessing the quality of the generated captions with consideration to the image. most of the previous studies on evaluating image captioning tasks rely on n-gram similarity metrics such as bleu @xcite or cider @xcite . these approaches bear limitations in dealing with the text's diverse nature, similarly found in other text generation tasks (e.g., abstractive summarization and dialog) @xcite . to alleviate the issues in the n-gram based approaches, researchers proposed word embedding-based techniques @xcite @xcite @xcite . these techniques shows robust performance and achieve higher correlation with human judgment than that of other previous metrics in many text generation tasks, including image captioning. especially, bertscore @xcite shows that using contextualized embedding is effective for evaluating the text. as bertscore does not utilize image content, it is still undiscovered how to effectively utilize the image content in the process of evaluating the captions. to further reflect image context while utilizing the advantages of bertscore, we propose vil-bertscore foot_0 by employing the vilbert @xcite , which is a task-agnostic pre-trained visiolinguistic representation. vilbertscore computes cosine similarity between token embeddings for reference and candidate sentences similar to bertscore. however, different from bertscore, the token embedding is computed with the consideration of image contexts. we evaluate our proposed method on three benchmark datasets (i.e., composite, flickr8k, and pascal-50s). extensive experiments show that vilbertscore achieves a significantly higher correlation with human judgments than previous metrics. this result demonstrates that the use of contextualized embedding from vision and language is we compute contextual embeddings with vilbert for x and x respectively. then, we extract the text embeddings h x v and h xv for each output embedding. finally, we compute the pairwise cosine similarity between h x v and h xv as in @xcite . ## vilbertscore we propose vilbertscore, a metric that utilizes visually-grounded representations for each token. the overall flow of our proposed vilbertscore is described in fig. 2 . similar to bertscore, we first compute contextual embeddings of both reference caption @xmath0, ..., x n ) and candidate caption @xmath1, ..., xm ). since we use vilbert, we compute the embeddings for each caption conditioning with the target image i. for the target image, we extract n region-level features @xmath2, ..., v n ) using pre-trained object detection model (see 4.2 for detailed information). then, we feed each pair of image and caption embeddings (x, v ), ( x, v ) to pre-trained vilbert and compute the contextual embeddings (h v x , h x v ) and (h v x , h xv ). note that h v and h x are image and text embeddings, respectively. among these embeddings, we only utilize the text embeddings, h x @xmath3, ..., h wt ) and h @xmath4, ..., ĥwt ), and compute cosine similarity among the pair of tokens from the candidate and reference caption. finally, the greedy matching process is exercised to the pair of tokens mentioned above for finding the most similar tokenmatch between two sentences. we can formulate vilbertscore as follows. vilbertscore each candidate caption and image pair. the images in this dataset are from flickr8k @xcite , flickr30k @xcite , and coco captions @xcite . the human judgments scores range from 1 to 5, depending on the relevance between candidate caption and image. ## conclusion in this paper, we propose vilbertscore, a metric for image captioning task by using pre-trained visio-linguistic representations. different from the bertscore, vilbertscore utilizes image conditional embeddings for each token which is critical in evaluating vision-language combined task. empirical results on composite, flickr8k, and pascal-50s datasets show that the proposed vil-bertscore correlates better with human judgments than all of the previous metrics.