QACE : Asking Questions to Evaluate an Image Caption (2021)

Abstract: In this paper we propose QACE, a new metric based on Question Answering for Caption Evaluation to evaluate image captioning based on Question Generation(QG) and Question Answering(QA) systems. QACE generates questions on the evaluated caption and check its content by asking the questions on either the reference caption or the source image. We first develop QACE_Ref that compares the answers of the evaluated caption to its reference, and report competitive results with the state-of-the-art metrics. To go further, we propose QACE_Img, that asks the questions directly on the image, instead of reference. A Visual-QA system is necessary for QACE_Img. Unfortunately, the standard VQA models are actually framed a classification among only few thousands categories. Instead, we propose Visual-T5, an abstractive VQA system. The resulting metric, QACE_Img is multi-modal, reference-less and explainable. Our experiments show that QACE_Img compares favorably w.r.t. other reference-less metrics.

## introduction image captioning is a task that aims to generate a description containing the main content of a given image. the field of caption generation is prolific @xcite , and it is, therefore, important to provide reliable evaluation metrics to compare the systems. most of the prior works still report n-gram similarity metrics such as bleu @xcite or cider @xcite . however, these ngram similarity metrics often fail to capture the semantic errors in the generated captions @xcite . to overcome this limitation, we propose qace, a radically different evaluation framework from n-gram metrics. qace first generates questions about the candidate caption, and then checks if the answers are consistent w.r.t. either the reference or the source image. we depict qace in figure 1 . specifically, we propose two variants of qace, depending on what content the evaluated caption is compared to: qace ref when it is compared to the reference, and qace img when it is compared to the source image. qace img has the desired feature to be reference-less, i.e., the score can be computed without requiring a gold reference. in this reference-less setup, a visual question answering (vqa) system is required to answer those questions. however, in the vqa literature @xcite , the task is usually seen as a classification task on 3k pre-defined answer choices (e.g., blue, sea, or banana). therefore, these vqa models are not general qa systems; their usage off-the-shelf for qace img would limit the comparison to these very few pre-defined categories, which is not satisfying. to solve this issue, we also propose an abstractive vqa system visual-t5 as a new module for qace img that can generate freeform abstractive answers given a textual question and an image. we conduct a human evaluation of visual-t5 and show that it is capable of generating accurate abstractive answers. using visual-t5, we are now able to compare the answers of the candidate caption directly with the answers of the corresponding image. experimental results show that our proposed qace ref and qace img show promising results compared to other reference and referenceless metrics on three benchmark datasets: pas-cal50s @xcite , composite @xcite and flickr8k @xcite . also, as shown in figure 1 , qace has a natural form of interpretability through the visualization of the questions and the answers. image captioning metrics similar to other text generation tasks such as machine translation and reference 𝒙 a man in a wet suit riding a surfboard on a wave candidate 𝒙 a man riding a 1)wave on 2)top of a 3)surfboard q1: what is a man riding on top of a surfboard? q2: what part of a surfboard is a man riding a wave on? q3: what is a man riding a wave on top of? a1: wave a2: top a3: surfboard a1: wave a2: top a3: surfboard qace-img qace-ref a1: wave a2: wet suit a3: surfboard visual qa textual qa generating questions figure 1: the overall flow of qace. qace extracts possible answer spans and generates answer-aware questions for a given candidate caption x. the vqa and tqa answer these questions given the image and reference captions, respectively. the correctness of the candidate caption is evaluated by comparing the answers. summarization, n-gram similarity metrics such as bleu, meteor (banerjee and lavie, 2005) and rouge (lin, 2004) are arguably the standard in automatic evaluation. among them, the most widely used metric is cider (vedantam et al., 2015) which uses tf-idf based weighted n-gram similarity. spice (anderson et al., 2016) metric is based on scene graph, while more recently, bertscore (zhang et al., 2019) compute the similarity of the contextualized embeddings. different from prior works, we are the first to use question generation (qg) and question answering (qa) to evaluate the image captions. question and answering for evaluation fisch et al. (2020) proposes a new method to generate informal captions that can answer the visual questions. in our work, we focus on caption evaluation using the qa systems, not on generating the captions. several qa-based evaluation metrics (scialom et al., 2019; wang et al., 2020) are recently proposed to evaluate abstractive summarization. however, all those prior works are limited to text-to-text evaluation, while our work develops a multi-modal metric. ## qace we propose qace, which is a qg-and qa-based framework for evaluating an image caption. as shown in figure 1 , qace first extracts answer candidates (i.e., 1) wave, 2) top, 3) surfboard) from a candidate caption and generates corresponding questions. with these questions, visual-qa (vqa) and textual-qa (tqa) models answers given their context (i.e., image and reference x). by comparing the answers from each source, we can directly judge the correctness of the candidate caption. ## synthetic data generation for vqa as discussed in 3.3, relying on a vqa dataset such as vqa v2 @xcite limits possible answers to a small size of pre-defined categories. to train a general and abstractive vqa model, we create synthetic abstractive vqa datasets. we generate questions/answers pairs using the captions in the training set of ms-coco @xcite . specifically, we extract noun phrases from a reference caption and generate an answer-aware question using our qg model. to increase the validity of these synthetic questions, we apply the round trip consistency @xcite , filtering out the questions for which the qa model predicts a different answer than the extracted noun phrase. we convert these synthetic qa dataset to create {question, answer, image} triples by concatenating the corresponding images to these captions. in addition, we randomly add 20% of unanswerable questions foot_1 to the synthetic training set, so that the model learns to judge the answerability of a given question. through this, if a candidate caption contains any hallucinating content that is not included in the image, questions about it can be marked as unanswerable by our vqa model, as shown in the second example of figure 3 . this synthetic dataset enables the training of the abstractive vqa model. we report the performance of the model through a human evaluation in section 5.2. ## conclusion in this paper, we propose qace, a captioning metric that directly compares each content in the candidate caption with either the source image or a gold reference caption by asking questions. to enable asking questions directly on the source image, we introduce visual t5, an abstractive vqa model to generate free-form visual answers, for which we report strong results based on a human evaluation. our proposed metric can be applied in both reference and reference-less settings. it holds high explainability and compares favorably to the state-of-the-art in terms of correlations with human judgments.