Quality Estimation for Image Captions Based on Large-scale Human Evaluations (2021)

Abstract: Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild. In this paper, we focus on the task of Quality Estimation (QE) for image captions, which attempts to model the caption quality from a human perspective and *without* access to ground-truth references, so that it can be applied at prediction time to detect low-quality captions produced on *previously unseen images*. For this task, we develop a human evaluation process that collects coarse-grained caption annotations from crowdsourced users, which is then used to collect a large scale dataset spanning more than 600k caption quality ratings. We then carefully validate the quality of the collected ratings and establish baseline models for this new QE task. Finally, we further collect fine-grained caption quality annotations from trained raters, and use them to demonstrate that QE models trained over the coarse ratings can effectively detect and filter out low-quality image captions, thereby improving the user experience from captioning systems.

## introduction image captioning technology produces automatic image descriptions using natural language with the goal of being consumed by end-users that may not be able to directly access the images. this need arises either because the user has a permanent condition (accessibility for visually impaired people), or due to a temporary situation where the user cannot use the visual modality (such as limited bandwidth, or smart voice-assistant). in any of these situations, exposing the end-users to a generated caption that is incorrect negatively impacts usertrust, as it can have undesirable consequences for how they act next (for example, how they comment on a social-media site based on their misguided understanding). in this paper, we propose to mitigate such risks through quality estimation (qe) of image captions. that is, we propose to automatically compute a quality estimation score qe(image, caption) for a generated caption, and use it to control the quality of the captions presented to the user. for example, by filtering out captions with a low qe score (below a carefully chosen threshold), only high scoring captions would be served thereby minimizing the risks associated with low-quality captions. we emphasize two aspects of qe that have guided us in our design choices: first, the qe task is distinct from the model selection task: model selection measures output similarity to a fixed, ground-truth annotated dataset during training time (with traditional offline solutions such as cider and spice). in contrast, a qe model estimates the caption quality with respect to the input image only and does so on previously unseen samples at prediction time where ground-truth captions are unavailable. second, a qe model's goal is to assess the caption as a whole and relate it to the image content in a way that qe(image, caption) aligns with human understanding of language and their perception of visual information. to address these aspects we develop an imagecaption evaluation process for collecting vast amounts of human judgements. specifically, we design the process to elicit only the type of human signal that is required for quality estimation -human annotators are shown the image and asked to evaluate the caption as a whole by simply answering whether it is good or not. this type of high level feedback trades away the ability to understand in what way the caption is wrong, but its simplicity enables scaling up human evaluations to cover many more images, which promotes the generalization of the qe model to unseen images. the dataset resulting from the evaluation process includes captions generated by various imagecaptioning model over 16,000 unique images from the open image dataset @xcite for a total of 55,000 unique image, caption pairs, over which we collected approximately 600,000 binary human ratings. we denote this dataset as caption-quality, provide extensive details on its generation process as well as make it publicly available foot_0 , available. the following summarizes our contributions: 1. we release the caption-quality dataset of roughly 65k human rated image-caption pairs, obtained by collecting approximately 600k binary human ratings in total. by analyzing the collected ratings, we show that they encode a stable and consistent signal about the caption. 2. we establish baseline results on the qe task and demonstrate that the signal encoded in the collected ratings is learnable, yet, cannot be trivially captured by an image-text similairty model trained over a large scale imagecaptioning dataset. 3. we further test our qe models, trained over the caption-quality dataset, and show that they can successfully rank correct-and-helpful captions higher than incorrect or unhelpful ones, even though they were never exposed to such a fine-grained signal. this is done by collecting additional fine-grained caption annotations from trained human raters, over images that are out-of-domain for the qe model. ## related work our paper is most similar to work done on evaluation metrics of image captions, where the main difference is that qe does not have access to the ground truth captions. quality estimation has more than a decade long history in the machine translation (mt) field, from the early work based on feature engineering @xcite , to more recent neural-network-based approaches @xcite @xcite . the qe track at the wmt conference @xcite has been running for several years, with multiple participants and notable improvements in model performance over the years. however, there are significant differences in the formulation of the qe task between mt and image captioning, most notably the fact that the mt formulation is uni-modal (text-only alignment). as a result, solutions for qe in the mt context tend to focus on feature-engineering that exploits this aspect @xcite @xcite . in contrast, qe for image captioning is a bi-modal problem (image-andtext alignment), and therefore better suited to approaches based primarily on deep feature representations and multi-modal feature integration, as we present in this paper. beyond quality estimation modeling, the issue of effectively using quality estimators to improve the accessibility use-case for blind or visually impaired (bvi) people has been previously studied @xcite . the main question of their study is how to best inform the bvi user about the uncertainty around the generated captions, experimenting with framing the captions using phrases like "i'm not really sure but i think it's @xmath0caption". the findings are relevant in that bvi users of this technology have difficulties calibrating themselves into trusting or distrusting $caption, mostly because there is no alternative form of reference for the image content. therefore, if the caption provided to them (even accompanied by "i'm not really sure but ...") is in dissonance with the rest of the context (as it may be available in text form, e.g., as part of a tweet thread as in the study cited above), they tend to resolve this dissonance not by believing that the caption is wrong, but by constructing scenarios or explanations that would somehow connect the two sources of information. to mitigate this problem, we propose a thresholding-based approach that simply decides whether to show a caption or not based on a qe model's prediction (see section 6.2). ## building the caption-quality dataset the key contribution of this paper is the caption-quality dataset, a large collection of binary human judgments on the quality of machine-generated image captions (in english). below, we describe the dataset generation process, as well as the rating collection process with which we collect approximately 600,000 binary ratings via crowdsourcing. we then provide an analysis of the ratings which shows that they contain a consistent signal about the captions. note that in the experiments (section 6.2), we further verify that indeed this signal captures the quality of the caption as perceived by trained humans annotators. ## a fine-grained caption evaluation we further collect fine-grained human annotations of image-captions to ascertain that the signal in the caption-quality dataset is beneficial for estimating the quality of image captions and filtering out lowquality ones. specifically, we ask professional human annotators to evaluate image-captions across two specific dimensions: helpfulness and correctness foot_3 . fig 4 shows the evaluation interface. distinguishing between correctness and helpfulness is particularly crucial for quality estimation, as it helps diagnose models that produce abstract or irrelevant captions which, while correct, do not provide useful image descriptions (specifically, for a person who is unable to see the image). for example, consider the correct yet abstract caption "person in a sport event" compared to the more descriptive caption "ice hockey player celebrates his goal against sports team" (see fig 4 ). another example of a correct but unhelpful caption is "a view of the game from my living room" because it conveys more information about the camera position rather than the actual image content. while the previously discussed fast&simple evaluation may assign all these captions with similar scores, the fine-grained evaluation is capable of capturing such nuanced differences. we posit that the large-scale annotations obtained by the fast&simple approach will enable a model to distinguish between correct-and-helpful captions, and those that are not. we ran the finegrained evaluation once over 2,700 images, collecting 3 ratings per image. the resulting dataset, denoted caption-ext is used for our extrinsic qe evaluations (sec. 6.2). ## models this section presents a simple bilinear qe model which learns to combine the image and caption features to arrive at a quality estimate qe(image, caption). to construct the bilinear model we rely on expressive image and text representations that are produced by pretrained models that were themselves trained on vast amounts of uni-modal data. note that aside from building on top of pretrained models, we restrict further modeling to a simple architecture. this was done in order to establish a baseline for our new qe task, as well as to remain focused on providing evidence that the signal in the caption -quality dataset is both learnable and beneficial for quality estimation of image captions. ## experimental results all qe models are trained on the caption-quality training set (section 3). we use mean squared error (m @xmath1, where ŷj are the predicted scores and y j the ground-truth human scores. for optimization, we use adam (kingma and ba, 2015) with batch size @xmath2, 1e-5, 1e-6}. dropout rate is set to 0.2, and applied on the inputs of all trainable layers. the following pretrained models are fixed during optimization: the image encoder, the use caption encoder, and object-label encoder. the number of object-labels is tuned over {0, 5, 10, 20}, while the pretrained variants were fixed to 16. model selection is done by picking the checkpoint that maximizes the dev set spearman's correlation ρ s (y, ŷ). specifically, compared to mse (the objective), the spearman-based selection criterion better matches the intended use of the qe model, where at inference time, only images whose qe scores pass some threshold will be served. since this threshold can be tuned, the absolute value of the predicted scores ŷ is not as critical as obtaining a monotonic relationship between the predicted and ground truth scores (using ρ s as the loss function is less feasible due to non-differentiability). ## future work beyond its relevance for the qe task, we expect that the collected signal in the caption-quality dataset will find usage in other image captioning tasks, such as (1) fine-grained caption evaluation (that is, caption classifiers that evaluate captions across multiple dimensions) for example, by way of pretraining against our dataset, as well as (2) improving caption generation itself, for example, by means of qe-based caption re-ranking, or by using the ratings in a reinforcement learning setup, as has recently been done by @xcite . ## conclusion in this paper we discussed how low-quality imagecaptions can negatively impact end-users and proposed a thresholding solution that relies on quality estimation of image captions, where caption quality is defined from a human perspective. to make this solution feasible we developed a scalable human evaluation process with which we annotated a large number of image-captions with their human estimated quality. we provided supporting evidence that the resulting dataset contains a consistent and reliable signal, as well as reported experimental results over professionally labeled finegrained caption annotations, which verify that qe models trained over the caption-quality dataset are effective at filtering out low-quality image captions. to encourage further research in automatic evaluation of image-captions, we make available our large-scale dataset of human judgments at https://github. com/google-research-datasets/ image-caption-quality-dataset .