Transparent Human Evaluation for Image Captioning (2022)

Abstract: We establish THumB, a rubric-based human evaluation protocol for image captioning models. Our scoring rubrics and their definitions are carefully developed based on machine- and human-generated captions on the MSCOCO dataset. Each caption is evaluated along two main dimensions in a tradeoff (precision and recall) as well as other aspects that measure the text quality (fluency, conciseness, and inclusive language). Our evaluations demonstrate several critical problems of the current evaluation practice. Human-generated captions show substantially higher quality than machine-generated ones, especially in coverage of salient information (i.e., recall), while most automatic metrics say the opposite. Our rubric-based results reveal that CLIPScore, a recent metric that uses image features, better correlates with human judgments than conventional text-only metrics because it is more sensitive to recall. We hope that this work will promote a more transparent evaluation protocol for image captioning and its automatic metrics.

## introduction recent progress in large-scale training has pushed the state of the art in vision-language tasks @xcite . one of these tasks is image captioning, whose objective is to generate a caption that describes the given image. the performance in image captioning has been primarily measured in automatic metrics (e.g., @xcite on popular benchmarks, such as mscoco @xcite and flickr8k @xcite . use of these metrics is justified based on their correlation with human judgments collected in previous work @xcite @xcite . continuous use of these previous human judgments, however, raises significant concerns for development of both captioning models and automatic metrics because of their lack of transparency. in previous work, annotators (crowdworkers, typically) rate image captions directly @xcite , pairwise @xcite , or along multiple dimensions such as thoroughness @xcite and truthfulness @xcite . these scoring judgments depend highly on individual annotators' discretion and understanding of the annotation scheme @xcite , making it difficult to decompose, interpret, and validate annotations. this lack of transparency also makes it difficult to interpret evaluation results for downstream applications where some aspects are particularly important (e.g., accessibility for people with visual impairments; @xcite . further, these annotations were done only on relatively old models (e.g., mscoco leaderboard submissions in 2015; @xcite . correlations of automatic metrics with human judgments can break down especially when model types change @xcite , or generation models become increasingly powerful @xcite . we thus develop an up-to-date, transparent human evaluation protocol to better understand how current models perform and how automatic metrics are correlated when applied to current models. at the core of our rubrics are two main scores in a tradeoff: precision and recall (fig. 1 ). the former measures accuracy of the information in a caption, and the latter assesses how much of the salient information in the image is covered. we then penalize a caption if we find a problem in fluency, conciseness, or inclusive language. two or more authors evaluate every instance and collaborate to resolve disagreements, ensuring high quality of the annotations. we assess outputs from four strong models as well as human-generated reference captions from mscoco. we call our scores thumb 1.0 (transparent human benchmark), and release them publicly. ## evaluation protocol we establish a transparent evaluation protocol for english image captioning models. our rubrics and rules are developed through discussions among all annotators (first four authors of this paper) and designed to increase the reliability of evaluation @xcite ## results and analysis we present results and analysis from our evaluations. our transparent evaluations facilitate assessments and analysis of both captioning models ( §3.1) and automatic metrics ( §3.2). ## related work human evaluations for image captioning several prior works conducted human evaluations for image captioning with varying models, datasets, and annotation schemes. much work used crowdworkers from amazon mechanical turk on flickrbased datasets, including the pascal @xcite , flickr8k/30k @xcite , and mscoco datasets. annotators scored the overall quality directly @xcite , pairwise @xcite , or along multiple dimensions, such as truthfulness/correctness @xcite , thoroughness @xcite , relevance @xcite , and grammaticality/readability @xcite . there are similarities between our rubrics and previous an-notations, but our framework defines every dimension in a decomposable way through discussions among all annotators, while focusing on outputs from strong models currently available. apart from these conventional flickr-based datasets, some other work evaluated image captions for social media @xcite ; accessibility for twitter users with vision impairments, @xcite and news articles @xcite . our transparent evaluations would enable us to adjust the aggregation method based on the nature of downstream applications. more specializing categories can be added for these applications in later versions (e.g., thumb 2.0). ## conclusion we developed thumb 1.0, transparent evaluations for the mscoco image captioning task. we refined our rubrics through extensive discussions among all annotators, and ensured the high quality by two-stage annotations. our evaluations demonstrated critical limitations of current image captioning models and automatic metrics. while recent image-based metrics show promising improvements, they are still unreliable in assessing highquality captions from crowdworkers. we hope that our annotation data will help future development of better captioning models and automatic metrics, and this work will become a basis for transparent human evaluations for the image captioning task and beyond. fluency error type penalty example obvious spelling error, one vs. two words 0.1 cel phone, surf board grammatical error that can be easily fixed 0.1 a otter casing issue 0.1 tv, christmas hyphenation 0.1 horse drawn carriage interpretable but unnatural wording 0.1 double decked bus non-trivial punctuation 0.2 a bird standing in the wooded area with leaves all around. misleading spelling error 0.5 a good stands in the grass next to the water. (@xmath0.5 a display case of donuts and doughnuts. ambiguity 0.5 a cat is on a table with a cloth on it. awkward construction 0.1-0.5 there is a freshly made pizza out of the oven. broken sentence 0.5+ a large concrete sign small buildings behind it. clipscore clipscore @xcite is the only referenceless metric out of the 7 metrics. it measures the cosine similarity between the generated caption and given image using the representations from clip. it is shown to correlate better with human judgments from prior work, compared to previous reference-based metrics @xcite . we use the official implementation by the authors. 11 refclipscore refclipscore augments clip-score with the maximum similarity between the generated and reference captions. we again use the official implementation.