Transparent Human Evaluation for Image Captioning
Abstract
We establish TH UM B, a rubric-based human evaluation protocol for image captioning models.
Our scoring rubrics and their definitions are carefully developed based on machine- and human-generated captions on the MSCOCO dataset.
Each caption is evaluated along two main dimensions in a tradeoff ( precision and recall ) as well as other aspects that measure the text quality ( fluency , conciseness , and inclusive language ).
Our evaluations demonstrate several critical problems of the current evaluation practice.
Human-generated captions show substantially higher quality than machine-generated ones, especially in coverage of salient information (i.e., recall), while most automatic metrics say the opposite.
Our rubricbased results reveal that CLIPScore, a recent metric that uses image features, better correlates with human judgments than conventional text-only metrics because it is more sensitive to recall.
We hope that this work will promote a more transparent evaluation protocol for image captioning and its automatic metrics.
Introduction
Recent progress in large-scale training has pushed the state of the art in vision-language tasks (Li et al., 2020; Zhang et al., 2021, inter alia ).
One of these tasks is image captioning, whose objective is to generate a caption that describes the given image.
The performance in image captioning has been primarily measured in automatic metrics (e.g., CIDEr,
Vedantam et al., 2015; SPICE, Anderson et al., 2016) on popular benchmarks, such as MSCOCO (Lin et al., 2014) and Flickr8k (Hodosh et al., 2013).
Use of these metrics is justified based on their correlation with human judgments collected in previous work (Hodosh et al., 2013; Elliott and Keller, 2014;
Kilickaya et al., 2017, inter alia ).
[Figure]
Continuous use of these previous human judgments, however, raises significant concerns for development of both captioning models and automatic metrics because of their lack of transparency.
In previous work, annotators (crowdworkers, typically) rate image captions directly (Hodosh et al., 2013), pairwise (Vedantam et al., 2015), or along multiple dimensions such as thoroughness (Aditya et al., 2015) and truthfulness (Yatskar et al., 2014).
These scoring judgments depend highly on individual annotators’ discretion and understanding of the annotation scheme (Freitag et al., 2021; Clark et al., 2021), making it difficult to decompose, interpret, and validate annotations.
This lack of transparency also makes it difficult to interpret evaluation results for downstream applications where some aspects are particularly important (e.g., accessibility for people with visual impairments; Gleason et al., 2019, 2020).
Further, these annotations were done only on relatively old models (e.g., MSCOCO leaderboard submissions in 2015; Anderson et al.,
Correlations of automatic metrics with human judgments can break down especially when model types change (Callison-Burch et al., 2006), or generation models become increasingly powerful (Ma et al., 2019;
Edunov et al., 2020).
We thus develop an up-to-date, transparent human evaluation protocol to better understand how current models perform and how automatic metrics are correlated when applied to current models.
At the core of our rubrics are two main scores in a tradeoff: precision and recall (Fig. 1).
The former measures accuracy of the information in a caption, and the latter assesses how much of the salient information in the image is covered.
We then penalize a caption if we find a problem in fluency , conciseness , or inclusive language .
Two or more authors evaluate every instance and collaborate to resolve disagreements, ensuring high quality of the annotations.
We assess outputs from four strong models as well as human-generated reference captions from MSCOCO.
We call our scores TH UM B 1.0 ( T ransparent Hum an B enchmark), and release them publicly.
We made several key observations from the evaluations.
• Machine-generated captions from recent models have been claimed to achieve superhuman performance using popular automatic metrics (human performance is ranked at the 250th place in the MSCOCO leaderboard), 2 but they still show substantially lower quality than human-generated ones.
• Machines fall short of humans, especially in recall (Fig. 1), but most automatic metrics say the opposite.
This finding is consistent with prior work that showed that machines tend to produce less diverse captions than humans (van Miltenburg et al., 2018).
• Human performance is underestimated in the current leaderboard paradigm, and there is still much room for improvement on MSCOCO captioning.
• CLIPScore and RefCLIPScore (Hessel et al., 2021), recently proposed metrics that use image features, improve correlations particularly in recall.
While they fail to score human generation
much higher than machine one, they capture an
aspect that is less reflected in text-only metrics.
• Currently available strong captioning models generate highly fluent captions.
Fluency evaluation is thus no longer crucial in ranking these models.
Evaluation Protocol
We establish a transparent evaluation protocol for English image captioning models.
Our rubrics and rules are developed through discussions among all annotators (first four authors of this paper) and designed to increase the reliability of evaluation (Jonsson and Svingby, 2007)
2. Evaluation Setups and Quality Control
We used images from the test data in the standard Karpathy split (Karpathy and Fei-Fei, 2015) of the MSCOCO dataset (Lin et al., 2014).
The dataset consists of 113K, 5K, and 5K train/dev./test everyday-scene photos sampled from Flickr.
We randomly sampled 500 test images and prepared one human- and four machine-generated captions for every image (§2.3).
We first performed developmental evaluations of 250 captions forimages and created rubrics.
We then proceeded with the rest of the captions.
For every image, captions were shuffled, and thus annotators did not know which caption corresponded to which model, thereby avoiding a potential bias from knowledge about the models.
We conducted two-stage annotations: the first annotator scores all captions for given images, and the second annotator checks and modifies the scores when necessary.
After the developmental phase, the κ coefficient (Cohen, 1960) was 0.86 in precision and 0.82 in recall for the rest of the evaluated captions (§2.2.1).
The first four authors of this paper conducted all evaluations; none of them are color blind or low vision, two are native English speakers, and one is a graduate student in linguistics.
We finally ensured that at least one native speaker evaluated the fluency of every caption (§2.2.2), meaning that if a caption is annotated by the two non-native speakers, one native speaker checks the fluency in an additional round.
TH UM B 1.
Similar to the framework of the automatic SPICE metric (Anderson et al., 2016), we base our manual evaluations on two main scores: precision and recall . We also consider three types of penalty: fluency , conciseness , and inclusive language .
The overall score is computed by averaging precision and recall and deducting penalty points.
2.2. Main Scores
The two main scores are assessed in the scale of 1–5.
They balance information accuracy and coverage.
Precision
Precision (P) measures how precise the caption is given the image.
For instance, Caption 1B in Table 1 is perfectly precise, while 1-A ( dog vs. otter , one vs. two frisbees ) and 1-C ( three vs. two frisbees ) are not precise.
Precision guards against hallucinations from the language model ( table in 2B) that are known to be common failures of image captioning models (Rohrbach et al., 2018).
The score of 4 is reserved for relatively minor issues, such as attributes that are almost correct (e.g., pink vs. red in 4-C, Table 1) or cases where the caption does not contradict the image but is not guaranteed to be true (e.g., it is unclear whether the girl is sitting on a couch in 3-B).
In addition to objects themselves, precision deals with information like properties, attributes, occasions, locations, and relations between objects (e.g., in a red suitcase vs. on a red suitcase in 4-A).
Recall
Recall (R) measures how much of the salient information (e.g., objects, attributes, and relations) from the image is covered by the caption.
This includes color (e.g., color of the frisbees in 1-A, 1-B, and 1-C) and guards against generic, uninformative captions that machines tend to produce (Wang et al., 2020).
For instance, an otter is a small animal , and thus small animal is precise (1C); however, it is much less informative (and less natural; Ordonez et al., 2013) than saying an otter.
Similarly, Caption 5-B only says a woman is standing behind a counter at a donut shop, but she is selling donuts, not buying or looking at donuts, which is salient information from the picture.
We do not take a point off if missing information is already expected from the caption (e.g., a double-decker bus is typically red).
We often find it useful to take a generative approach when evaluating recall: what image does the caption lead us to imagine?
When the caption entails many potential images that substantially diverge from the given image, the recall score should be low.
2.2. Penalties Fluency
Fluency (Flu.) measures the quality of captions as English text regardless of the given image.
Initially, we scored fluency in the scale of 1–5, similar to P and R, but we found most captions from modern neural network models were highly fluent.
Thus, we instead decided to take points off from the average of P and R if there’s a fluency problem to account for minor issues that are much less problematic than losing one P/R point.
The four annotators had extensive discussions and developed rubrics for fluency.
Similar to recent work on professional evaluations for machine translation (Freitag et al., 2021), we evaluated under the following principle: if a fluency problem is expected to be easily corrected by a text postprocessing algorithm (e.g., grammatical error correction:
Yuan and
Briscoe, 2016; Sakaguchi et al., 2017), the penalty should be 0.1.
This includes obvious misspellings or grammatical errors (e.g., A otter in 1-B) and missing determiners/hyphens ( multi colored in 2C).
0.5+ points were subtracted for more severe problems, such as duplication (e.g., A display case of donuts and doughnuts ), ambiguity (e.g., A cat is on a table with a cloth on it ), and broken sentences (e.g., A large concrete sign small buildings behind it. ).
See Table 6 in §A.1 for more extensive fluency rubrics.
Note that the average fluency penalty was 0.01; this confirms that fluency is no longer crucial in ranking models for MSCOCO captioning and contrasts with human evaluations previously done for older captioning models.
Conciseness
The scores so far do not take into account conciseness of captions.
Specifically, a model could simply increase all scores by describing every detail in a picture.
For instance, the following caption is overly repetitive: a woman lying on her back with knees bent on a beach towel under a multicolored, striped beach umbrella, surrounded by sand, and with clear blue sky above .
We subtract 0.5 points for these captions.
Note that most machine captions were short, and this penalty was only applied to two human-generated captions.
It might become more crucial for future models [Table] with a more powerful object detection module that catches many objects in the picture.
Inclusive Language
We found that some instances substantially diverge from inclusive language when humans are described (van Miltenburg, 2020), raising a concern for downstream applications.
In these cases, we added a penalty: 0.5 points were deducted for a subjective comment about appearance (e.g., very pretty girl ), and 2 points for more severe problems (e.g., beautiful breasts ).
2.2. Rules of TH UM B
In our development phase, we established the following additional rules to clarify our annotation scheme.
Avoiding Double Penalties
When an error is accounted for in precision, we correct the error before scoring the recall, thereby avoiding penalizing the precision and recall for the same mistake.
For example, P=3 is given to Caption 1-A in Tablebecause of its wrong detection ( dog vs. otter ; one vs. two frisbees ), but we score the recall assuming that the caption is now an otter playing with two frisbees on the ground .
This ensures that a generic, useless caption, such as there is something on something (P=5, R=1), would be ranked considerably lower than a dog on the beach with two pink and yellow frisbees (P=3, R=5).
Similarly, the wrong detection in 5-A ( man vs. woman ) is handled only in precision.
Note that such error correction is not applicable to hallucinations because there is no alignment between a part of the image and a hallucinated object (e.g., table in 2-B).
This rule departs from the definition of recall in SPICE (Anderson et al., 2016), an automatic metric that measures the
F 1 score in scene graphs predicted from reference and generated captions; their alignment is limited to WordNet synonyms (Miller, 1995).
This means that classifying an otter as a dog or even a small animal would result in cascading errors both in precision and recall, overrating captions that completely overlook the otter or ones that make a more severe classification error (e.g., miscategorize the otter as a car, compared to a dog).
Object Counts as Attributes
All counts are considered as object attributes, and wrong counts are handled in precision.
This simplifies the distinction between precision and recall.
For instance, both a frisbee (1-A) and three frisbees (1-C) are precision problems, while saying some frisbees would be a recall problem when it is clear that there are exactly two frisbees.
Note that this is in line with SPICE, which treats object counts as attributes in a scene graph, rather than duplicating a scene graph for every instance of an object (Anderson et al., 2016).
Black and White Photo
MSCOCO contains black and white or gray-scale pictures.
Some captions explicitly mention that they are black and white, but we disregard this difference in our evaluations.
The crowdsource instructions for creating reference captions do not specify such cases (Chen et al., 2015).
Further, we can potentially run postprocessing to determine whether it is black and white to modify the caption accordingly, depending on the downstream usage.
Text Processing
Image captioning models often differ slightly in text preprocessing.
As a result, we found that generated captions were sometimes slightly different in format (e.g., tokenized or detokenized; lowercased or not).
For better reproducibility, we follow the spirit of S ACRE BLEU (Post, 2018), which has become the standard package to compute BLEU scores for machine translation: all evaluations, including automatic metrics, should be done on clean, untokenized text, independently of preprocessing design choices.
We apply the following minimal postprocessing to the model outputs and human captions.
• Remove unnecessary spaces at the start or end of every caption.
• Uppercase the first letter.
• Add a period at the end if it doesn’t exist, and remove a space before a period if any.
We keep the postprocessing minimal for this work and encourage future model developers to follow the standard practice in machine translation: every model has to output clean, truecased, untokenized text that is ready to be used in downstream modules.
This also improves the transparency and reproducibility of automated evaluations (Post, 2018).
2. Evaluated Captions
We evaluated the following four strong models from the literature as well as human-generated captions.
They share similar pipeline structure: object detection followed by crossmodal caption generation.
They vary in model architecture, (pre)training data, model size, and (pre)training objective.
Evaluating captions from them will enable us to better understand what has been improved and what is still left to future captioning models.
• Up-Down (Anderson et al., [-
 )-]2018) trains Faster R-CNN (Ren et al., 2015) on the Visual Genome datset (Krishna et al., 2016) for object detection.
It then uses an LSTM-based crossmodal generation model.
• Unified-VLP (Zhou et al., 2020) uses the same object detection model as Up-Down.
The transformer-based generation model is initialized with base-sized BERT (Devlin et al., 2019) and further pretrained with 3M images from Conceptual Captions (Sharma et al., 2018).
• VinVL-base and VinVL-large (Zhang et al., 2021) train a larger-scale object detection model with the ResNeXt-152 C4 architecture (Xie et al., 2017) on ImageNet (Deng et al., 2009).
The transformer generation model is initialized with BERT and pretrained with 5.7M images.
• Human randomly selects one from the five human-generated reference captions in MSCOCO.
Those captions
were created by crowdworkers on Amazon Mechanical Turk (Chen et al., 2015).
Further details are described in §A.3 of Appendix.
Results and Analysis
We present results and analysis from our evaluations.
Our transparent evaluations facilitate assessments and analysis of both captioning models (§3.1) and automatic metrics (§3.2).
3. Comparing Models
Seen in Table 2 (left section) is the model performance that is averaged over the 500 test images and broken down by the rubric categories.
Overall, Human substantially outperforms all machines in the
[Table]
[Equation]
R, and total scores.
In particular, we see a large gap between Human and the machines in recall (e.g., Human 4.35 vs. VinVL-large 3.97).
This contrasts with the automatic metric-based ranking of the MSCOCO leaderboard, where Human is ranked at the 250th place.
This result questions claims about human parity or superhuman performance on MSCOCO image captioning.
The four machine captioning models are ranked in the expected order, though the small difference between VinVL-large and VinVL-base suggests that simply scaling up models would not lead to a substantial improvement.
We see that the three models that are initialized with pretrained BERT (VinVL-large/base,
Unified-VLP) are particularly fluent, but the problem is small in the other models as well.
While we compute representative, total scores, our transparent rubrics allow for adjusting weighting of the categories depending on the application of interest.
For instance, in the social media domain, recall can be more important than precision to make captions engaging to users (Shuster et al., 2019).
To assess the models independently of these aggregation decisions, we count the number of times when each model outperforms/underperforms all the others both in P and
R ( strictly best/worst, Table 3).
We see patterns consistent with Table 2.
For example, Human is most likely to be strictly best and least likely to be strictly worst.
This suggests that machine captioning models would still fall short of crowdworkers in a wide range of downstream scenarios.
[Table]
3. Comparing Automatic Metrics
While carefully-designed human judgments like ours should be considered more reliable, automatic metrics allow for faster development cycles.
Our transparent evaluations can also be used to analyze how these automatic metrics correlate with different aspects of image captioning.
Table 2 (right section) shows automatic scores of the captioning models over 7 popular metrics for image captioning.
CLIP-S (Hessel et al., 2021) is a referenceless metric that uses image features from CLIP (Radford et al., 2021), a crossmodal retrieval model trained on 400M image-caption pairs from the web.
RefCLIP-S augments CLIP-S with similarities between the generated and reference captions.
All other metrics, such as SPICE (Anderson et al., 2016) and CIDEr (Vedantam et al., 2015), only use reference captions without image features.
These automatic metrics generally agree with our evaluations in ranking the four machines, but completely disagree in the assessment of Human.
Most metrics rank Human near the bottom, showing that they are not reliable in evaluating high-quality, human-generated captions.
The two metrics with powerful image and text features (CLIP-S and RefCLIP-S) give high scores to Human compared to the other metrics, but they still fail to score Human substantially higher than
VinVL-large.
This suggests that automatic metrics should be regularly updated as our models become stronger (and perhaps more similar to humans), and raises a significant concern about the current practice that fixes evaluation metrics over time (Kasai et al., 2022). [Table]
All of these reference-based metrics use the same subset of four captions as in Table 2 that exclude Human metrics correlations
Seen in Table 4 are instance-level Pearson correlation scores between automatic scores and our evaluations.
6 We also add an ablation study: RefOnlyC removes image features from RefCLIP-S to quantify the effect of image features.
We consider two types of scenarios: one with Human and one without .
Correlations drop from the latter to the former for all metrics and aspects except CLIP-S, again showing that the metrics are not reliable in assessing human-generated captions.
Interestingly,
CLIP-S correlates best in recall (0.28 w/ Human) but suffers in precision (0.17 w/ Human).
RefOnlyC, in contrast, achieves the best correlations in P at the expense of R. RefCLIP-S balances the two and achieves the best correlation in total scores.
This indicates that the CLIP image features particularly help assess coverage of salient information that can be ignored in some reference captions from crowdworkers.
7 Prior work (Hessel et al., 2021) found that SPICE can still improve correlations when combined with CLIP-S, even though
CLIP-S better correlates with human judgments than SPICE.
This implies that image-based and reference-only metrics capture different aspects of image captioning.
Our analysis indeed agrees with their finding and, further, identifies that recall is one such aspect.
For an extensive description of these metrics and their configurations, see §A.2.
3. Score Distributions
Seen in Fig.
2 are distributions of precision and recall scores for human and machine-generated captions. We see that the precision distribution looks similar between Human and machines, but not recall. This provides further support for our claim that current machines fall short of humans particularly in recall. [Figure]
3. Machine vs. Human Examples
Table 5 provides examples that contrast machineand human-generated captions.
We see that machine-generated captions ignore salient information or make critical errors for these images.
These problems often occur in relatively rare cases: a tennis player is showing excitement rather than hitting a ball; a bride and groom are cutting a wedding cake; a boy is wearing a tie without a shirt; a man is putting clothing and a tie on a dummy instead of a person.
But these situations are exactly the most important information because of their atypicality (Feinglass and Yang, 2021).
This illustrates fundamental problems of current image [Table] captioning models that are left to future work.
Related Work
Human Evaluations for Image Captioning
Several prior works conducted human evaluations for image captioning with varying models, datasets, and annotation schemes.
Much work used crowdworkers from Amazon Mechanical Turk on Flickrbased datasets, including the PASCAL (Rashtchian et al., 2010),
Flickr8k/30k (Hodosh et al., 2013;
Young et al., 2014), and MSCOCO datasets.
Annotators scored the overall quality directly (Kulkarni et al., 2011;
Hodosh et al., 2013), pairwise (Vedantam et al., 2015), or along multiple dimensions, such as truthfulness/correctness (Yatskar et al., 2014; Anderson et al., 2016), thoroughness (Aditya et al., 2015), relevance (Yang et al., 2011; Li et al., 2011), and grammaticality/readability (Mitchell et al., 2012; Elliott and Keller, 2013).
There are similarities between our rubrics and previous an- notations, but our framework defines every dimension in a decomposable way through discussions among all annotators, while focusing on outputs from strong models currently available.
Apart from these conventional Flickr-based datasets, some other work evaluated image captions for social media (engagingness, Shuster et al., 2019; accessibility for Twitter users with vision impairments,
Gleason et al., 2019, 2020) and news articles (Biten et al., 2019).
Our transparent evaluations would enable us to adjust the aggregation method based on the nature of downstream applications.
More specializing categories can be added for these applications in later versions (e.g., TH UM B 2.0).
Human Evaluations for Other Generation Tasks
Much previous work explored human evaluations for other language generation tasks than image captioning.
The WMT shared task (Akhbardeh et al., 2021) conducts human evaluations of state-of-theart machine translation systems every year; participants or crowdworkers directly rate a translation in a 100-point scale, which is a method developed by Graham et al.
G ENIE takes a similar approach but hosts human evaluations in leaderboards for machine translation, summarization, and commonsense reasoning (Khashabi et al., 2021).
Kryscinski et al.
and Fabbri et al.
(2021) assessed many summarization models in a similar annotation scheme to the DUC 2006/ evaluations (Dang, 2006).
Our transparent evaluation framework is inspired by rubric-based machine translation judgments by professional translators (Freitag et al., 2021), which resulted in different system rankings than the WMT evaluations.
As top-performing models and automatic metrics are becoming increasingly similar across various natural language generation tasks, our findings on image captioning may be useful for other generation tasks as well.
Conclusion
We developed TH UM B 1.0, transparent evaluations for the MSCOCO image captioning task.
We refined our rubrics through extensive discussions among all annotators, and ensured the high quality by two-stage annotations.
Our evaluations demonstrated critical limitations of current image captioning models and automatic metrics.
While recent image-based metrics show promising improvements, they are still unreliable in assessing highquality captions from crowdworkers.
We hope that our annotation data will help future development of better captioning models and automatic metrics, and this work will become a basis for transparent human evaluations for the image captioning task and beyond. Acknowledgments We thank Jack Hessel, Daniel Khashabi, Rowan
Zellers, Noriyuki Kojima, Desmond Elliott, and Koji Shiono as well as the anonymous reviewers for their helpful feedback on this work. We also thank Jack Hessel, Xiujun Li, Ani Kembhavi, Eric Kolve, and Luowei Zhou for their help with captioning models and automatic evaluation metrics.
This work was supported in part by the DARPA
MCS program through NIWC Pacific (N66001-192-4031) and Google Cloud Compute.
A Appendix A. Fluency Rubrics
Table 6 presents our fluency rubrics.
They were developed by the first four authors (two of whom were native English speakers, and one was a graduate student in linguistics).
Generally, if a fluency problem is expected to be easily corrected by a text postprocessing algorithm, the penalty is 0.1.
More severe errors (e.g., broken sentence and ambiguity) are penalized more.
A. Automatic Metrics
Here we discuss details and configurations of the automatic metrics used in §3.2.
CLIPScore and RefCLPScore use image features from CLIP (Radford et al., 2021), a crossmodal retrieval model trained on 400M image-caption pairs from the web.
All the other five metrics only use reference captions.
BLEU
BLEU (Papineni et al., 2002) is a precision-oriented metric and measures n-gram overlap between the generated and reference captions.
We use the S ACRE BLEU implementation of BLEU-4 and get sentence-level scores (Post, 2018).
ROUGE
ROUGE (Lin, 2004) measures the number of overlapping n-grams between the generated and reference captions.
We use the rouge-score implementation of ROUGE-L.
CIDEr
CIDEr (Vedantam et al., 2015) measures the cosine similarity between the n-gram counts of the generated and reference captions with TF-IDF weighting.
We use the implementation from the pycocoevalcap package.
SPICE
SPICE (Anderson et al., 2016) predicts scene graphs from the generated and reference captions using the Stanford scene graph parser (Schuster et al., 2015).
It then measures the F 1 score between scene graphs from the generated and reference captions.
WordNet Synsets are used to cluster synonyms (Miller, 1995).
We again use the implementation from the pycocoevalcap package.
BERTScore
BERTScore (Zhang et al., 2020) aligns tokens between the generated and reference captions using contextual word representa- [Table] tions from BERT (Devlin et al., 2019).
We use the
HuggingFace implementation (Wolf et al., 2020) and compute the F 1 score.
As in Zhang et al.
(2020), we take the maximum score over all reference captions.
CLIPScore
CLIPScore (Hessel et al., 2021) is the only referenceless metric out of the 7 metrics.
It measures the cosine similarity between the generated caption and given image using the representations from CLIP.
It is shown to correlate better with human judgments from prior work, compared to previous reference-based metrics (Hessel et al., 2021).
We use the official implementation by the authors.
RefCLIPScore
RefCLIPScore augments CLIPScore with the maximum similarity between the generated and reference captions.
We again use the official implementation.
A. Evaluated Captions
We evaluated the following four strong models from the literature as well as human-generated captions.
They share similar pipeline structure but vary in model architecture, (pre)training data, model size, and (pre)training objective.
Evaluating captions from them will enable us to better understand what has been improved and what is still left to future captioning models
. Up-Down
The bottom-up and top-down attention model (Up-Down, Anderson et al., 2018) performs pipelined image captioning: object detection that finds objects and their corresponding image regions and crossmodal generation that predicts a caption based on the features from object detection.
The bottom-up attention finds salient image regions during object detection, and the top-down one attends to these regions during crossmodal generation.
Up-Down uses Faster R-CNN (Ren et al., 2015) and LSTMs (Hochreiter and Schmidhuber, 1997) for object detection and crossmodal generation respectively.
Faster R-CNN is trained with the Visual Genome dataset (Krishna et al., 2016), and the crossmodal generation model is trained on the MSCOCO dataset.
We generate captions for the test data with a model optimized with crossentropy.
Unified-VLP
Unified-VLP (Zhou et al., 2020) also runs a pipeline of object detection and crossmodal generation.
Faster R-CNN and the transformer architecture (Vaswani et al., 2017) are used for object detection and crossmodal generation respectively.
Similar to Up-Down, the Faster RCNN object detection model is trained with the Visual Genome dataset.
The transformer generation model, on the other hand, is initialized with basesized BERT (Devlin et al., 2019) and pretrained on the Conceptual Captions dataset (3M images,
Sharma et al., 2018) with the masked and left-toright language modeling objectives for the captions.
The crossmodal generation model is then finetuned on the MSCOCO dataset.
We apply beam search of size 5 to the model with CIDEr optimization.
VinVL-base, VinVL-large
VinVL with Oscar (Li et al., 2020; Zhang et al., 2021) performs a similar pipeline of object detection, followed by crossmodal generation.
The crossmodal model is initialized with BERT (Devlin et al., 2019) as in
Unified-VLP but uses detected object tags to encourage alignments between image features and word representations.
The object detection model with the ResNeXt-152 C4 architecture (Xie et al., 2017) is pretrained with ImageNet (Deng et al., 2009) and trained on 2.5M images from various datasets.
The transformer-based crossmodal generator is initialized with BERT, pretrained with 5.7M images, and finetuned for MSCOCO captioning.
We use VinVL-base and VinVL-large that are both finetuned with CIDEr optimization 13 and generate captions with beam search of size 5.
Human
In addition to machine-generated captions from the four models, we assessed the quality of human-generated reference captions from
MSCOCO.
This will allow us to understand the performance gap between machines and humans, as well as the quality of crowdsourced captions.
Human-generated captions were created using
Amazon Mechanical Turk (Chen et al., 2015).
Crowdworkers were only given the following instructions (Chen et al., 2015):
• Describe all the important parts of the scene. • Do not start the sentences with “There is.”
• Do not describe unimportant details.
• Do not describe things that might have happened in the future or past. • Do not describe what a person might say.
• Do not give people proper names.
• The sentences should contain at least 8 words.
Every image has five human-generated captions, and we randomly selected one for each to evaluate.
We found, however, a non-negligible number of noisy captions in the MSCCOCO dataset from annotation spammers.
We often find subjective adjectives (e.g., very nice/clean/cute ) or words that diverge from inclusive language in reference captions, probably because crowdworkers increased the number of words in captions effortlessly (see the last instruction item that says captions have to have 8+ words).
To better estimate the performance of a human that invests reasonable effort into the captioning task, we resampled a caption for 13% of the test images, which would have been given a total score lower than 4.0.
A. Additional Machine vs. Human Examples
Table 7 provides an additional example that contrasts machine- and human-generated captions.
All machines generate generic captions and ignore the most important information that a traditional
Thanksgiving dinner is being served on the table.
[Table]