Paper: ObjectNet Captions: Models are not superhuman captioners (ICLR-2024)

Reviewer comment (Experiments):
The authors currently did not use GPT-related captioning models, such as BLIP2. According to my usage, BLIP2 outperforms the compared methods used in this paper.

Author response:
In response to reviewers’ comments, we evaluated the publicly available 6.7B parameter BLIP-2 model on ObjectNet Captions, which is an LLM-based captioning model. It received a HUMANr score of -0.41 +/- 0.03 which places it ahead of GIT_L but far below human performance. This is further evidence of the arguments we articulate in our paper: ObjectNet Captions is a challenging dataset and HUMANr is invaluable in its ability to quantify model performance in comparison to humans. We will revise our paper to include this result.

Although very promising, the recent LLaVA models are—according to the ICLR reviewing guidelines—considered contemporaneous with our submission (checkpoints and peer-review were released <4 months ago). Although this excuses us from engaging with the models, we will include them as compute power and time to run on mechanical turk allows. GPT4V is difficult to evaluate because OpenAI doesn't allow new signups and throttles current accounts to only 100 images per day. We are working on including both in the final submission. LLaVA so far does show promise and our preliminary results say that it does close some of the gap with humans. Having established ObjectNet Captions and HUMANr as effective tools for measuring captioning performance, we can apply these models in future work to exploring and measuring the performance gains of these recent models. 

As models become increasingly good, automatic metrics will fail to reliably measure just how good they are. Indeed in preliminary results, GPT4v scores very poorly across all the automatic metrics we report in our paper despite performing much better in HUMANr. We need to explicitly ground model evaluation in systematic human judgment in order to be able to quantify this progress. It may be obvious anecdotally that GPT4v is an improvement over other methods, but HUMANr allows us to quantify that improvement in a way that is aligned with our conception of what qualitative improvement is.

Reviewer comment (Evaluation):
Apart from revealing how big is the difference between machine-generated captions and human-generated captions, it would be meaningful to reveal what is the difference between machine-generated captions and human-generated captions. Though the authors have revealed some differences, such as spurious objects and caption lengths, the root cause seems still unclear.

Author response:
We agree! We are also interested in what differences exist between model and human captions such that human evaluators prefer human captions. In Section 4.3, we choose 7 dimensions to evaluate model and human captions on and do some controlled analysis. What we find is that models are more likely to misclassify objects, misattribute properties to objects, and misjudge the physical relationships between objects. However, all together, our 7 dimensions only account for 21% of the HUMANr score which underscores the difficulty of explicitly characterizing the reasons for the gap in performance. We are interested in working on this in future work, however. HUMANr provides a good way to establish human qualitative judgments that can be investigated to determine features that may explain the judgments.

Reviewer comment (Reproducibility):
Some experiment details are missing. For instance, how to compute the HUMANr score?

Author response:
We apologize for not explicitly describing the calculation of HUMANr. As described in the text, each task is given a rating of 1-9 by a human worker where 1 means only the caption on the left matches the image, 9 means only the caption on the right matches the image, and 5 means the two captions match equally well. That is, the magnitude of the value 1-9 should be interpreted as a slider to indicate which side to give preference to (1 and 9 are equally valent, just on opposite sides). This interpretation is made very clear to the workers.

Then we transform the scores such that each caption is scored as if the human caption were on the left. So if a task with a model caption on the left and human caption on the right was given a score of 7, it would be transformed to a 3. Now that all the scores have the same alignment, we average them all together and then normalize to [-1, 1] where a negative score implies the human captions were preferred on average and a positive score means the model captions were preferred on average.

Reviewer comment (Evaluation):
Asking human participants to rate between 1-9 seems subjective. If two new image captioning models are evaluated with two different groups of people, will the results be comparable? It would be interesting to show the deviation of two different groups of people rating the same model in Figure 4.

Author response:
As figure 4 demonstrates, the difference in HUMANr score between random groups of workers is very small in our experiments, even when those groups are disjoint. Even though individual workers may have subjective preferences, when aggregated over hundreds or even thousands of images, the HUMANr score is quite stable and robust to the exact makeup of the worker pool.

Reviewer comment (Novelty):
The ObjectNet cannot be regarded as a contribution as the authors only select some images with longer captions.

Author response:
We feel that by augmenting ObjectNet and Spoken Objectnet with transcriptions, we have enhanced its usefulness and made a significant contribution to the image captioning field.

Reviewer comment (Writing):
In Section 4.3, paragraph 2, what does “we eliminated all images where GITL failed any of the seven checks above—human failures were not considered’’ mean?

Author response:
This means that we took all the captions that we investigated for which GIT_L passed all 7 of the questions in our quality assessment, but we did not filter the human captions. The result is an optimistic comparison (in favor of GIT_L) between GIT_L captions that didn’t fail the assessment on any dimension versus human captions that may have failed some dimension of the assessment. This shows that even when we consider GIT_L at its best against fallible humans, it still significantly underperforms a human level in terms of HUMANr score.

Reviewer comment (Experiments):
the scale of the dataset is small.

Author response:
The dataset is rather small, but we don’t believe that is a significant limitation. Because ObjectNet Captions inherits the ObjectNet license, it cannot be used to train models so its purpose is only to evaluate which we recommend is done using our new metric HUMANr. As shown in fig 4 in our paper, the variance in HUMANr score declines sharply with the number of images used in the evaluation. Even just 500 images drives the standard deviation very close to zero. It should also be noted that our dataset is not small compared to other captioning test sets. For example, the nocaps test set contains only 10,600 images.

Reviewer comment (Experiments):
the auther only considers traditional image captioning models. Some LLM-based models like LLaVA should be considered and the comparison among these models should be more interesting.

Author response:
Although very promising, the LLaVA models are—according to the ICLR reviewing guidelines—considered contemporaneous with our submission (checkpoints and peer-review were released <4 months ago). Although this excuses us from engaging with the models, we will include them as compute power and time to run on mechanical turk allows. GPT4V is difficult to evaluate because OpenAI doesn't allow new signups and throttles current accounts to only 100 images per day. We are working on including both in the final submission. LLaVA so far does show promise and our preliminary results say that it does close some of the gap with humans. Having established ObjectNet Captions and HUMANr as effective tools for measuring captioning performance, we can apply these models in future work to exploring and measuring the performance gains of these recent models. 

As models become increasingly good, automatic metrics will fail to reliably measure just how good they are. Indeed in preliminary results, GPT4v scores very poorly across all the automatic metrics we report in our paper despite performing much better in HUMANr. We need to explicitly ground model evaluation in systematic human judgment in order to be able to quantify this progress. It may be obvious anecdotally that GPT4v is an improvement over other methods, but HUMANr allows us to quantify that improvement in a way that is aligned with our conception of what qualitative improvement is.

Reviewer comment (Novelty):
the findings that there is a large gap betten human and models is a common sense, so I do not think it is a significant contribution. But if the author can show that the most advanced models like GPT-4v is inferior to humans and the proposed metric is able to measure the gap, it should be more interesting.

Author response:
We know that the gap in performance between humans and models in image captioning is no secret. Every researcher knows anecdotally that models fail in ways that humans don’t. However, until now, there has been no way to measure that gap. Automatic metrics cannot quantify this gap for us. Rather, if we relied solely automatic metrics and not on anecdotal evidence, we would be led to believe that the gap does not exist. We introduce HUMANr not to prove that the gap exists—like the reviewer notes, everyone knows that the gap exists—but rather to present a method for measuring that gap. Our contribution is not discovering that gap, but rather presenting and motivating a metric which actually reflects that gap which has not been done systematically in the literature. We will revise the language of the text to reflect this.

Please see above for discussion regarding GPT4v limited availability.

Reviewer comment (Experiments):
The dataset's focus on home environments and its relatively small size (17,674 images) may limit its generalizability.

Author response:
The restriction to in-home environments is a limitation, but it still allows ObjectNet to meaningfully probe models abilities. The in-home/indoor environment is an important context where many robots or future AI systems could be deployed. It provides a wide variety of objects and significant visual complexity. Also, non-expert human annotators have the appropriate visiolinguistic experience to caption indoor in-home images which makes captioning easy. Future work could explore captioning in other environments, but to start with, the indoor environment is likely the most broadly generalizable and the most difficult.

As for the size of ObjectNet Captions, we don’t believe its size is a limitation. As shown in fig 4 in our paper, the variance in HUMANr score declines sharply with the number of images used in the evaluation. Even just 500 images drives the standard deviation very close to zero. It should also be noted that our dataset is not small compared to other captioning test sets. For example, the nocaps test set contains only 10,600 images.

Reviewer comment (Experiments):
Not including state-of-the-art models like BLIP2 or LLM-based models in the analysis.

Author response:
In response to reviewers’ comments, we evaluated the publicly available 6.7B parameter BLIP2 model on ObjectNet Captions. It received a HUMANr score of -0.41 +/- 0.03 which places it ahead of GIT_L but far below human performance (negative is worse than humans, 0 is human level, and positive is better than humans; with a range of -1 to +1). This is further evidence of the arguments we articulate in our paper: ObjectNet Captions is a challenging dataset and HUMANr is invaluable in its ability to quantify model performance in comparison to humans. We will revise our paper to include this result.

As for recent LLM-based methods, the LLaVA models are very promising but are—according to the ICLR reviewing guidelines—considered contemporaneous with our submission (checkpoints and peer-review were released <4 months ago). Although this excuses us from engaging with the models, we will include them as compute power and time to run on mechanical turk allows. GPT4V is difficult to evaluate because OpenAI doesn't allow new signups and throttles current accounts to only 100 images per day. We are working on including both in the final submission. LLaVA so far does show promise and our preliminary results say that it does close some of the gap with humans. Having established ObjectNet Captions and HUMANr as effective tools for measuring captioning performance, we can apply these models in future work to exploring and measuring the performance gains of these recent models. 

As models become increasingly good, automatic metrics will fail to reliably measure just how good they are. Indeed in preliminary results, GPT4v scores very poorly across all the automatic metrics we report in our paper despite performing much better in HUMANr. We need to explicitly ground model evaluation in systematic human judgment in order to be able to quantify this progress. It may be obvious anecdotally that GPT4v is an improvement over other methods, but HUMANr allows us to quantify that improvement in a way that is aligned with our conception of what qualitative improvement is.

Reviewer comment (Evaluation):
The human-centric approach, while insightful, may introduce new biases and subjectivities.

Author response:
There are, of course, some trade-offs to be made when deciding to use human evaluation. 

Although we note that any human annotators bring their own biases to every experiment, we attempted to make instructions as clear as possible to maximize reproducibility. The fact that workers agree on HUMANr (as evidenced by the fact that the variance is small / inter-coder agreement is high) means that workers understood the instructions and answered consistently. AMT is spread around the world, its population changes radically across continents during a long experiment, we would not have this level of consistency if each group was strongly influenced by local biases. There may be AMT-specific biases that are shared by all workers, but there is little evidence for this.

Even if human workers were to be biased in a particular way, there’s no clear reason why we would want to remove this bias. Parity with human capability is, after all, our goal. We want to build models that are as good as humans as judged by humans. If a model scores 0 HUMANr, that means that a large group of humans did not find its captions to be distinguishable on average from human captions. That is no small feat regardless of what bias you think might exist in the individual worker subjectivity. In future work, as models move closer to human level performance on this general captioning task our same method could be used to study vision tasks with more culturally subjective responses and the background of human evaluators could then be taken into account when aggregating results.

Reviewer comment (Novelty):
The revelation of a performance gap between humans and models is not a novel insight and lacks depth without comparing the most advanced models.

Author response:
We know that the gap in performance between humans and models in image captioning is no secret. Every researcher knows anecdotally that models fail in ways that humans don’t. However, until now, there has been no way to measure that gap. Automatic metrics cannot quantify this gap for us. We introduce HUMANr not to prove that the gap exists, but rather to present a method for measuring that gap which we show is not measured by automatic metrics and is only anecdotally recognized. We will revise the language of the text to reflect this.

As for analysis with the most advanced models, please refer to our response above. Models such as BLIP2 have improved in their HUMANr score, and we plan to include even more advanced models on a leaderboard website as those become available. However, automatic metrics fail to measure this performance improvement which further supports the need to systematize human-in-the-loop evaluation like we do with HUMANr.

Reviewer comment (Reproducibility):
The paper omits crucial experimental details, like the computation of HUMANr and handling discrepancies in human evaluations.

Author response:
As described in the text, each task is given a rating of 1-9 by a human worker where 1 means only the caption on the left matches the image, 9 means only the caption on the right matches the image, and 5 means the two captions match equally well. We randomize which caption is on the left and right so our rating scale requires a simple transformation before we can aggregate the scores such that each caption is scored as if the human caption were on the left. So if a task with a model caption on the left and human caption on the right was given a score of 7, it would be transformed to a 3. Now that all the scores have the same alignment, we average them all together and then normalize to [-1, 1] where a negative score implies the human captions were preferred on average and a positive score indicates that the model captions were preferred on average. We will include this more detailed explanation in the revision.

As for discrepancies in human evaluations, each task contains an attention check, as described in our paper on page 7, in which a caption from a random image was added to the task. If the worker fails the attention check by selecting the random caption, all of their responses to any task are ignored when computing HUMANr. That way, we ensure that all our results come from attentive workers.
We also show in figure 4 that the HUMANr score variance between disjoint sets of the worker pool is very low. When aggregated over hundreds of images, the subjectivity of individual workers is largely averaged out resulting in a convergent HUMANr score with low variance.