IC 3: Image Captioning by Committee Consensus (2023)

Abstract: If you ask a human to describe an image, they might do so in a thousand different ways. Traditionally, image captioning models are trained to generate a single “best’ (most like a reference) image caption. Unfortunately, doing so encourages captions that are “informationally impoverished,’ and focus on only a subset of the possible details, while ignoring other potentially useful information in the scene. In this work, we introduce a simple, yet novel, method: “Image Captioning by Committee Consensus’ (IC3), designed to generate a single caption that captures high-level details from several annotator viewpoints. Humans rate captions produced by IC3 at least as helpful as baseline SOTA models more than two thirds of the time, and IC3 can improve the performance of SOTA automated recall systems by up to 84%, outperforming single human-generated reference captions, and indicating significant improvements over SOTA approaches for visual description. Code is available at [https://davidmchan.github.io/caption-by-committee/](https://davidmchan.github.io/caption-by-committee/)

## introduction generating a high-quality description of an image is not only an open research problem, but it is also a challenging task for humans @xcite @xcite . image captioning datasets usually acknowledge this fact; rather than providing a single gold standard caption for each image, they instead rely on human annotators to provide multiple captions for each image, hoping that the set of collected captions collectively captures all of the relevant semantic information. ## background and related work the idea that captioning models tend to produce single-viewpoint captions has been prevalent in the image captioning community for many years under several names @xcite . notably, research has focused on quantifying and improving the "diversity" of output captions, including specific methods @xcite @xcite @xcite @xcite and metrics @xcite @xcite @xcite @xcite . as an alternate approach to increasing and quantifying diversity, some methods @xcite @xcite have focused on explicitly modeling the vari-ance in the caption space, and introduced human, or statistical controls to reduce the variance, turning the multi-modal problem into several uni-modal problems. while these methods are effective at describing the same image multiple times from multiple perspectives, they have not demonstrated an effective approach that generates a single caption covering all of the information in each of the diverse captions. dense captioning methods @xcite @xcite @xcite attempt to generate a full description of all of the objects in the image, however, dense image captions are long and unwieldy, and often contain redundant or repetitive information. similar long-form captions have been explored in the form of paragraph captioning @xcite @xcite @xcite , however in all cases, no efforts have explored using additional post-processing or models to distill the relevant information for alt-text or for downstream applications. in this work, we explore beyond singleview captioning and move towards captions that are short, succinct summaries of the full visual context. a natural way of summarizing and filtering a dense caption, paragraph caption, or set of captions, is with a pre-trained model for summarization. while end-to-end methods for abstractive and extractive text summarization exist @xcite , recently, large language models (llms) such as @xcite , lamda @xcite and palm @xcite have demonstrated remarkable zero-shot performance when performing languageonly summarization tasks @xcite @xcite @xcite , so it is natural that they would be capable of summarizing multimodal information in a zero-shot way. indeed, recently, large-scale vision and language models (vlms) and large-scale language-only models (llms) have revolutionized a number of sub-fields in ai, including in the image captioning space. models such as blip @xcite , ofa @xcite and @xcite have all demonstrated strong performance in single-view image captioning tasks, and indeed, many of these approaches are rated as good or better than human users in some evaluations. ## methods in this work, we introduce a simple framework for visual description, based on a committee generation then summarization process, which we call "image captioning by committee consensus" (ic 3 ). the approach consists of two stages. in the first stage, we leverage a standard pre-trained image captioning model to sample several (potentially noisy) captions using temperature-based sampling from the caption distribution. this generates a representative samples from the caption distribution, each possibly describing different parts of the image. in the second stage, we leverage a summarization model to summarize the information in each of the captions in a short and succinct way that can be presented to a user. an overview of our method is given in figure 2 . ## results & discussion in this section, we compare captions generated using our baseline seed image captioning models, blip @xcite , blip-2 @xcite , and ofa @xcite , to captions generated using ic 3 . we leverage two image captioning datasets for evaluation: ms-coco @xcite and the flickr-30k dataset @xcite (see appendix d.2). figure 3 and appendix g give some qualitative examples of our method compared to several baseline methods. we can see that descriptions using ic 3 are often longer, and contain more detail than their counterpart baseline models. further, most display uncertainty about the content of the image in a natural way, which the baselines are not able to accomplish (see appendix c.2). ## limitations while ic 3 significantly outperforms baseline captioning approaches, as well can outperform single human image captioning references, it also suffers from several distinct limitations. hallucination: while ic 3 often produces high-quality summaries of the associated captions, it has several distinct failure modes, mostly coming down to hallucinations induced by the underlying captioning model. in some cases, objects that are hallucinated by the model can propagate to the summary, even if they are internally inconsistent with other captions in the candidate set k. another distinct failure mode of the captions is when uncertainty in the samples is interpreted as two distinct parts of the image. for example, if 50% of the captions refer to a dog, and 50% of the captions refer to a cat, the model may infer that both a dog and a cat are present in the image, even though only a single, unknown, animal is there. examples of these failure cases are shown in appendix i. we believe that such failure cases can largely be solved by introducing a visually aware summarization model, however, as of writing, no sufficiently large-scale general-purpose multi-modal model exists which is capable of serving this purpose. ## conclusion in this work, introduce ic 3 , a method for image captioning that first samples captions from multiple viewpoints and then summarizes and filters them to create high-fidelity descriptions. as far as we are aware, ic 3 is the first work to demonstrate a pipeline for generating a single caption by integrating distributionally-faithful candidate captions, and does so without changing model architecture or retraining by leveraging summarization to produce a single omnibus caption capturing the full distribution of information. further, ic 3 is the first work for paragraph captioning or image captioning that uses summarization of distributionally-faithful caption samples and the first to demonstrate in human experiments that long-form captions encoding this distribution are preferable to single reference captions. human users rate ic 3 captions at least as helpful as baseline captions up to 80% of the time, and such ic 3 captions are capable of inducing up to 84% relative improvements in recall approaches over baseline captioning methods. while our implementation of ic 3 is relatively simple, it demonstrates significant gains over traditional paradigms, suggesting that this is only the beginning for caption sampling and summary methods. table a3: content coverage and clip recall demonstrating the use of "this is a hard problem" in the prompt, using blip on a 200 element randomly sampled subset of the ms-coco dataset. exact fuzzy clip recall model noun verb noun verb mrr r@1 r@5 r@10 candidates with 0.322 0.216 0.647 0.503 0.770 0.645 0.96 0.99 candidates without 0.316 0.208 0.638 0.496 0.765 0.635 0.94 0.99 references with 0.516 0.308 0.744 0.560 0.833 0.745 0.97 0.995 references without 0.518 0.305 0.745 0.558 0.830 0.745 0.97 0.99 (a) k vs. clip mrr (b) k vs. noun recall the amount of information discovered by the visual model, and generate better representative samples of the input data distribution, however it can increase the context length passed to the large language model, straining the summarization capabilities of the model, and leading to increased cost for the llm. we ablate the choice in figure a1 . here, we can see that increasing the number of captions can lead to increases in automated measure performance (as more captions will capture more information), however more captions can also increase execution time linearly with the number of candidate captions. we can see here that much of the benefit is captured at 10 candidate captions, which we chose for this work, since it represents a good trade-off between execution time, and caption quality. is performance just due to llms correcting caption errors? table table a4 shows both the automated performance of the gpt-3 model, and the human performance of the gpt-3 model for several values of k. we can see that while gpt can help to improve on single captions through error correction (as evidenced by slightly higher scores for gpt-3 (@xmath0. the baseline), the best scores are achieved with higher values of k.