I mage I n W ords: Unlocking Hyper-Detailed Image Descriptions (2024)

Abstract: Despite the longstanding adage ”an image is worth a thousand words,” generating accurate hyper-detailed image descriptions remains unsolved. Trained on short web-scraped image-text, vision-language models often generate incomplete descriptions with visual inconsistencies. We address this via a novel data-centric approach with ImageInWords (IIW), a carefully designed human-in-the-loop framework for curating hyper-detailed image descriptions. Human evaluations on IIW data show major gains compared to recent datasets (+66%) and GPT-4V (+48%) across comprehensiveness, specificity, hallucinations, and more. We also show that fine-tuning with IIW data improves these metrics by +31% against models trained with prior work, even with only 9k samples. Lastly, we evaluate IIW models with text-to-image generation and vision-language reasoning tasks. Our generated descriptions result in the highest fidelity images, and boost compositional reasoning by up to 6% on ARO, SVO-Probes, and Winoground datasets. We release the IIW-Eval benchmark with human judgement labels, object and image-level annotations from our framework, and existing image caption datasets enriched via IIW-model.

## introduction today's state-of-the-art vision-language models (vlms) are trained using large, noisy web datasets. webimagetext @xcite , align @xcite , conceptual captions @xcite and laion @xcite rely on alt-text scraped from the internet as an imperfect image caption. yet alt-text may only mention the photo location (e.g. "europe"), the camera model used (e.g. "canon eos r6 mark ii"), or is seospecific (e.g., "keep calm and carry on"). while data filtering and post-processing can remove noisy text, alt-text ambiguously captures image content or intent (wikipedia contributors, 2023a). therefore, only using image descriptions from the web is fundamentally flawed and limits model capabilities @xcite @xcite @xcite . to curate better image-text data, recent work has released dense human written (docci @xcite , dci @xcite ) or model generated caption datasets (pixlore @xcite , dac @xcite ). both have limitations, as using annotators without comprehensive guidelines results in outputs that vary by human attention, bias, and effort @xcite @xcite . in contrast, model-generated captions are cheaper but incomplete and rife with hallucinations @xcite . in this work, we describe imageinwords (iiw), a human-in-the-loop framework for curating hyperdetailed image descriptions, and its resulting annotations. iiw combines the irreplaceable quality of human annotators with seeded metadata from machine generations. first, a vlm generates granular captions for each object in the image to seed our human annotation process, where crowd workers augment and fix the object-level captions to make them richer and hallucination free. next, at the image-level, a vlm generates a global caption to seed the final image description. crowd workers consume the image-level seed caption and object-level human annotations to fill in contextual gaps. we design guidelines to attend to concepts beyond objects, such as visual perspective, spatial arrangement, and human object interactions. to ensure quality, multiple annotators iterate on a sample sequentially and we also incorporate active learning to produce better vlm seeds (fig. 1 ). with this process, we construct the iiw dataset of 9018 hyper-detailed image descriptions. we find iiw has richer statistics than prior dense description datasets, with an average of 217.2 tokens, 52.5 nouns, 28 adjectives, 5 adverbs, and 19.1 verbs (tab. 1). we assess quality with human side-byside (sxs) comparisons to human-written datasets (dci, docci) and gpt-4v. our descriptions are rated as more comprehensive, specific, human-like, with fewer hallucinations and better leading sentences at an average of +66% (dci, docci) and +48% (gpt-4v). we then fine-tune with iiw data and evaluate generated descriptions with the same sxs rubric: iiw model outputs are better by +31% compared to models fine-tuned on prior work. to better understand iiw models, we also perform text-to-image generation and vision-language reasoning experiments. images generated with our model's descriptions are considered a closer reconstruction to the original image than when using other models. for vision-language compositionality, we replace images from aro @xcite , svo-probes (hendricks and @xcite and winoground @xcite datasets with generated descriptions. iiw model descriptions help to better reason over attributes, relations, and word order compared to llava-v1.5 and instructblip descriptions. in summary, our contributions include: we also release human sxs labels between iiw, dci, and docci for comparison in future work. ## related work however, few dense image description datasets exist. pixlore @xcite used multiple visionlanguage datasets to generate verbose captions with blip-2 @xcite . dac @xcite uses a machine-generated approach: pretrained llms expand the original image caption and pretrained vlms generate captions over smaller image regions. the resulting descriptions are used to fine-tune a vlm model for better compositional reasoning. while model-only approaches are cost effective and avoid the challenges of designing annotation instructions, they risk introducing hallucinations and systematic biases. docci @xcite collects image descriptions with only crowd workers, which we later show can be considerably improved. closest to iiw is dci @xcite , which uses human annotators to reach denser descriptions. dci uses the sam @xcite object detector to generate smaller regions to be described and then composes them into an overall description. dci's available annotations and metadata can be concatenated with additional text to reach 1k+ length. however, filler text and image labels are used to reach this length, and repeated or highly overlapping sentences are often present. as a result, we use their "extra_caption" field for fair comparison as it is the only coherent description available. in contrast to dci, we also allow crowd workers to update or correct every component of the seeded information. iiw output is then sequentially refined over multiple annotation rounds to produce a single coherent annotation. in comparison to dci's "extra_caption" annotation, we collect significantly better descriptions, as reflected in tab. 1 statistics. ## imageinwords dataset collection the iiw dataset is composed of 9018 (train: 8573, test: 445) images that are sampled from a we-bli @xcite like dataset and human annotated. details on the human annotator pool are provided in appendix b.1. in 3.1, we briefly review our foundational guidelines for crowd workers. annotation methodology and the types of image-text annotations we collect are described in 3.2 and 3.3. ## iiw human-authored data eval to evaluate the iiw annotation framework and resulting human annotations, we start with human sxs evaluations to compare our human annotations to prior work (e.g. dci, docci, gpt-4v). to run a sxs experiment on human-authored description quality, we first need a common pool of human annotated images. for this, we additionally annotate the dci test set (112) and a comparable number of samples (100) from the docci test set with our iiw annotation framework. we thus have humanauthored iiw annotations for direct comparison on images in the dci and docci datasets, which contribute to our open-source iiw-eval benchmark. our human sxs framework evaluates 5 metrics: comprehensiveness, specificity, hallucinations, quality of the first few line(s) as a tldr (too long didn't read; meant to serve as a succinct summary), and human-likeness. comprehensiveness concerns whether a description covers all key information and objects present in an image. specificity is the degree of detail in which each of we also include tldr quality as one of our metrics as initial sentences set a precedence for what details to expect, both for the reader and models trained on this data. from a practical perspective, we would like hyper-detailed descriptions to still be useful in a setting that is constrained by input text length; i.e., if we truncate an image description, it should contain the most salient information for vision-language training. while iiw guidelines instruct annotators to include a first sentence which provides an overall summary of the image content, prior work also designed their descriptions to start with either a short caption that summarizes the full image @xcite or have important information covered in earlier sentences @xcite . as a result, we believe the tldr metric is reasonable and should be an established practice for hyper-detailed descriptions moving forward. the evaluation is done on a 5 point scale defined using "substantially better" (+ +) or "marginally better" (+) ratings on both sides of a "neutral" (-). higher numbers indicate higher quality across each metric, and our tables report percentages for ease of comparison. we emphasize that this is an extremely challenging human annotation task, where per image, two text pieces of 100+ words need to be evaluated across 5 metrics in a sxs setting. on average, we observe each comparison takes 15-20 minutes. details on the annotation setup and ui are in appendix b.4. ## iiw model evaluation after evaluating iiw human annotations, we turn to quantifying the impact of fine-tuning with iiw data versus fine-tuning with prior work. we finetune separate pali-3 5b models on dci, docci and iiw training splits, with their detailed humanauthored text as target. each model is trained with an identical setup (∼40 epochs, learning rate 3e-4, batch size 32) and the generic input instruction: "generate a detailed image description." more finetuning details are provided in appendix c and d. as shown in prior work, existing text similarity metrics like bleu @xcite and rouge @xcite have been shown to poorly correlate with human judgement as they are heavily dependent on n-gram overlaps, and thus ill-suited for long texts @xcite . prior works dac, dci, and docci also are limited by existing image caption metrics, and use llm summaries of their descriptions or human sxs for evaluation. we report bleu, rouge, cider, bertscore @xcite , and bleurt @xcite in appendix d.5 but look to human sxs for more accurate judgements. ## iiw-eval benchmark release we release the iiw-eval benchmark (tab. 6) of human-and model-annotated image descriptions, human sxs results on human-human and model-human pairs of descriptions. iiw-400 is a new eval set of 400 images randomly sampled from docci-aar @xcite . we re-annotate dci and docci test samples and enrich two existing datasets with new iiw descriptions: localized narratives (locnar (pont-tuset et al., 2020)) and crossmodal-3600 (xm3600 @xcite ). we provide locnar and xm3600 annotations with significantly improved quality (see statistics in appendix e). the model generated descriptions may have hallucinations, information recall losses, or non-human like writing style artifacts. by releasing this subset along with human sxs judgements, we encourage the development of new metrics and evaluation systems to detect them in an automated, scalable manner. it also promotes fair comparison across methods in future work. the dataset is released under a cc by 4.0 license. ## future work in future work, robust and effective automatic metrics are needed to evaluate the quality of detailed image descriptions. next steps may include training model-based metrics or preference models (i.e., autoraters) with human preference data to learn a global quality metric. for additional analysis, we could further break down our current sxs metrics. for example, the human sxs hallucination metric could be broken down to capture fine-grained categories like how many hallucinations are with respect to color, size, or spatial location. we are working to extend the imageinwords framework to additional languages and geographically diverse images. in next steps, we note that images need to be sampled globally (across both geographic and cultural identity); this sampling must also be done across different image topics and categories, making equal coverage more complicated. we are currently working on adapting our proposed framework to accommodate locale specific annotators, which are required for cultural specificity. our continued goal is to make the annotation guidelines holistic, reduce human effort and dependency in the annotation process, and help shift the narrative from captions to descriptions. ## conclusion in this work, we proposed imageinwords (iiw), a new framework for hyper-detailed image descriptions. our annotation guidelines and seeded, sequential annotation process lead to human authored descriptions that are strongly preferred over both prior work's human annotations (+66%) and prior work's fine-tuned models (+31%). images reconstructed with iiw generated descriptions were ranked 1st more often, regardless of how much of the image description was used, reflecting higher saliency earlier and better overall quality. our compositional reasoning evaluation showed iiw generated descriptions to best contain fine-grained visual detail needed to decipher true from false visual attributes and semantics, with accuracy gains of up to 6% over our most performant baselines. our results collectively demonstrate the quality and utility of iiw image descriptions as state-of-the-art.