ICC : Quantifying Image Caption Concreteness for Multimodal Dataset Curation (2024)

Abstract: Web-scale training on paired text-image data is becoming increasingly central to multimodal learning, but is challenged by the highly noisy nature of datasets in the wild. Standard data filtering approaches succeed in removing mismatched text-image pairs, but permit semantically related but highly abstract or subjective text. These approaches lack the fine-grained ability to isolate the most concrete samples that provide the strongest signal for learning in a noisy dataset. In this work, we propose a new metric, Image Caption Concreteness (ICC), that evaluates caption text without an image reference to measure its concreteness and relevancy for use in multimodal learning. Our unsupervised approach leverages strong foundation models for measuring visual-semantic information loss in multimodal representations. We demonstrate that this strongly correlates with human evaluation of concreteness in both single-word and caption-level texts. Moreover, we show that curation using ICC complements existing approaches: It succeeds in selecting the highest quality samples from multimodal web-scale datasets to allow for efficient training in resource-constrained settings.

## introduction pre-training large vision-language models (vlms) on web-crawled datasets consisting of imagecaption pairs has become the standard practice in achieving state-of-the-art results in vision-andlanguage tasks such as image captioning and multimodal representation learning. however, raw web data are often noisy and contain many low-quality samples, which impair vlms' learning in terms of quality and efficiency @xcite @xcite . while various factors impact data quality, we focus on semantic noise, characterized by analyzing the meaning of data items rather than, e.g., identifying low resolution images or quantifying token repetitions. figure 1 : given an image caption, icc measures its visual concreteness. we show samples from ms-coco @xcite illustrating captions annotated by different annotators with low (↓) and high (↑) icc scores. as seen above, our method successfully differentiates between concrete and abstract or subjective captions, even for high-quality datasets such as ms-coco. this is done by quantifying visual-semantic consistency using multimodal foundation models. existing datasets are commonly filtered using vlms such as clip @xcite to identify image-text semantic misalignments @xcite , i.e. captions irrelevant to their images; using rule-based proxies such as measuring the complexity of captions via semantic parsing @xcite ; or removing images that contain text that overlaps with the caption @xcite . however, these approaches fail to identify captions that are highly abstract and may contain subjective, non-visual information, despite being semantically aligned with the image and having a sufficiently complex grammar. figure 1 shows examples of such imagecaption pairs. a caption such as "it does not look like something i would want to eat" is semantically related to the image, yielding high clip similarity, but contains subjective details which provide a confounding signal when training vlms (see also figure 2 ). a model trained to generate such captions from images may learn to hal-lucinate details, e.g., liking a certain type of food in our example, which are not visually grounded and are highly subjective. similarly, such imagecaption pairs provide a weaker signal for representation learning than images with visually concrete captions (e.g. "a sandwich sits on a small blue plate"), which may impede the learning processparticularly in a resource-restricted setting where data or compute is limited. thus, we suggest filtering image captions by their visual concreteness, referring to the extent to which a text describes visual aspects of a scene in a manner that can be vividly imagined @xcite 1 . this contrasts with abstract text, which may correspond to many possible visual interpretations or include subjective information. we show that this new dimension of textual quality enables selecting imagecaption pairs that provide a strong supervision signal for vision-and-language tasks, particularly in resource-constrained settings where training directly on noisy web-scale multimodal data fails to converge to a satisfactory solution in a limited number of iterations. we propose the image caption concreteness (icc) metric for quantifying the visual concreteness of image captions calculated from text alone, i.e., without an image reference. we measure concreteness using unsupervised autoencoding pipelines with visual-semantic information bottlenecks. specifically, we use a visual-bottleneck autoencoder that leverages text-to-image generative models' competence and a semantic-bottleneck autoencoder that identifies how well a large language model (llm) recovers the input caption from its semantic clip embedding. as these models require costly inference through large generative models, they cannot feasibly run on a large scale; therefore, our icc metric is distilled from these pipelines, enabling fast, computationally-efficient inference. in our experiments, we demonstrate that when dealing with limited training iterations, employing icc for filtering multimodal datasets leads to enhanced performance in image captioning and representation learning. moreover, our results indicate a strong correlation between icc and both 1 some works have treated this as roughly synonymous with imageability (visual association), while others use concreteness to refer more generally to association with sensory experiences of all types @xcite . our work focuses on the visual modality. while these captions are semantically related to the images, they are abstract or contain subjective non-visual information that, unlike icc, clip fails to detect. stated explicitly, our contributions are as follows: (1) we propose the icc metric distilled from foundation vlm models with a novel combination of unsupervised autoencoding pipelines; ## method given an image caption (of an unseen image), we aim to predict its degree of visual concreteness. our underlying assumption is that more visually concrete text can be mapped to a visual representation with less information loss. conversely, we expect that visually abstract or subjective text cannot be converted to or from a visual representation without significant information loss, since it does not clearly describe a well-defined image. as an example, consider the text "wheat bread is always the healthy choice for lunchtime" in figure 2b . the notion of wheat bread being a healthy choice is inherently non-visual and is unlikely to be directly depicted in an image. therefore, this information is likely to be lost in an autoencoding process that includes an image as the bottleneck, when the encoded image is decoded back to the textual modality. we first acquire training data using a semantic-bottleneck autoencoder (sba, top left) and an visual-bottleneck autoencoder (vba, bottom left). we then distill a weighted combination of their reconstruction scores into a smaller language model (lm, right), which learns to produce icc scores for new text. we visualize reconstruction scores for highly concrete ("a black dog") and highly abstract ("a nice location") text. high and low scores are colored in green and red, respectively. our final score, which combines the two pipelines, yields more accurate concreteness predictions than each of them. we model this effect with multimodal autoencoders @xcite . in our setting, we use multiple autoencoder components that convert text to and from visual-semantic representations using foundation vlms, and quantify the information loss of this process as a proxy for visual concreteness. while these autoencoders provide a strong signal, they are composed of slow, computationally-intensive large generative models making inference infeasible on a large scale. therefore, we distill their scores into a small model which allows for an efficient calculation of the icc scores. we proceed to describe our proposed visualbottleneck autoencoder and semantic-bottleneck autoencoder components, and their distillation into the final icc metric. see figure 3 for an overview of our full pipeline. ## results we turn to show icc's benefit in data curation for downstream tasks (section 3.1), followed by its correlation to human judgement (section 3.2). ## conclusion we present a new metric for measuring the visual concreteness of image captions without an image reference. by leveraging strong foundation models, we quantify visual-semantic information loss in an unsupervised manner and find that this highly correlates with human concreteness judgments. our results demonstrate that icc is effective at selecting a core of high-quality image-caption samples from web-scale multimodal datasets for training models in the resource-constrained setting. we foresee the use of icc in additional tasks requiring the curation of web-scale multimodal data, where high-quality, visually-concrete text is needed.