Rainbow - A Benchmark for Systematic Testing of How Sensitive Visio-Linguistic Models are to Color Naming (2024)

Abstract: With the recent emergence of powerful visio-linguistic models comes the question of how fine-grained their multi-modal understanding is. This has lead to the release of several probing datasets. Results point towards models having trouble with prepositions and verbs, but being relatively robust when it comes to color.To gauge how deep this understanding goes, we compile a comprehensive probing dataset to systematically test multi-modal alignment around color. We demonstrate how human perception influences descriptions of color and pay special attention to the extent to which this is reflected within the predictions of a visio-linguistic model. Probing a set of models with diverse properties with our benchmark confirms the superiority of models that do not rely on pre-extracted image features, and demonstrates that augmentation with too much noisy pre-training data can produce an inferior model. While the benchmark remains challenging for all models we test, the overall result pattern suggests well-founded alignment of color terms with hues. Analyses do however reveal uncertainty regarding the boundaries between neighboring color terms.

## introduction visio-linguistic models, which jointly process image and text, have started to yield increasingly promising results on tasks such as visual question answering @xcite , image captioning @xcite , and image-text retrieval @xcite . with their success come efforts of probing the depth of alignment between the two modalities. a frequently used approach that is visualized in figure 1 is to use two minimally different descriptions of the same image. while one description matches the image, the other does not. the task is for a model to compare the descriptions to the image, ideally accepting the matching description and rejecting the wrong one. such probing has revealed different abilities across linguistic categories. results are more promising for categories that are more strongly associated with certain image areas, such as nouns, than for other aspects, such as prepositional information or verbs @xcite . in this paper, we focus on color as an important visual aspect that is central to image description and immediately associated with certain image areas. when it comes to visio-linguistic alignment around color, results are reasonably promising @xcite @xcite , although in probing datasets, color is often merged into a more general category of attribute understanding @xcite . this prohibits straightforward performance evaluation on color probes alone. to provide a benchmark that is tailored specifically towards color, we release rainbow , which collects suitable image-text pairs from existing data and enriches them by systematically varying the color words in the textual descriptions. the resulting probes are split into two categories, which are visualized in figure 1 . in the color replacements, color mentions are systematically replaced with alternative colors. this allows to assess whether a consistent performance pattern emerges, such as results for neighboring color pairs differing from results for complementary ones. to gauge whether models are not just checking for the overall presence of the colors mentioned in a description, we also include color swaps. we derive these from descriptions that contain exactly two color mentions: swapping the two colors creates misaligned descriptions where models have to check whether the respective objects appear in the color they are described as. we provide a language-only baseline for the probes in rainbow and test a set of eight visiolinguistic models to get an impression of how well current models are able to solve them. in addition, we enrich a subset of rainbow with the rgb values corresponding to the color names in the descriptions. this permits analysis of how human perception influences color naming and how the actual hue is related to classification decisions of models. we release rainbow publicly and make all our code available. 1 as this work focuses on color, it is our responsibility to nonetheless ensure accessibility to the extent possible: we include rgb values and names of colors instead of merely relying on the depiction of the hues we discuss. where color is used to encode information, we use shades that are distinguishable for people with deuteranopia. ## probing visio-linguistic models with the growing success of visio-linguistic models comes interest in their inner workings and limitations. a promising approach to probing their understanding is to design contrast sets @xcite . their idea is to create minimally different examples that fall in different classes and thus test the decision boundary of a model. for probing visio-linguistic models, this usually means to alter a word in a description of an image, thereby deriving a mismatched description. such datasets have been designed to target nouns @xcite , numbers @xcite , verbs @xcite ; hendricks 1 https://github.com/mariebexte/vl-probing @xcite and a number of other linguistic categories @xcite @xcite . some of these datasets include probes around attributes, among which are color probes @xcite . since these are however often mixed in with probes targeting other attribute types, they do not permit a dedicated estimate of performance regarding color. vl-checklist @xcite includes such a set of probes dedicated to color alone, and in their experiments model performance on these probes is higher than for the other attribute types they test, such as size or material. this is in line with other previous work on visiolinguistic model performance around color: tasking a model with editing an image so that the response to a color-focused question about this image changes @xcite works better than asking for alterations targeting size @xcite . similarly, classification models that predict a masked-out color word based on a visio-linguistic embedding yield promising results @xcite . what these rather technical approaches do however not take into account is how the probed aspects are processed in human perception. in this vein, @xcite assess the extent to which human processing of number information is mirrored in visio-linguistic models. in this work, we take a similarly humaninformed approach. we aim to systematically assess color alignment in visio-linguistic models by creating an extensive collection of probing examples. during evaluation, we take into account the relations between colors and the influence of human perception on color naming. ## color naming computers process color with the objectiveness of a machine: in the rgb system of the digital world, colors are combinations of intensities of red, green, and blue. the textual data processed by visio-linguistic models does however consist of descriptions of these hues in human language. models therefore have to pick up on how color words align with certain shades. this is complicated by three factors rooted in the human perceptual system and the language used to describe color: context, coverage, and subjectivity. while we do not experience it as such, we do not perceive every shade as its exact value. a key driver in this is context: the perceptual system demonstrates a number of constancy phenomena. one of these is color constancy, which makes the same sheet of paper appear white both in direct sunlight and a dimly lit room @xcite . we are thus applying a sort of color correction to account for the overall setting. this helpful quirk does however make us susceptible to illusions (see @xcite for an overview). one such illusion is depicted in figure 2 : the surrounding colors make the center square appear more yellow on the left and more orange on the right, even though both are the same exact shade. this means that in visiolinguistic datasets the same hue will not necessarily always be described the same, because its context may cause it to be perceived differently. another relevant dimension of context is the interplay of a color word and the object it refers to: white skin has a different shade of white than a white shirt, just like a red wine is a different red than red hair. in general, the complementary rgb color pairs blue-yellow and red-cyan align well with how color is processed by the human perception system @xcite , but the coverage of the rgb space with color terms varies. @xcite formulate a set of eleven basic colors, which they postulate to follow a certain order of emergence across languages. some languages do not use all eleven colors (bornstein, 2007) and others have separate terms for different shades of the same basic color @xcite , but these eleven basic colors match the common color terms in the english language. these terms are not equidistant on the rgb color wheel (see figure 3 fective than for cool colors @xcite . @xcite postulate that this emerged out of a necessity to discriminate objects from background, finding that objects tend to have warm and backgrounds cool colors. the uneven coverage of the rgb space with color names means that a visio-linguistic model has to notice how variations in the rgb values matter more in some areas than in others: while there is a rather restricted orange area, it is a much wider range of values that can be described as blue. a third aspect that skews human color descriptions is that there is a certain level of subjectiveness to them: some hues may permit multiple descriptions, or even cause disagreement regarding the appropriate color word to describe them. an example of this is the tank top of the woman in the exemplary flickr30k image in in summary, learning the association between rgb values and color names is not a straightforward case of discretizing equal-sized subareas of the color space into a set of vocabulary. color naming is influenced by the context in which a shade occurs, the language a person speaks and may at times be ambivalent. to assess these effects on crowdworker's descriptions of images, we manually enrich one of the subsets of rainbow with rgb values of the described elements. ## rainbow we now describe rainbow , our benchmark to systematially test the sensitivity of visio-linguistic models to color naming. we start by describing the underlying datasets. then we outline how these datasets were processed to collect descriptions that mention color, from which we derive two kinds of probes: color replacements and color swaps. ## analysis we first analyze the shades corresponding to color names in flickr30k, then establish a languageonly baseline, and finally probe a number of visiolinguistic models. ## conclusion we present rainbow , a benchmark to probe color understanding of visio-linguistic models. our annotation of one of its subsets with rgb values demonstrates how human perception influences color naming. testing a set of visio-linguistic models showed them to pick up on which color terms describe neighboring hues. swapping colors in descriptions revealed an inferior ability of models that rely on pre-extracted image features to ground color names to the objects they describe. since the benchmark remains challenging for all models we probe, it will be exciting to see how future models fare on it.