C olor S wap: A Color and Word Order Dataset for Multimodal Evaluation (2024)

Abstract: This paper introduces the ColorSwap dataset, designed to assess and improve the proficiency of multimodal models in matching objects with their colors. The dataset is comprised of 2,000 unique image-caption pairs, grouped into 1,000 examples. Each example includes a caption-image pair, along with a “color-swapped” pair. We follow the Winoground schema: the two captions in an example have the same words, but the color words have been rearranged to modify different objects. The dataset was created through a novel blend of automated caption and image generation with humans in the loop. We evaluate image-text matching (ITM) and visual language models (VLMs) and find that even the latest ones are still not robust at this task. GPT-4V and LLaVA score 72% and 42% on our main VLM metric, although they may improve with more advanced prompting techniques. On the main ITM metric, contrastive models such as CLIP and SigLIP perform close to chance (at 12% and 30%, respectively), although the non-contrastive BLIP ITM model is stronger (87%). We also find that finetuning on fewer than 2,000 examples yields significant performance gains on this out-of-distribution word-order understanding task.

## introduction recent years have seen remarkable developments in pretrained vision and language models @xcite @xcite @xcite @xcite . their performance is exceptional in tasks such as visual questionanswering @xcite ), text-to-image generation and manipulation @xcite , and image captioning @xcite . despite the success, recent work reveals that vision and language models often struggle to comprehend fine grained distinctions in images @xcite and compositional relationships, particularly in differentiating captions with the same words but different word orders @xcite . the winoground dataset from @xcite demonstrates that, given two captions composed of the same words in a different order, the performance of well-known models in correctly matching these captions to their respective images is close to random chance. it is not well understood whether known multimodal architectures are even capable, in practice, of learning how to perform well at a balanced word-order-understanding task with smallscale finetuning data. finetuning has been done on some balanced word-order datasets, but only in settings where half of the images are lacking, as far as we are aware @xcite . to address such questions, we introduce the col-orswap dataset, along with a data generation process that enables the quick creation of a larger dataset fitting the same schema as winoground, as shown in figure 1 . our dataset specifically focuses on a subset of winoground-style examples, emphasizing the swapping of color words in captions for two main reasons: 1) accurately associating colors with objects is of practical importance in ai-generated art, and 2) it is a conceptually simple and targeted word order understanding task -it is very clear what the human judgements are and why. one of the simplifying features of this task is that color words are often directly adjacent to the objects they modify. for example, "someone holding a [yellow umbrella] wearing a [white dress]", although it is important to note that it isn't always this simple: see figure 2 . @xcite when given the caption "the key to the shed is blue" (we ensured that the caption was not rewritten by chatgpt (openai, 2022)). dall•e 3 does not always make this mistake, but it is unreliable. even though "the shed is blue" is a substring, the full sentence is saying that the key is blue. our dataset does not target difficult cases where colors modify far objects in the string. finally, we conduct evaluations of several vision and language models using this dataset. we find that all models, even gpt-4v (openai, 2023a), make a significant number of mistakes. although, contrastive models such as clip and siglip struggle drastically. the dataset could be a useful benchmark for contrastive models in particular, but also for visual language models and even industry diffusion models such as midjourney @xcite . 1 further, we found no model which lacks the capability, in practice, to learn some level of generalizable word-order judgements from a fairly small set of naturalistic finetuning data. ## colorswap in this section, we introduce the colorswap dataset, comprised of 1,000 unique examples created by four expert annotators (each with two or more years or nlp experience and living in the united states) with the aid of generative models. each example includes a caption paired with an image, along with a color-swapped version of the caption and image. we randomly select 700 examples for training and 300 for evaluation. our data collection methodology uses three key steps: 1) caption generation, where we generate a variety of initial captions using three different methods; 2) image generation, which involves employing various text-to-image models to create images corresponding to these captions; and 3) post-processing, which includes human review to ensure accuracy, maintain quality, and re-caption images. post-processing is essential as current text-toimage models often mix up the colors. see figure 1 for an illustration of our data collection process and ## experiments here, we outline our experiments on the color-swap dataset. we evaluate both image-text matching (itm) models and visual language models. all of these models make a significant number of errors in our color composition task, although con-trastive itm models in particular struggle substantially. subsequently, we fine-tune these models using the training split of the dataset, aiming to understand whether minimal tuning can significantly improve their ability to understand word order. ## conclusion we introduce the colorswap dataset, a collection of 2,000 unique image-caption pairs and 2,000 hard negative pairings. it is specifically designed to evaluate and improve minimal compositional color comprehension abilities of vision and language models. our methodology for assembling this dataset involved the use of diffusion models for image generation and the incorporation of human input to ensure naturalness and accuracy. we show that popular off-the-shelf vision and language models exhibit extreme limitations in comprehending even this basic color composition task. however, minimal fine-tuning of these models on the col-orswap dataset significantly improves their basic understanding of word order.