Text encoders bottleneck compositionality in contrastive vision-language models (2023)

Abstract: Performant vision-language (VL) models like CLIP represent captions using a single vector. How much information about language is lost in this bottleneck? We first curate CompPrompts, a set of increasingly compositional image captions that VL models should be able to capture (e.g., single object, to object+property, to multiple interacting objects). Then, we train text-only recovery probes that aim to reconstruct captions from single-vector text representations produced by several VL models. This approach does not require images, allowing us to test on a broader range of scenes compared to prior work. We find that: 1) CLIP’s text encoder falls short on more compositional inputs, including object relationships, attribute-object association, counting, and negations; 2) some text encoders work significantly better than others; and 3) text-only recovery performance predicts multimodal matching performance on ControlledImCaps: a new evaluation benchmark we collect and release consisting of fine-grained compositional images and captions. Specifically, our results suggest text-only recoverability is a necessary (but not sufficient) condition for modeling compositional factors in contrastive VL models. We release our datasets and code.

## introduction clip text encoder clip image encoder an iguana t5 text decoder (finetuned) a happy dinosaur a surfer carrying a lifeguard an orangutan eating and an officer flying an iguana an amusing dinosaur a lifeguard carrying a surfer an orangutan and an officer eating an orangutan ❄ ✗ ✗ ✓ ✓ (unused) figure 2 : we probe the representations of single-vector text encoders used in popular vl models. using a corpus of increasingly compositional image captions, compprompts, we attempt to generatively decode the original input sentence. text encoders of popular models like clip fail to effectively encode precise aspects of their captions like attribute attachments and object relationships (real examples shown, as in figure 1 ). our strategy is as follows: for a fixed pooled text encoder t : @xmath0, which maps from captions x to vectors @xmath1, we test how accurately x can be recovered by an expressive generative decoder given y, i.e., p(x|t (x)). in an ideal case, t should result in no information loss, i.e., an exact reconstruction of the original caption should be possible, to account for specific visual factors. however, we hypothesize that if a specific visually descriptive property (e.g., a spatial relation) cannot be accurately decoded from y (using a decoder trained with significant supervision), then it is unlikely a multimodal model can effectively use that property of x using t . different from existing probes, ours does not require images, enabling exploration of a broader range of captions, e.g., creative text-to-image queries for which there may be no associated image (like "a penguin on mars..."). we execute our probe using an increasinglycompositional hierarchy of image captions we curate, compprompts, which covers cases ranging from a single object with no attributes (e.g. "a cat") to multiple objects with attributes and relations (e.g. "an orange cat to the left of a dog"). we also test counting (e.g. "three cats") @xcite and negations (e.g. "a cat that is not yawning"). we compare five text encoders, and find that top contrastive vl models: (1) are broadly ineffective at textually encoding spatial relations, numbers, and negations; (2) frequently cannot match attributes to their corresponding objects; and (3) fail more as inputs grow more compositional. while some text encoders perform significantly better than others, all underperform a proof-of-concept model which demonstrates that our prompts can indeed be compressed into single vectors with little information loss. in order to verify that our text-only probe predicts performance in multimodal settings, we curate an evaluation set of image-caption pairs, con-trolledimcaps, which operationalizes the compositional factors of compprompts in a multimodal setting. results on this corpus suggest our textonly probe gives a necessary condition: if the text-only recovery probe fails to recover a textonly property on compprompts, then the associated multimodal model also performs poorly for that property on controlledimcaps. however, our results also suggest that text-only recoverability is not a sufficient condition: a model can achieve low text-only information loss on a particular prompt type but not fully solve it on con-trolledimcaps. to facilitate future probing experiments, we release our code alongside the newly collected compprompts and controlledimcaps corpora at https://github.com/amitakamath/vl_ text_encoders_are_bottlenecks . ## text-only recovery for a given text encoder t , our first step is to obtain a training corpus of representations to fit a decoding probe p(x|t (x)). we use (just the text of) conceptual captions 3m @xcite (cc3m) split into a 90/10 train/val set; this corpus consists of cleaned alt-texts from web images, and thus is similar to the pretraining corpora of many vl models. for p, we use t5-large: specifically, we condition the decoder on t (x), followed by a linear transformation and layer normalization. we train using adafactor @xcite with proof-of-concept5 1024 92.9 a batch size of 512 for 4 epochs over cc3m; we select checkpoints with the lowest val loss. models are trained using 4xa6000 gpus with 48gb of memory using transformers @xcite and accelerate foot_2 . at evaluation time, we generate captions for compprompts set using @xmath2. text models. we evaluate several t models: clip vit-b/32 (12 layers, 512 dim) and vit-l/14 (12 layers, 768 dim) @xcite , clip with a roberta-pretrained text encoder @xcite , and yuksekgonul et al. ( 2023 )'s more-order-aware clip encoder finetuned with hard negatives, negclip. for comparison, we also consider the uni-modal sen-tencebert (reimers and gurevych, 2019) model all-mpnet-base-v2, which is trained on several sentence similarity datasets including coco captions @xcite . ## experiments and results in the multi-modal setting we investigate the hypothesis that if a textual property cannot be decoded from the vl text encoder's vector representation with a highly expressive decoder (like t5-large), then it also cannot be readily modeled in the multimodal setting. controlled-imcaps studies the attributes from compprompts in the multimodal setting. we then compare the text type comparable prompts in x example from x spatial-1 1 unique object + 1 1-obj preposition (left/right only) spatial-2 2 unique objects + 1 2-obj preposition (left/right only) temporal 1 unique object + temporal relation of a 1-obj verb, 2 unique objects + temporal relation of a 1-obj verb, 2 unique object + temporal relation of a 2-obj verb verb-1 1 unique object + 1 1-obj verb verb-2 2 unique objects + 1 2-obj verb adjective 1 unique object + 1 adjective a cat on the right a cat on the left a person to the right of a horse a person to the left of a horse a dog before catching a frisbee a dog after catching a frisbee a bird sitting a bird flying a person feeding an elephant a person riding an elephant a blue fire hydrant a white fire hydrant encoder performance of a vl model on a particular prompt type in compprompts with the performance of the overall vl model on that prompt type in controlledimcaps. as discussed in section 2.2, the two captions in every example differ by only one word which changes or inverts the relation, allowing us to perform fine-grained analyses in controlled settings without conflating multiple types of compositional reasoning. figure 3 depicts the six types of attributes studied in controlledimcaps, their corresponding prompt type in compprompts, and an example of each. vl models. we evaluate the same vl models as in section 3: clip vit-b/32, clip vit-l/14, clip with a roberta-pretrained text encoder @xcite , and negclip @xcite . each of these models can return a score when given an image and a caption, representing how well they match. evaluation. we follow the evaluation scheme from winoground @xcite : for a given pair of images with corresponding captions, we measure both a text score, the fraction of instances where a model scores the correct caption higher than the incorrect caption when given an image, and an image score, the fraction of instances where a model scores the correct image higher than the incorrect image when given a caption. ## related work building models capable of reasoning jointly about visual and textual inputs is a long-standing goal of ai @xcite , with potential applications in the fields of vision-language navigation @xcite , human-robot interaction @xcite , accessible image captioning @xcite , etc. recent challenge datasets have been designed to probe the capacity of multimodal models to represent descriptions of precise visual compositions @xcite @xcite . @xcite and @xcite study clip specifically, demonstrating its shortcomings (and some potential fixes) in terms of modeling syntax. @xcite study openclip models for various types of compositional reasoning, with programmatically sourced hard negatives. different from these works, our textual probe does not require access to images. our image-and-text evaluation most closely resembles @xcite . however, we stratify the examples based on type of input (e.g., temporal relations) to provide more detailed insights. we also keep our prompts relatively simple, never having more than two objects or two attributes in the input. we believe this is a more realistic goal for our current vision-language models. the word order shuffling aspect is also discussed in @xcite . however, as their proposed benchmark does not provide pairs of images with corresponding captions, it is possible to achieve state-of-the-art with a text-only model (specifically, 2-shot chatgpt foot_5 @xcite , details in appendix and the recent @xcite ). while this does not detract from their finding that visionlanguage models ignore word order, our benchmarks have an additional advantage of being insensitive to text-only priors. ## conclusion and discussion we present probing results that suggest significant information loss upon text encoding of compositional inputs in vision and language models. this information loss is quantified using compprompts, a test set of increasingly compositional image descriptions, and controlledimcaps, a test set that we use to verify that this information loss affects the performance of multimodal models on compositional inputs. harder negatives, more text pretraining, and larger models all improve encoder quality, but information is still lost even for the most performant models, compared to the unimodal sbert as well as a t5-based auto-encoder. going forward, even more difficult test sets than compprompts and controlledimcaps might be required to analyze and evaluate vision-language model capabilities. returning to riedl (2022)'s tweet from the intro, "a penguin on mars wearing a spacesuit and walking a robot dog next to santa claus.", even our highly accurate proof-of-concept5 model struggles, predicting: "compulsory penguin onexposition wearing a spacesuit and walking a dog robot next tohoc". to support imaginative text-to-image generation queries (for images that may not exist yet), future work would be wellsuited to design text encoders that can generalize to captions that contain compositions never-beforeseen in web alt-text corpora. our probing results suggest two future modeling directions: (1) modifying contrastive vl models' training objectives to additionally encourage ability to reconstruct the text input, either through an additional reconstruction loss on the text encoder during finetuning, or through the addition of even harder negatives than @xcite and @xcite , would be an exciting avenue for future work. alternatives to contrastive training, such as captioning, have also shown promise in recent work @xcite ; and (2) explicitly encouraging linear recovery with a modified loss function: while the gap between sbert and the vl text encoders can be partially explained by the superior pooling method and training data, sbert's training objective does not require linear recoverability (whereas clip's dot product interaction term might): explicitly encouraging linear text-text recoverability might improve multimodal performance. finally, we hope that controlledim-caps can facilitate research beyond single-vector bottleneck vl models.