Measuring Progress in Fine-grained Vision-and-Language Understanding (2023)

Abstract: While pretraining on large-scale image–text data from the Web has facilitated rapid progress on many vision-and-language (V&L) tasks, recent work has demonstrated that pretrained models lack “fine-grained” understanding, such as the ability to recognise relationships, verbs, and numbers in images. This has resulted in an increased interest in the community to either develop new benchmarks or models for such capabilities. To better understand and quantify progress in this direction, we investigate four competitive V&L models on four fine-grained benchmarks. Through our analysis, we find that X-VLM (Zeng et al., 2022) consistently outperforms other baselines, and that modelling innovations can impact performance more than scaling Web data, which even degrades performance sometimes. Through a deeper investigation of X-VLM, we highlight the importance of both novel losses and rich data sources for learning fine-grained skills. Finally, we inspect training dynamics, and discover that for some tasks, performance peaks early in training or significantly fluctuates, never converging.

## introduction fine-grained multimodal skills (e.g., understanding relationships and recognising verbs) require identifying and relating various entities across both image and text modalities. vision-and-language models (vlms) need such skills to robustly perform well on real-world vision-and-language (v&l) applications; e.g., a coarse-grained model tested on image retrieval to "find an image where something is on a sofa" might incorrectly return an image of a cat sitting below the sofa. as another example, in captioning, a model might incorrectly describe an image where "someone is selling a sweater" as "someone is buying a sweater," if it does not have a precise understanding of the two verbs. however, common v&l benchmarks (e.g., @xcite @xcite do not explicitly shed light on such fine-grained understanding. indeed, in the last few years, there has been an increase in the number of benchmarks which demonstrate that current, coarsegrained models struggle with fine-grained understanding @xcite @xcite . meanwhile, more models have been designed specifically to learn a better mapping between visual and textual modalities (e.g., @xcite @xcite . while such models perform well on coarse-grained retrieval and other downstream tasks, they have not been directly evaluated on fine-grained understanding. consequently, it is unclear if the performance gains are due to tighter, more fine-grained representations introduced by model innovations at the pretraining stage. to fill this gap, we analyse several recent models with innovations designed for a better image-text alignment and their corresponding baselines on a suite of fine-grained benchmarks. we centre our study on three key questions. first we consider: which models perform well on fine-grained tasks? to answer this, we evaluate models from four different model families trained with different amounts of pretraining data, as well as recent architectures that leverage frozen large language models (llms). we observe that modelling innovations have more impact than simply scaling image captions from the web. furthermore, explicitly modelling localisation can improve performance, but it is crucial how it is done, and simply using localisation data is not enough. our observations motivate our next question: how do data and losses impact fine-grained understanding? we focus our study on the best performing model, x-vlm @xcite , which learns to map specific objects and regions (not a full image) to a label (word or phrase describing the region). we reformulate the x-vlm loss to better disentangle the contribution of data and losses, observing that more data does not improve performance unless paired with losses designed to learn a mapping between regions and labels. furthermore, the diversity of class labels is important for performance on coarse-grained retrieval, and region descriptions (as opposed to single word labels) are crucial for performance on fine-grained tasks. finally, it is unclear if all fine-grained skills are learned at the same time during training so we consider: how does fine-grained understanding evolve during training? surprisingly, we find that while performance steadily improves on coarse-grained retrieval tasks through training, performance fluctuates substantially on many fine-grained tasks, with some skills, like counting, becoming increasingly worse. additionally, performance across different fine-grained tasks that should test for similar skills are not always well correlated. contributions. in this work, we 1) provide indepth analyses of how data and modelling decisions impact performance on fine-grained tasks, and 2) further disentangle the gains given by data and pretraining losses on our best performing model (x-vlm). our results suggest that to make progress in fine-grained understanding, modelling innovations (e.g., through object-centric losses) as well as data quality and richness are more effective than scaling up web data alone. finally, we 3) shed light on vlms' pretraining dynamics and suggest that future work should revisit pretraining strategies in order to consistently improve across several tasks. ## benchmarks we describe the recent (english) benchmarks proposed to measure fine-grained v&l understanding in zero-shot setups. foot_1 see svo-probes (hendricks and nematzadeh, 2021) focuses on verb understanding: it tests whether a model can identify if an image matches a sentence, and includes negative images which differ on a specific part of speech (subject, verb, and object). the dataset consists of 421 verbs and over 48k image-sentence pairs. 2 the authors show that their baselines fail more in situations requiring verb understanding than other parts of speech. valse @xcite consists of six tasks that cover basic linguistic phenomena, such as plurality, actions and coreference. for each task, given a visual input, a model is asked to distinguish real captions from foils @xcite , where a foil is constructed from a caption by altering a word or phrase that realises a specific linguistic phenomenon (e.g., semantic number of nouns). the authors show that vlms can identify objects in images, but struggle to ground their interdependence with specific linguistic indicators. vsr @xcite tests for 65 types of visual spatial relationships (e.g., under, in front of) grouped into seven categories (e.g., adjacency, orientation). each sample consists of an imagesentence pair; a model needs to predict whether the sentence correctly describes the spatial relation between two objects in the image. we evaluate models in a zero-shot setup on the 'random' split. 3 winoground @xcite is an expertcurated benchmark aiming to test models' compositional reasoning. given two images and two captions, the goal is to match them correctly; wherein both captions contain the same set of words, but in a different order. the authors define three scores: text (whether a model can match the correct caption for a given image), image (vice versa), and group (whether a model can match each pair). several competitive vlms have been shown to often perform close to or below random chance. we also report zero-shot performance on coarsegrained retrieval in flickr30k @xcite and coco @xcite in our analysis. model loss data downstream cl text obj det unsupervised supervised vqav2 nlvr2 refcoco+ albef4m ✓ mlm -4m: coco+sbu+vg+cc3m -74.7 80.5 -albef14m ✓ mlm -14m: 4m + cc12m -76.0 83.1 -blip14m ✓ lm -capfilt/b(14m) -77.6 82.3 -blip129m ✓ lm -capfilt/b(14m + laion) -78.2 83.1 -blip129m-capfilt/l ✓ lm -capfilt/l(14m + laion) -78.3 82.2 -blip-vit/l129m ✓ lm -capfilt/l(14m + laion) ----pevl14m ✓ mlm mlm 14m refcoco{,+,g}+f30ke+gqa+vcr+vg --74.5 x-vlm4m ✓ mlm regress 4m coco + vg 78.1 84.2 71.0 x-vlm16m ✓ mlm regress 14m coco + vg + objects365 + openimages 78.4 84.4 76.9 ## evaluated models recent work has shown that two components are crucial ingredients of strong coarse-grained vlms (e.g., @xcite @xcite : 1) a contrastive objective that aligns vision and language modalities, and 2) a cross-attention mechanism that fuses the two modalities. as we are interested in high performance on both fine-and coarse-grained tasks, to select models for our study, we surveyed recent work that uses these building blocks, 4 but also incorporates new losses or data that can potentially improve fine-grained v&l understanding. we find that many recent models build on albef @xcite @xcite (which we also study as a coarse-grained baseline). other than strong performance on coarsegrained and downstream tasks, we also considered: 1) the possibility to study the role of new modelling innovations and data for fine-grained skills, and 2) the availability of open-source code and pretrained weights. this resulted in four models briefly described next (more details in app. a.1). albef @xcite , with strong downstream performance, matches all our criteria and serves as a coarse-grained baseline. albef is a dual-stream encoder @xcite that first encodes images and captions independently, and then fuses them with cross-modal attention. blip @xcite uses an autoregressive language model (lm), and employs a dataset bootstrapping technique (capfilt) to generate synthetic captions and to remove noisy pairs from large-scale 4 by studying models with well-established modules, we expect our findings to be more informative for future work. web data. blip outperforms albef on most coarse-grained downstream tasks; thus, we study blip as another coarse-grained baseline to test if its generative lm and data contributions also lead to better fine-grained understanding. pevl @xcite ) is a fine-grained model building on albef, but leverages more supervised datasets such as referring expressions, captions with visual coreferences, object detection and region descriptions data, etc. (see x-vlm @xcite is our second finegrained model that enhances albef by adding both new losses and additional supervised data. in contrast to pevl, x-vlm models visual position through an additional bounding box prediction head that regresses the object's bounding box (bbox) coordinates. the authors use both object detection labels and region descriptions to learn coarse-and fine-grained alignments (we provide an in-depth analysis of this model in section 5). we remark that pevl and x-vlm were the only open-source fine-grained vlms at the time of our evaluation, and both of them build on top of al-bef. in addition to these core models, we also evaluate a dual-encoder network (clip; @xcite as well as recent architectures that rely on frozen, autoregressive (l)lms: clipcap @xcite , flamingo @xcite and blip-2 @xcite . as these models perform generally worse than our best fine-grained model, x-vlm, and differ significantly from it, we do not discuss their performance further. for more details, we refer the reader to tables 6 to 11 in app. b.1. we compare two strong vlms (albef and blip) with two models with explicit object modelling (i.e., fine-grained; x-vlm and pevl). we evaluate on fine-grained tasks (see ## data & losses for fine-grained tasks recent fine-grained models build on coarse-grained ones by introducing additional training data (e.g., object detection data in x-vlm and pevl) and new losses (e.g., bounding box regression loss in x-vlm). we study how data and losses influence fine-grained understanding, focusing on x-vlm as it outperforms other models on fine-grained benchmarks. while @xcite perform ablations to show the importance of their new objective function, they do not study the impact of data and losses independently; moreover, they do not evaluate on find-grained benchmarks. we start with a description of x-vlm, emphasising details in its pretraining procedure, that we reveal to have significant impact on the final performance. ## dynamics of fine-grained tasks we now analyse the dynamics of fine-grained skills for our models to investigate (i) when and whether they are acquired, and (ii) how they relate to one another, especially when they aim at measuring similar capabilities. for example, does action understanding in valse correlate with verb understanding in svo-probes? are there skills that vastly differ from each other that they would require different modelling contributions (e.g., counting)? experimental setup. we evaluate checkpoints (every 10k steps) from pretraining our albef and x-vlm re-implementations with 4m and 14m data points. we focus on 14m results as we see similar patterns with 4m (see app. b.2). when evaluating correlation patterns, we report both pearson and spearman correlation coefficients. different skills, different patterns. figure 3 (top) shows how the average model performance evolves during pretraining for the four benchmarks. interestingly, the performance on these benchmarks converges at different rates: both albef and x-vlm models easily improve on svo-probes. moreover, we observe that modelling objects (à la x-vlm) leads not only to better fine-grained understanding after 200k steps (tables 3 and 4 ), but also to remarkably quicker learning rates. figure 3 (bottom) shows performance on indicative valse tasks, as well as on coarse-grained image retrieval on coco. while some skills, such as spatial relations understanding, are learned progressively during pretraining, others, such as counting, degrade after a first, short learning phase. finally, other skills, such as coreference resolution, oscillate significantly throughout pretraining, showing how models can not properly acquire them. this is in contrast to the coarse-grained coco retrieval task for which the performance steadily increases over time. we conclude that it is particularly important to examine the training dynamics of fine-grained tasks, and that a single checkpoint might be inadequate for a number of skills. results on all tasks are provided in app. b.2, including on winoground for an albef 4m that we pretrained on gcp using the original codebase. same skills, same patterns? we next investigate whether closely-related tasks in different benchmarks have high correlation throughout pretraining. while we find that valse action replacement and svo-verb have a +55/67% pearson/spearman correlation, there is a -13/11% correlation between valse actant swap and svo-subject. looking at valse spatial relations, we find high correlation (+75/65%) with average vsr performance, and especially with relations such as on top of, on, inside, by, and in; mostly belonging to the 'topological' category in vsr. on the other hand, we find almost no correlation with several 'directional' (e.g., across from) and 'orientation' (e.g., parallel to) relations, as well as with some 'topological' ones (e.g., touching); and even negative correlation (-40% or less) with alongside, below, toward, part of and near. finally, surprisingly, vsr dev and test splits are not positively correlated for all relations. while average performance is highly correlated (+77/78%), only a few relations have pearson/spearman coefficients larger than 30% (in, on, above, within, and consists of). on the other hand, near, ahead of and adjacent to are negatively correlated between dev and test sets, and most relations show very low correlations between the two sets. as a result, improvement in a given relation type on the dev set, will likely not transfer at test time. takeaways. when tested on fine-grained benchmarks, we observe that, compared to albef, x-vlm is more sample efficient as it achieves higher performance with fewer training steps. also, while some tasks steadily improve during pretraining, for others, the performance degrades or fluctuates. moreover, surprisingly, the performance of tasks measuring similar skills but from different benchmarks do not always positively correlate. ## discussion while recent pretrained vlms achieve impressive performance on various downstream benchmarks (such as visual question answering and image retrieval), recent benchmarks have highlighted that they still struggle with tasks that require finegrained understanding-where a model needs to correctly align various aspects of an image to their corresponding language entities. yet, it is still not known to which extent recent fine-grained vlms (e.g., @xcite @xcite fare on such benchmarks. we address this gap by evaluating strong and fine-grained models on four benchmarks @xcite @xcite , and encourage future work to report zero-shot finegrained performance on our selection of benchmarks, especially if models are not open-source. our work contributes to a growing thread of research devoted to understand what is learned by pretrained vlms, such as studying cross-attention patterns @xcite , cross-modal input ablations @xcite , probing linguistic and visual structure @xcite @xcite , robustness to words order @xcite , and incorrectly fusing image and language modalities @xcite . here, we show that object modelling through a prediction loss (as done in x-vlm) results in notable improvements across all benchmarks, outperforming models trained on much larger amounts of web data. our analysis highlights that teaching vlms concepts of objects (e.g., by masking irrelevant parts of the image) is crucial for effectively learning fine-grained skills. though our models rely on supervised data to learn better localisation, we hope our findings can encourage researchers to design better loss functions for image-text mapping from unsupervised, webscale data as well. finally, our results also highlight the challenges of evaluating fine-grained understanding: the recent benchmarks capture a variety of subtasks (from counting to relation understanding); to perform well on these subtasks, a model requires different skills. indeed, we observe that, during training, model performance does not always increase for all subtasks, and in particular, fluctuates a lot for counting, coreference resolution, and various spatial relations. an important future direction is designing models that perform well on a larger range of these subtasks, where improving on one subtask does not degrade performance on the rest. it is unclear why benchmarks do not always correlate; possible reasons include the data itself (images selected for analysis, annotator instructions), or that different competencies are required for different fine-grained tasks. we hope future work can explore this further, possibly by closely examining data in fine-grained benchmarks or expanding the models used in analysis beyond what we used here.