VALSE : A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena (2022)

Abstract: We propose VALSE (Vision And Language Structured Evaluation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on specific linguistic phenomena. VALSE offers a suite of six tests covering various linguistic constructs. Solving these requires models to ground linguistic phenomena in the visual modality, allowing more fine-grained evaluations than hitherto possible. We build VALSE using methods that support the construction of valid foils, and report results from evaluating five widely-used V&L models. Our experiments suggest that current models have considerable difficulty addressing most phenomena. Hence, we expect VALSE to serve as an important benchmark to measure future progress of pretrained V&L models from a linguistic perspective, complementing the canonical task-centred V&L evaluations.

## introduction general-purpose pretrained vision and language (v&l) models have gained notable performance on many v&l tasks @xcite @xcite @xcite . as a result, v&l research has changed its focus from task-specific architectures to fine-tuning large v&l models. current benchmarks give a good perspective on model performance on a wide range of v&l tasks @xcite @xcite , but the field is only starting to assess why models perform so well and whether models learn specific capabilities that span multiple v&l tasks. specifically, we lack detailed understanding of the extent to which such models are able to ground linguistic phenomena-from morphosyntax to semantics-in the visual modality @xcite . for example, recent evidence suggests that models are insensitive to linguistic distinctions of verb-argument structure (hendricks and @xcite and word order @xcite . our work addresses this gap with valse (vision and language structured evaluation), a benchmark for v&l model evaluation comprising six tasks, or 'pieces', where each piece has the same structure: given a visual input, a model is asked to distinguish real captions from foils, where a foil is constructed from a caption by altering a word or phrase that realizes a specific linguistic phenomenon, e.g., semantic number of nouns, verb argument structure, or coreference. valse uses a resource-lean diagnostic setup that dispenses with large-scale annotation (e.g., of bounding boxes), and builds on existing high-quality image captioning and vqa data. valse is designed to leverage the existing prediction heads in pretrained (or finetuned) v&l models; for that reason, our benchmark does not include any re-training and can be interpreted as a zero-shot evaluation. we build test data for each piece so as to safeguard against the possibility of models exploiting artefacts or statistical biases in the data, a well-known issue with highly parameterised neural models pretrained on large amounts of data @xcite @xcite . with this in view, we propose novel methods to guard against the emergence of artefacts during foiling. our main contributions are: i) we introduce valse, a novel benchmark aimed at gauging the sensitivity of pre-trained v&l models to foiled instances. ii) we cover a wide spectrum of basic linguistic phenomena affecting the linguistic and visual modalities: existence, plurality, counting, spatial relations, actions, and entity coreference. iii) we investigate novel strategies to build valid foils that include automatic and human valida-tion. we balance word frequency distributions between captions and foils, and test against pretrained models solving the benchmark unimodally by relying only on text. we employ masked language modeling (mlm) in foil creation and semantic inference for validating foils, and finally collect human annotations for the entire benchmark. iv) we establish initial experimental results for pretrained v&l models of diverse architectures on valse. the overall weak performance of these models indicates that the time is ripe for a novel, reliable foiling dataset targeting the visual grounding capabilities of v&l models through the lens of linguistic constructs. 1 2 background and related work pretrained v&l models learn to combine vision and language through self-supervised multitask learning. tasks include multimodal masked modeling-where words in the text and object labels or regions in the image are masked out, then predictedand image-sentence alignment, whereby a model learns to predict whether an image and a text correspond. major architectures are single-and dualstream multimodal transformers: single-stream models concatenate word and image features, and encode the resulting sequence with a single transformer stack; dual-stream models use distinct transformer stacks to handle visual and textual inputs, and additional layers (e.g. co-attention) to fuse these into multimodal features. benchmarking v&l models v&l models @xcite @xcite @xcite are commonly evaluated on v&l tasks such as vqa @xcite , visual reasoning @xcite , or image retrieval @xcite . given how well transformer-based models perform across unimodal and multimodal tasks, research efforts have recently started to address what makes them so effective, and to what extent they learn generalisable representations. techniques to address these questions in unimodal and multimodal v&l contexts include: adversarial examples @xcite ; investigation of the impact of bias, be it linguistic @xcite , visual semantic @xcite , or socio-economic @xcite ; and the use of linguistically-informed counterfactual and minimally-edited examples @xcite . a trend within the latter research line that is specific to v&l models is vision-and-language foiling @xcite @xcite @xcite , where the idea is to create counterfactual (i.e., foiled) and/or minimally edited examples by performing data augmentation on captions @xcite or images @xcite . since most v&l models are pretrained on some version of the image-text alignment task, it is possible to test their ability to distinguish correct from foiled captions (in relation to an image) in a zeroshot setting. the construction of foils can serve many investigation purposes. with valse, we target the linguistic grounding capabilities of v&l models, focusing on pervasive linguistic phenomena that span multiple tokens, described in §3.1- §3.6. at the same time, we ensure that our data is robust to perturbations and artefacts by i) controlling for word frequency biases between captions and foils, and ii) testing against unimodal collapse, a known issue of v&l models @xcite , thereby preventing models from solving the task using a single input modality. the issue of neural models exploiting data artefacts is well-known @xcite @xcite and methods have been proposed to uncover such effects, including gradient-based, adversarial perturbations or input reduction techniques (cf. @xcite ). yet, these methods are still not fully understood @xcite and can be unreliable @xcite . our work is related to @xcite , who construct task-specific contrast sets for nlu. however, our focus is on modelling linguistic phenomena instead of tasks, and we construct carefully curated, balanced, single foils from valid instances that we select from multiple multimodal datasets. ## constructing the valse benchmark we resort to a musical analogy to describe valse: vision and language structured evaluation is composed of 6 pieces, each corresponding to a specific linguistic phenomenon (see data collection & metadata pieces existence plurality counting relations actions coreference instruments existential quantifiers semantic number balanced, adversarial, small numbers prepositions replacement, actant swap standard, clean #examples † 505 851 2, 459 535 1, 633 812 foil generation method nothing ↔ something np replacement (sg2pl; pl2sg) & quantifier insertion numeral replacement spanbert prediction action replacement, actant swap yes ↔ no mlm gruen nli src. dataset visual7w mscoco visual7w mscoco swig visdial v1.0 image src. mscoco mscoco mscoco mscoco situnet mscoco example data caption (blue) / foil (orange) there are no animals / animals shown. a small copper vase with some flowers / exactly one flower in it. there are four / six zebras. a cat plays with a pocket knife on / underneath a table. a man / woman shouts at a woman / man. buffalos walk along grass. are they in a zoo? no / yes. image all instruments are built by applying foiling functions (ffs) specific to the linguistic phenomenon under study. ffs take a correct caption as input and change a specific part to produce a foiled caption (or foil). we design ffs such that the sentences they produce fail to describe the image, while still being grammatical and otherwise valid sentences. of course, a foiled caption may be less likely than the original caption from which it was produced, and such unwarranted biases can be easily picked up by overparameterised v&l models. moreover, an automatic ff may fail to produce a foil that contradicts the image, for example by altering the original caption to yield a near-synonymous one, or one that is entailed by the original caption. for phenomena that make it difficult to control these crucial properties of foils, we apply additional filters: i) some ffs make use of strong lms to propose changes to captions, so that the generated foils are still high-probability sentences; ii) we use state-of-the-art natural language inference (nli) methods to detect cases where there is an entailment between caption and foil, and filter out such foils from the dataset (see §4 for discussion). as a final measure, we employ human annotators to validate all generated testing data in valse. valse data is sourced from existing v&l datasets. below, we describe each piece and its instruments, and the corresponding task setup in valse. for each instrument, we follow the same procedure: i) we identify captions that contain instances of the targeted linguistic phenomenon; ii) we apply a ff that automatically replaces the expression with a variant that contradicts the original expression's visual content, thereby constructing one or more foils from each target instance in the original caption, as discussed in §4; we then iii) subject the obtained foils to various filters, with the aim of distilling a subset of valid and reliable foils that cannot be easily tricked by a new generation of highly parameterised pretrained v&l models. ## reliable construction of valid foils in valse, an instance consisting of an imagecaption-foil triple is considered valid if: the foil minimally differs from the original caption; the foil does not accurately describe the image; and independent judges agree that the caption, but not the foil, is an accurate description of the image. we consider a foiling method to be more reliable the more it ensures that a generated foil does not substantially differ from a human caption regarding distributional and plausibility bias, and cannot be easily solved unimodally. 3 visdial annotations are not available for the test set. in this section, we discuss automatic and manual means to reliably construct valid foils. in this context, two types of bias are especially worthy of note: distributional bias ( §4.1) and plausibility bias ( §4.2). in §4.3 we discuss how we apply a natural language inference model to filter examples in our data pipeline, and §4.4 show how we manually validate all examples in our benchmark. random samples from the final version of each instrument are shown in tab. 6-11. ## benchmarking with valse we propose valse as a task-independent, zeroshot benchmark to assess the extent to which models learn to ground specific linguistic phenomena as a consequence of their pretraining (or fine-tuning). valse is built in the spirit of approaches such as checklist @xcite , including pairs consisting of captions and minimally edited foils. the only requirement to evaluate a model on our benchmark is: i) to have a binary classification head to predict whether an image-sentence pair is foiled, or ii) to predict an image-sentence matching score between the image and the caption vs. the foil, returning the pair with the highest score. systems reporting results on valse are expected to report any data used in model training prior to testing on valse, for comparability. ## conclusions and future work we present the valse benchmark to help the community improve v&l models by hard-testing their visual grounding capabilities through the lens of lin-guistic constructs. our experiments show that v&l models identify named objects and their presence in images well (as shown by the existence piece), but struggle to ground their interdependence and relationships in visual scenes when forced to respect linguistic indicators. we encourage the community to use valse for measuring progress towards v&l models capable of true language grounding. furthermore, valse could be used as an indirect assessment of datasets, as models could be evaluated before and after training or fine-tuning to see if a dataset helps models improve on any of the aspects tested by valse. valse is designed as a living benchmark. as future work we plan to extend it to further linguistic phenomena, and to source data from diverse v&l datasets to cover more linguistic variability and image distributions.