The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models (2023)

Abstract: Despite the impressive performance achieved by pre-trained language-and-vision models in downstream tasks, it remains an open question whether this reflects a proper understanding of image-text interaction. In this work, we explore to what extent they handle basic linguistic constructions—active-passive voice, coordination, and relative clauses—that even preschool children can typically master. We present BLA, a novel, automatically constructed benchmark to evaluate multimodal models on these Basic Language Abilities. We show that different types of Transformer-based systems, such as CLIP, ViLBERT, and BLIP2, generally struggle with BLA in a zero-shot setting, in line with previous findings. Our experiments, in particular, show that most of the tested models only marginally benefit when fine-tuned or prompted with construction-specific samples. Yet, the generative BLIP2 shows promising trends, especially in an in-context learning setting. This opens the door to using BLA not only as an evaluation benchmark but also to improve models’ basic language abilities.

## introduction powered by the transformer architecture, extensive pre-training, and task-specific fine-tuning, recent language and vision models @xcite @xcite @xcite @xcite have achieved unprecedented performance in many downstream multimodal tasks. despite the impressive results, it remains an open question whether, and to what extent, this improvement goes hand in hand with a genuine understanding of image, text, and their interaction. in particular, the ability of models to handle linguistic skills that are essential to understanding an event or situation has recently been questioned. hendricks and nematzadeh (2021), for example, showed that these models fail in scenarios that require understanding verbs and verb arguments; @xcite active-passive voice t: the woman feeds the man. t: the man is fed by the woman. f: the man feeds the woman. f: the woman is fed by the man. coordination t: the man wears a wetsuit and carries a surfboard. t: the woman wears a red bikini and rides a red bike. f: the man wears a wetsuit and rides a red bike. f: the woman carries a surfboard and wears a red bikini. relative clause t: the man who wears a gray polo holds a stuffed bear. t: the man who wears a striped shirt holds a cow. f: the man who wears a gray polo holds a cow. f: the man who wears a striped shirt holds a stuffed bear. figure 1 : one example for each of the linguistic constructions included in the benchmark. sentences in each bla dataset (both true and false ones) share the same template, with 3 slots being filled by as many arguments (nps, predicates, or clauses), that we signal with different colors. e.g., in the active-passive construction, the bits in red and green are the subject and object of the sentence, respectively, while the blue bit is the predicate. arguably, all these previous studies require models to do more than plain language comprehension, including mathematical, spatial @xcite , pragmatic @xcite , and compositional reasoning abilities @xcite . while mastering these abilities is clearly desirable for any intelligent system, we notice that models may struggle with them due to their pre-training data and objectives. indeed, these models are typically trained to verify whether a fragment of text is about the content of an image-via the image-text matching (itm) objective-which closely resembles the language comprehension tests administered to children to assess their lexical and grammatical abilities. to illustrate, in these tests, children are presented with a sentence, e.g., the red monkey is being scratched by the blue monkey, and asked to either pick the corresponding image from a set of alternatives (for an overview of this work, see @xcite or color the entities in a drawing accordingly @xcite . consistent with their goal, these tests are aimed at excluding, or at least minimizing, the need for reasoning, which has been shown to be separate from language and to recruit different brain areas @xcite . in this work, we take inspiration from this line of research and investigate whether, and to what extent, language-and-vision models deal with language comprehension tasks that require virtually no reasoning abilities. we focus on three basic language constructions-active-passive voice, coordination, and relative clauses-that even preschool children typically master @xcite @xcite . we refer to these constructions as basic language abilities (bla) and propose an automatically constructed benchmark (see figure 1 ) to assess pre-trained language-and-vision models, either in a zero-shot or in fine-tuning and in-context learning scenarios. we test several types of transformerbased systems, i.e., clip, lxmert, vilbert, blip2 and openflamingo, and show that, while human (adult) speakers have no trouble at verifying these linguistic constructions, models generally struggle. yet, the generative blip2 model shows promising trends, especially in an in-context learning setting. this reveals that, while bla is a challenging benchmark, it can be used not only as an evaluation tool but also to improve sota models' basic language abilities-which are currently generally poor. we release the bla benchmark and the code to reproduce our results at: https://github.com/shin-ee-chen/bla . ## the bla benchmark in this section, we describe our basic language abilities (bla) benchmark. ## vision & language transformers we experiment with 5 pre-trained language-andvision models: three discriminative models (clip, vilbert, and lxmert), and two generative models (blip2 and openflamingo). 7 we briefly describe the models below. ## exp 1: zero-shot evaluation to explore whether, and to what extent, the pretrained models described in section 4 can deal with linguistic constructions in bla without any taskspecific fine-tuning, we evaluate them on the three datasets in a zero-shot setting. for each dataset, we frame the problem as a binary task: given @xmath0, @xmath1, the models are asked to evaluate whether the sentence is true or false. vilbert and lxmert can be straightforwardly evaluated on the binary task thanks to their pre-trained image-text matching (itm) classification head. for clip, we compute similarity scores between the image and each sentence, and rank the @xmath2, @xmath3. we consider the 2 top-ranked sentences as true and the 2 lower-ranked sentences as false. we evaluate blip2 and openflamingo by prompting. the prompt template used with blip2 is similar to the one proposed by @xcite for visual question answering: 'question: is the sentence [sentence] appropriate for this image? yes or no? answer:'. for openflamingo, following awadalla et al. (2023), we use the following prompt template: '<@xmath4: is the sentence [sentence] appropriate for this image? yes or no? short answer:'. we let the models generate a response and consider 'yes' answers as true and 'no' answers as false. 11 we use three metrics to measure model performance: (1) accuracy, measuring how well the models perform on the binary task, (2) precision_true, measuring how well models identify the true sentences, and (3) precision_false, measuring how well the models identify the false sentences. ## exp 2: bla-specific learning to explore whether the models-which obtain poor performance in the zero-shot setting-can learn to deal with the linguistic constructions in bla via some degree of task-specific learning, we expose them to a relatively little amount of data from each dataset. we use these samples to fine-tune the discriminative models and prompt the generative models to allow them to perform in-context learning. in particular, we experiment with two types of bla-specific learning, i.e., (1) we train and test a model with data from the same dataset (sd), and (2) we train a model with data from one dataset and test it with data from a different dataset (dd) in a cross-task setting. with the former, we evaluate whether the models can learn some key properties of the linguistic construction at hand by having exposure to the same phenomenon in other visual contexts. with the latter, we test whether the models can develop general language comprehension abilities that are key to all linguistic constructions. we downsize each dataset to include 400 samples. then, we randomly split it into training (100 samples), validation (100), and test (200) sets. while each sample in the validation and test sets has the standard format-one image and four corresponding sentences: two true sentences and two false ones-in the training set that we use to finetune or prompt our models we only keep two sentences per image. in particular, we randomly sample one true and one false sentence to keep the supervision of the models relatively little. image? yes or no? answer:'. in dd, the setup is the same as above, except that the [true] and [false] sentences are from another bla dataset (different linguistic construction), and yet about the same image (same entities and attributes). while images in co and rc greatly overlap, images in ap do not overlap much with those in the other two datasets. therefore, we only experiment with co and rc in this setting. ## conclusions we introduced a novel benchmark, bla, aimed at investigating how well multimodal models understand basic linguistic constructions-active-passive voice, coordination, and relative clauses. we showed that the linguistic constructions in bla are challenging for current language-and-vision models, which lag well behind human performance. yet, the recent generative model blip2 exhibits a better performance than discriminative models, both in the zero-shot and task-specific learning setting. we highlight that prompting generative models with examples embedding both the same or a different linguistic construction is a promising method to improve their understanding of specific linguistic constructions. this opens the door to using bla not only to evaluate pre-trained models but also to improve their basic language abilities.