T ra VLR : Now You See It, Now You Don’t! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning (2023)

Abstract: Numerous visio-linguistic (V+L) representation learning methods have been developed, yet existing datasets do not adequately evaluate the extent to which they represent visual and linguistic concepts in a unified space. We propose several novel evaluation settings for V+L models, including cross-modal transfer. Furthermore, existing V+L benchmarks often report global accuracy scores on the entire dataset, making it difficult to pinpoint the specific reasoning tasks that models fail and succeed at. We present TraVLR, a synthetic dataset comprising four V+L reasoning tasks. TraVLR’s synthetic nature allows us to constrain its training and testing distributions along task-relevant dimensions, enabling the evaluation of out-of-distribution generalisation. Each example in TraVLR redundantly encodes the scene in two modalities, allowing either to be dropped or added during training or testing without losing relevant information. We compare the performance of four state-of-the-art V+L models, finding that while they perform well on test examples from the same modality, they all fail at cross-modal transfer and have limited success accommodating the addition or deletion of one modality. We release TraVLR as an open challenge for the research community.

## introduction research in psycholinguistics has found that human processing of spatial words activates brain regions associated with the visual system @xcite , suggesting the latter's involvement in processing linguistic input. it is reasonable to expect multimodal neural models to resemble humans in being able to leverage capabilities in the visual domain to solving problems in the text domain, and vice versa. following its recent success in the text domain @xcite , the pretrainingfine-tuning paradigm has been applied to the vision and text modalities to create unified visio-linguistic (v+l) representations. just as pretrained multilingual models have been shown capable of zero-shot cross-lingual transfer on various nlp tasks @xcite , we may expect true v+l models to be capable of generalising to a modality not seen during fine-tuning. however, current approaches for benchmarking v+l models involve reporting global accuracy scores on the entire dataset, rendering the specific sources of success and failure difficult to diagnose @xcite . for instance, visual question answering (vqa, @xcite tasks may allow models to exploit dataset bias @xcite , or may reduce to object recognition problems which do not evaluate the models' ability to perform more complex tasks, beyond aligning words or phrases in the text to a portion of the image @xcite . as @xcite note, the ability to reason over multiple objects and answer relational questions is crucial to the genuine mastery of language. datasets such as nlvr2 @xcite address this limitation, but do not allow for finegrained evaluation along specific dimensions @xcite . clevr @xcite and shapeworld (kuhnle and copestake, 2017) enable targeted evaluations of a v+l model's reasoning abilities but only encode the scene unimodally, as images. additionally, their test examples may still be in the training distribution with respect to task-relevant dimensions, making it difficult to draw conclusions about generalisation ability. we thus contribute travlr, a synthetic dataset comprising four v+l reasoning tasks: spatiality, cardinality, quantifiers, and numerical comparison, all of which require reasoning over multiple objects and have been shown to be challenging for v+l models @xcite . unlike shapeworld, we fully leverage the benefits of using a synthetic dataset by controlling the train-test split such that examples in the out-of-distribution (ood) test set are ood with respect to task-relevant dimensions. we thus argue that travlr serves as a basic sanity check for the abstract reasoning and out-of-distribution generalisation capabilities of models, and is complementary to datasets that evaluate real-world object recognition and compositional reasoning abilities, such as gqa @xcite . inspired by the word/picture sentence verification task from psycholinguistics @xcite , we further propose various novel evaluation settings by representing the scene bimodally as both an image and a caption. first, travlr supports the novel cross-modal transfer setting (figure 1 ): if pretrained v+l models have learnt a truly multimodal representation, they should be able to learn a reasoning task with input from one modality and perform inference using input from the other modality with little to no extra training. being able to transfer cross-modally in a zero-or few-shot manner may improve data efficiency in applications where diverse image data is difficult to obtain. furthermore, models should also succeed on test settings where either an unseen modality is added, or a seen modality is dropped. using travlr, we perform extensive analysis of the ability of four transformer-based v+l models to perform various reasoning tasks. we show that current v+l models: other synthetic datasets focusing on reasoning include clevr @xcite , a fully synthetic and annotated 3d dataset, and shape-world, a 2d dataset targeting linguistic phenomena such as spatial relationships and quantifiers. spartqa @xcite is another 2d synthetic dataset built upon nlvr focusing on spatial reasoning among other linguistic phenomena. v+l models. pretrained v+l models differ in their architecture and pretraining methods. vl-bert @xcite , uniter @xcite and visualbert @xcite are singlestream models with a single transformer while vilbert @xcite , lxmert @xcite , and albef @xcite are dual-stream models which encode image and textual inputs separately before fusing them. these models all use masked language modelling and image-text matching objectives for pretraining, with lxmert additionally pretraining on vqa and albef using a contrastive loss to align the image and language representations. uniter, visu-albert, and lxmert use a frozen faster r-cnn @xcite to extract region-based features from the image, while albef directly encodes the image with a vision transformer @xcite . cross-modal transfer. prior work has found models trained on multimodal data to perform better on unimodal downstream tasks than models trained only on one modality. @xcite found models trained on multimodal input to perform better than text-only models on three nlp tasks, while @xcite showed that models trained on textual, visual, and auditory input were better at a quantification task than models trained only on a single modality. using a task involving queries about typical colours of objects, @xcite found bert trained on linguistic and visual features to outperform bert trained on language data filtered for mentions of colour. @xcite investigated the crossmodal alignment of pretrained v+l models with an ablative method based on masked modelling. @xcite propose an image-text fusion model to solve a novel image generation task from a textual description and an image prompt based on nlvr2. summary. the datasets commonly used to evaluate v+l models such as vqa and nlvr2 lack fine-grained interpretability, due to the lack of annotations for semantic phenomena involved in each example. additionally, multiple semantic phenomena co-occur within a single example, making it difficult to control the training distribution and assess the generalisation abilities of models. furthermore, existing v+l datasets only present the scene in the visual modality and cannot be used to evaluate a v+l model's cross-modal transfer ability. existing synthetic datasets (e.g. shapeworld, clevr) fail to split the training and testing distributions along a dimension relevant to the specific task, because they generate captions based on randomly generated images. unlike existing datasets, travlr fully exploits the benefits of a synthetic dataset by strictly controlling the training and evaluation distributions to test the generalisation abilities of v+l models and avoid statistical biases from language priors and non-uniform distributions. ## travlr: cross-modal transfer of visio-linguistic reasoning we construct travlr, a synthetic dataset comprising four visio-linguistic reasoning tasks: spatiality, cardinality, quantifiers and numerical comparison. these tasks were previously identified to be challenging for text-only models @xcite @xcite . travlr aims to evaluate the extent to which pretrained v+l models already encode or are able to learn these four relations between entities present in input scenes. we first describe the general task format, and then describe each task. given a scene with objects, @xmath0, ..., o n }, where each object can be represented as a @xmath1, shape, position >, and a textual query q involving some relation r(o 1 , ..., o i ) between two or more objects in s, each task involves learning a function @xmath2, q) where @xmath3, false}. this is essentially a binary classification task. for instance, in the spatiality task, the relation r (e.g., above) compares the positions of two objects. in the numerical comparison task, noun phrases in the query refer to subsets of objects, while the relations (e.g., more) compare the cardinality of two sets of objects. assigning a truth value to the query thus involves reasoning over several objects. however, a model can never have direct access to the underlying representation scene and must operate on visual or textual forms, and s may be represented in the form of an image or a textual description. in prior work such as vqa, s is presented as an image. in travlr, s is represented bimodally as @xmath4, @xmath5. each example consists of an image, an accompanying caption, and a query. images include abstract objects arranged in a 6 × 6 grid. we draw from five colours and seven shapes, giving 35 unique objects in total. the shapes used in travlr are from the visual genome @xcite , a commonly-used pretraining dataset. each caption fully describes the image with the coordinates of each object (e.g., "there is a red circle at a 1, a blue square at b 2..."). a description of the coordinate system, e.g., "columns, left to right, are ordered a to f. rows, top to bottom, are ordered 1 to 6." is prepended to the caption. the caption and query are separated by the [sep] token when presented to the models. removing the caption reduces our tasks to vqa-like tasks. ## experiments models. we perform experiments on two singlestream models, visualbert and uniter, and two dual-stream models, lxmert and albef. we use @xcite 's implementation of visu-albert, lxmert, and uniter, and the original implementation of albef. the image features of lxmert and uniter are 36 regions of interest extracted using @xcite 's implementation of a pretrained faster r-cnn @xcite . we use a detectron model @xcite to extract image features for visualbert. we also train two text-only models, bert @xcite and roberta @xcite , as caption-only baselines. setup. we train models on each task for 80 epochs. following @xcite 's finding that early stopping may lead to underestimation of model performance, we do not do early stopping. we use the recommended hyperparameters for fine-tuning albef on snli-ve @xcite and visualbert on nlvr2. learning rates were adjusted downwards for models/tasks where recommended hyperparameters did not lead to convergence. each experiment is repeated with three random seeds. ## discussion comparing modalities. as discussed in §4.3, v+l models exhibit a clear bias towards the textual modality across single and dual stream models, corroborating findings by @xcite . this finding also applies to lxmert, even though it generally performs more poorly on caption than image inputs. furthermore, the v+l models perform more poorly than unimodal roberta on various tasks in the caption-only setting, similar to @xcite , who show that v+l pretraining on degrades performance on nlu tasks. comparing tasks. the spatiality task is the hardest task, requiring at least 32k examples to converge in some cases, as opposed to the 8k examples for the other tasks (see appendix c for details). furthermore, convergence of the models on the spatiality task is highly sensitive to random seeds. among the other tasks, the easiest task is the quantifiers task, followed by cardinality, and finally numerical comparison. in the caption-only and image+caption settings, all models apart from lxmert achieve a close to perfect f 1 on the cardinality and quantifiers tasks. however, on the numerical comparison task, uniter and lxmert exhibited a limited ability to generalise outside the training distribution in both image and textual modalities. thus, while success on the cardinality task indicates that models possess an understanding of the meaning of numbers in absolute terms, the numerical comparison task was able to differentiate models in terms of their understanding of numbers' relative positions on a numeral scale. comparing models. as argued in §5.1, the most significant factor differentiating models seems to be their pretraining rather than model architecture. our findings corroborate @xcite 's findings that differences between models cannot be primarily attributed to differences in model architecture. unlike lxmert and albef, visual-bert and uniter both succeed on all tasks in all settings. while uniter generally outperforms vi-sualbert in the image-only setting, visualbert at times outperforms uniter in the caption and image+caption settings. the superior performance of visualbert in caption-only settings could be due to its comparatively minimal pretraining on v+l objectives, enabling it to retain performance closer to that of text-only models. the encoding of image features also has a sig-nificant impact on performance in the image-only setting. visualbert's poor performance on the spatiality task is likely due to the fact that it does not explicitly encode the spatial coordinates of the bounding boxes in its input. additionally, it is likely that the pretrained object detector used by every model except albef helped them outperform albef in the image-only setting, despite underperforming it in common v+l benchmarks @xcite . a possible explanation is that the information captured by the faster r-cnn detector was more relevant to our visio-linguistic tasks. finally, the poor performance of lxmert on caption-only and image+caption settings reflects its lack of initialisation with bert parameters before pretraining. given the superior performance of roberta over bert, it could be beneficial to initialise models with roberta weights instead. ## conclusion travlr allows us to evaluate specific visiolinguistic reasoning skills in isolation, enabling finer-grained diagnosis of model deficiencies. we found some models to learn better from one modality than the other, and some task-setting combinations to be more challenging across the board. furthermore, existing models may require unreasonably large amounts of data and training steps to learn simple tasks. improving the sample efficiency and training time of v+l models is a potential direction for future research. furthermore, while pretrained multilingual models have been shown to demonstrate zero-shot cross-lingual transfer abilities, it is unclear whether v+l models can similarly perform cross-modal transfer of downstream task abilities to a modality unseen during fine-tuning. we hence contribute travlr, which enables the evaluation of cross-modal transfer ability by encoding scenes bimodally. we found all models to suffer from a strong textual bias, and an inability to perform zero-and few-shot cross-modal transfer. given the success of multilingual models with cross-lingual transfer, future work might attempt to bridge the gulf between the visual and textual modalities, perhaps by forcing models to transform images into intermediate representations which more closely resemble language. visio-linguistic representations capable of such transfer will unlock a whole host of new applications, and we pose this as the next challenge for multimodal modelling.