Are Language-and-Vision Transformers Sensitive to Discourse? A Case Study of V i LBERT (2023)

Abstract: Language-and-vision models have shown good performance in tasks such as image-caption matching and caption generation. However, it is challenging for such models to generate pragmatically correct captions, which adequately reflect what is happening in one image or several images. It is crucial to evaluate this behaviour to understand underlying reasons behind it. Here we explore to what extent contextual language-and-vision models are sensitive to different discourse, both textual and visual. In particular, we employ one of the multi-modal transformers (ViLBERT) and test if it can match descriptions and images, differentiating them from distractors of different degree of similarity that are sampled from different visual and textual contexts. We place our evaluation in the multi-sentence and multi-image setup, where images and sentences are expected to form a single narrative structure. We show that the model can distinguish different situations but it is not sensitive to differences within one narrative structure. We also show that performance depends on the task itself, for example, what modality remains unchanged in non-matching pairs or how similar non-matching pairs are to original pairs.

## introduction large language models are considered "black boxes" as it is often hard to explain their predictions. therefore, it is essential to evaluate such models on tasks different from downstream applications to see if they have acquired necessary knowledge, including linguistic knowledge, during the pre-training process @xcite @xcite . moreover, model performance on linguistic tasks has been shown to correlate with downstream applications @xcite . although the models have been tested on morphology, semantics, and syntax @xcite @xcite @xcite , little has been done on the evaluation of discourse @xcite . models such as gpt-3 are good at generating long, coherent sequences but they are often not sensitive to intents or communicative purposes that the narratives are written with @xcite . in this work, we focus on the evaluation of the sensitivity of multi-modal models to discourse. we understand discourse as contextual information defined by the situation of communication connected to social and cultural background. discourse can be local defined by individual items such as an utterance or an image or narrative defined by a sequence of several items. discourse meaning includes but is not limited to: as we are interested in how a language-andvision model in general captures all three components of discourse outlined above and to investigate the effects and the role of discourse on the generation of text across different discourses, for experiments we evaluate a single multi-modal transformer, vilbert @xcite , that was trained on a (simple) discourse task of image-text matching. this is also one of the first transformer-based models with a cross-modal alignment pre-training objective. for the data we use the visual storytelling dataset (vist) @xcite which consists of descriptions of images at two different levels of annotation and therefore discourses: descriptions-in-isolation and stories-in-sequence. while descriptions-in-isolation were collected following standard instructions for image captioning in ms-coco @xcite (for example, annotators have to name all important parts, i.e. local discourse), for stories-in-sequence crowd workers were asked to write a story about a sequence of images (narrative discourse, see figure 1 ). storiesin-sequence do not tend to name objects in images but rather connect images to a coherent situation, and therefore we expect them to be more challenging for a language-and-vision model. we assume that during pre-training a model acquires the knowledge of situation described by a specific image-text pair. since we understand discourse as a situation itself rather than a sequence of images, we adapt a standard image-caption matching task to our experiments by creating different perturbations, i. e. non-matching pairs that we call distractors, by permuting data in several ways, across stories and within one story. we investigate the model's abilities of grounding discourse at different levels: in the first two cases we are testing the model's ability to identify distractors across different stories and situations, hence testing the model's ability to identify local discourse. in the third case, we are testing the model's ability to identify distractors within the same narrative and hence we are testing the model's ability to identify narrative discourse. we introduce a new method of constructing evaluation tasks for language-and-vision models. we then analyse the role of different discourse structures and different modalities on the performance on our tasks. our results reveal details about the behaviour of large multi-modal transformer for a setup that is beyond simple image-text matching. ## related work little has been done on evaluating models' knowledge of discourse. for language models ettinger (2020) shows that bert produces pragmatically incorrect outputs as it does not take into account broader context. most of the previous work on discourse evaluation of bert and bert-like models was focused on local discourse structures. for example, @xcite evaluate models on explicit discourse relations expressed with conjunctions. @xcite propose a benchmark for model evaluation on different discourse tasks such as prediction of implicit discourse relations based on the penn discourse treebank annotation @xcite , discourse coherence, and others. @xcite attempt to improve the results on discourse tasks from discoeval by changing the pretraining objective of models. recently, @xcite introduced a dataset for image-based story generation and proposed a character-based story generation model that is evaluated based on the coherence of generated texts. multimodal models were shown to ground semantic and syntactic knowledge. @xcite provide results that the models ground both semantic and syntactic relations. @xcite argue that models can ground objects but struggle with interdependence relations. most of the work on discourse evaluation of languageand-vision models focuses on downstream tasks such as visual question answering (vqa) and visual coreference. @xcite provide an overview of vqa systems and their challenges related to reasoning, language ambiguity etc. several studies address the problem of discourse-coherent image generation. @xcite introduce a generation mechanism that produces captions grounded not only in the visual context but also in the established common ground. @xcite improve the quality of generated captions by feeding models with additional information on the types of connections between two clauses. @xcite show how different decoding strategies for image captioning reflect discourse structure in comparison to reference captions. as for existing image-caption datasets, @xcite argue that existing datasets do not reflect all possible coherence types and are limited in the word usage. however, even existing datasets are challenging for models, as @xcite show models such as clip or vil-bert do not capture differences in coherence relations. however, recently discourse has started to receive increased attention in the research community, including development of new approaches to building discourse datasets @xcite . ## models we examine vilbert (vision-and-language bert) @xcite , a dual-stream multi-modal bert-based model. unlike single-stream models that encode both modalities at the same time, dualstream models initially represent each modality separately. these models then learn cross-modal grounding which should include some knowledge of discourse structure, depending on the context in which data was collected. vilbert performs well on the image-text matching task and it is frequently used in the studies of multi-modality. one important feature of vilbert is that its text module's parameters are initialised with bert. in comparison, lxmert @xcite , a very similar model to vilbert, learns its text module from scratch. although there are more recent transformers trained for different discourse contexts we focus on a transformer that was trained on a simple referring context (the properties of which are well-known to us) in order to investigate how well the knowledge of that discourse transfers to other (more complex) discourses. we expect that the performance of a model will vary depending on how well its pre-training objective(s) and datasets that it was trained on match the visual storytelling task. vilbert was trained on three objectives: masked language modelling, image region masking, and an image-text matching task. in the latter task, the model has to predict if a given sequence of tokens consisting of a visual input (an image) and a language input (a caption) match or do not match. this objective helps the model to ground descriptions in images and differentiate them from non-matching counter-parts. this task therefore also involves learning about contextual sensitivity of descriptions to discourse and therefore, provided that non-matching items are controlled, we see it as a suitable task for our investigation. we test the vilbert model from volta @xcite , a framework that provides the code base for several transformer-based language-and-vision encoders and allows working with custom datasets. the model takes a set of pre-processed features: masked sentences, token ids, visual features, image location, masked images with regions of interest and their object labels. textual features are generated by the bert tokeniser * that returns a sequence of token ids. to extract image features, we use the caffe vg faster r-cnn implementation (anderson et al., 2018) † . the model extracts 36 proposal boxes with features of dimension 2048 and object labels. we do not mask any tokens or regions as we only focus on one output head of vilbert that predicts if an image and a text match. ## data for experiments we use images and descriptions from the visual storytelling dataset (vist) @xcite . the dataset includes stories from 10,117 flickr albums containing 210,819 images split into train (80%), dev (10%), and test (10%) samples. stories reflect narrative structure and sentences are linked with discourse relations. an example of a story taken from the dataset is illustrated in figure 1 . although there are other datasets that include narrative captions, such as recipeqa @xcite , we choose vist as it includes several types of captions, which allows us to look at different discourse structures. the dataset includes three levels of description annotation, descriptions-inisolation, descriptions-in-sequence ‡ , and storiesin-sequence, by crowd-workers hired through amazon's mechanical turk. for stories-in-sequence a worker selected at least 5 images and wrote a story about them. then, another worker received the same images and wrote their story. for descriptions-in-isolation workers followed the instructions of image captioning tasks from ms coco @xcite . for each album, 5 stories were collected, and for each image 3 workers wrote descriptions-in-isolation. not to repeat images, we limit our data to only one annotation. we expect that the first type of captions, descriptions-in-isolation, is more descriptive, in other words, annotators tend to mention objects on a picture rather than describe events. stories-insequence, on the other hand, might not be directly related to pictures, and annotators can omit some information that could be extracted from visual context. hence, two types of annotation reflect different types of image-text coherence. ## experimental setup we run three experiments based on type of descriptions in vist and whether descriptions were considered as a part of local or narrative discourse as shown in for constructing non-matching pairs based on similarity, we extract textual and visual representations of original descriptions or images and calculate pairwise scores between each dataset item of the same type with cosine similarity. we then take the upper quartile of the distribution of similarity scores as a threshold for high similarity items and the lower quartile as a threshold for low similar- to extract representations for texts we use the hugging face implementation of bert (devlin et al., 2018) § . for image representations we use resnet-101 @xcite , a deep residual network based on a cnn architecture. we choose these models since these are used by most languageand-vision transformers. ## conclusion in this work we examined whether a language-andvision model is sensitive to discourse structure at different levels of granularity of distractors, random, similar and dissimilar to control for the diffi-culty of the task. we focused on local and narrative discourse in the task of image-caption matching which is one of the pre-training objectives of multimodal models. we ran three experiments on vil-bert under different conditions using data from the visual storytelling dataset where images are collected into stories. the dataset has several layers of annotation including captions of images in isolation and captions of images in stories in sequence. in experiment i we test vilbert's imagecaption matching performance on the captions of images in isolation under five different conditions with different distractors, random, similar and dissimilar. we observe that the model performs better than 90% in all cases. in experiment ii we take the stories-in-sequence annotation level and test them for local discourse matching against distractors from different stories. the results drop by 0.1 in comparison with the previous experiment which shows that captions that are part of a narrative are harder to distinguish from captions from different narratives. however, the model still achieves high performance and has therefore learnt to distinguish the differences in local discourse structure by identifying from other local discourses. in experiment iii we focus on stories-insequence discourses where distracting items are sampled from the same story. the model's performance drops to 0.6. the model assigns false positives to the matching class with high confidence and therefore finds it challenging to distinguish items from the same story discourse that refer to the same entities but over a sequence of relations. the model is more successful in distinguishing between different entities and situations as shown by experiment i and ii. the effect of dis/similarity of distractors becomes more pronounced when the nature of discourse becomes more challenging. overall, such behaviour of the model is expected since it has never been trained on captions that come from stories in sequence. while this might be achieved through fine-tuning which we will attempt in our future work it also raises a more profound question about the nature of semantic knowledge captured in large models. on the one hand, the models are exposed to contextual discourse knowledge while they are trained since all descriptions are made within some context. on the other hand, their per-formance quickly degrades when the nature of the context changes. understanding the discourse context is therefore important both at the level of data creation and at the level of utilisation of pre-trained models. it is important to note that our results should be considered relative to the dataset and the model we investigated and variations are expected based on the contexts in which the model was trained and the contexts it is applied to. assessing the similarities between discourses (even by expert linguists) may not be straightforward as parameters may not be directly observable or known. in our future work, we are planning to expand our methods to other language-and-vision models, such as single-stream models, and other narrative discourse datasets, such as recipeqa, which is an example of naturally created narrative in comparison with crowd-sourced vist dataset.