B e D ifferent to B e B etter! A B enchmark to L everage the C omplementarity of L anguage and V ision (2020)

Abstract: This paper introduces BD2BB, a novel language and vision benchmark that requires multimodal models combine complementary information from the two modalities. Recently, impressive progress has been made to develop universal multimodal encoders suitable for virtually any language and vision tasks. However, current approaches often require them to combine redundant information provided by language and vision. Inspired by real-life communicative contexts, we propose a novel task where either modality is necessary but not sufficient to make a correct prediction. To do so, we first build a dataset of images and corresponding sentences provided by human participants. Second, we evaluate state-of-the-art models and compare their performance against human speakers. We show that, while the task is relatively easy for humans, best-performing models struggle to achieve similar results.

## introduction human communication, in real-life situations, is multimodal @xcite : to convey and understand a message uttered in natural language, people build on what is present in the multimodal context surrounding them. as such, speakers do not need to "repeat" something that is already provided by the environment; similarly, listeners leverage information from various modalities, such as vision, to interpret the linguistic message. integrating information from multiple modalities is indeed crucial for attention and perception @xcite since combined information from concurrent modalities can give rise to different messages @xcite . the argument that language and vision convey different, possibly complementary aspects of meaning has been largely made to motivate the need for multimodal semantic representations of words @xcite . however, computational approaches to language and vision typically do not fully explore this complementarity. to illustrate, given an image (e.g., the one depicted in figure 1 ), popular tasks involve describing it in natural language, e.g., "a tennis player about to hit the ball" (image captioning; see @xcite ; answering questions that are grounded in it, e.g., q: "what sport is he playing?", a: "tennis" (visual question answering; see @xcite ; having a dialogue on its entities, e.g., q: "is the person holding a racket?", a: "yes." (visuallygrounded dialogue; see de @xcite . while all these tasks challenge models to perform visual grounding, i.e., an effective alignment of language and vision, none of them require a genuine combination of complementary information provided by the two modalities. all the information is fully available in the visual scene, and language is used to describe or retrieve it. in this work, we propose a novel benchmark, be different to be better (in short, bd2bb), where the different, complementary information provided by the two modalities should push models develop a better, richer multimodal representation. as illustrated in figure 1 , models are asked to choose, among a set of candidate actions, the one a person who sees the visual context depicted by the image would do based on a certain intention (i.e., their goal, attitude or feeling). crucially, the resulting multimodal input (the sum of the image and the intention) will be richer compared to that conveyed by either modality in isolation; in fact, the two modalities convey complementary or nonredundant information @xcite . to illustrate, a model that only relies on the (nongrounded) linguistic information conveyed by the intention, i.e., "if i have tons of energy", might consider as equally plausible any actions that have to do with playing a sport, e.g., "i will play basegiven an image depicting, e.g., a tennis player during a match and the intention "if i have tons of energy", the task involves choosing, from a list of 5 candidate actions, the target action that unequivocally applies to the combined multimodal input: "i will play a game of tennis with the man". the task is challening: a model exploiting a language or vision bias could fall into the trap of decoy actions containing words highlighted in blue or orange, respectively. therefore, selecting the target action requires models perform a genuine integration of the two modalities, whose information is complementary. best viewed in color. ## related work since the introduction of the earliest multimodal tasks, such as image captioning (ic; see @xcite and visual question answering (vqa; @xcite , a plethora of tasks dealing with language and vision have been proposed. in parallel, baseline models have been replaced by more powerful attention-based systems @xcite and, more recently, by transformer-based architectures pretrained on several tasks @xcite @xcite . these latter models build on multimodal representations that are meant to be task-agnostic; as such, they can be transferred to virtually any other multimodal task with minimal fine-tuning. our work contribute to these two lines of research by (1) introducing a novel multimodal task, and (2) by evaluating a soa multimodal encoder on it. multimodal tasks vqa was originally proposed to overcome the challenge of quantitatively evaluate ic models. the task (and its evaluation) moving from objects to actions, several tasks have been proposed to mimic more realistic settings where a higher degree of integration between modalities is required. one is visual storytelling @xcite @xcite , where models have to understand the action depicted in each photo and their relations to generate a story. similar abilities are required in the task of generating non-grounded, human-like questions about an image @xcite , and in that of asking discriminative questions over pairs of similar scenes @xcite . related tasks are also those of predicting motivations of visually-grounded actions @xcite or generating explanations for a given answer @xcite . an even higher level of understanding of vision and language is required in the tasks of filling the blank with the correct answer @xcite ; answering questions from videos and subtitles @xcite ; having a dialogue on objects @xcite or events @xcite ; answering and justifying commonsense questions @xcite . however, all these tasks require making commonsense inferences over the two modalities rather than integrating their complementary infor-mation to answer a grounded question. more akin to ours are the approaches by @xcite , which aims to predict the subsequent scene and dialogue in a comic strip, and @xcite , where the goal is to compute the communicative intent of a social media post. though they both require a challenging integration of language and vision, these tasks (as well as the type of data they use) are crucially different from bd2bb, where the task is to predict the action that is consequent to a given intention based on the image. transformer-based multimodal models developing universal multimodal encoders whose pretrained representations are suitable for virtually any multimodal task is a crucial challenge. inspired by the success of bert, a pretrained transformerbased language encoder @xcite , similar architectures have been recently proposed in the domain of language and vision @xcite @xcite @xcite . while these architectures achieve state-of-the-art performance in many tasks, their novelty and complexity leave several questions open, and further work is needed to better understand, e.g., which layers are more suitable for transferability @xcite , or what is the relation between pretraining and downstream tasks @xcite . moreover, to prove they are readily applicable to novel multimodal benchmarks, pretrained universal encoders should be ideally effective with only minimal fine-tuning on the target tasks. in this light, we believe that more efforts should be put in developing datasets that are challenging and yet relatively small, in line with the 'diagnostic' datasets proposed for vqa @xcite and the easy vs. hard subsets introduced by akula et al. (2020) for visual referring expression recognition. our contribution follows this line of thought. where each of the 10, 191 images depicts at least one person. this choice was aimed to make the participants' task more natural: indeed, the presence of people in the image allows more possibilities of interaction, and therefore guarantees that some actions can be performed in that situation. ## task we introduce the be different to be better (bd2bb) task, where the different, i.e., complementary information provided by the two modalities should push models develop a better, i.e., richer multimodal representation. to evaluate these abilities, we frame our task as a multiple-choice problem (similar to @xcite @xcite where either modality is necessary but not sufficient to perform a correct prediction. the task is the following (see figure 1 ): given an image and a corresponding intention, the model has to choose the correct action over a set of 5 candidate actions. we refer to the correct action as the target action; to the wrong actions as the decoy actions. similarly to @xcite , decoy actions are carefully selected to be as plausible as possible when evaluated against either the intention (2 decoys) or the image (the other 2) only. below, we explain how language-based and image-based decoys were selected based on the meta-annotation. language-based decoys for each of the 10, 287 image, intention, action datapoints, we randomly selected a number of datapoints from the entire data that had the following criteria: 1) their action belonged to a different topic cluster than the one including the target action; 2) their action did not share any noun with the target action, i.e., their code2 and code4 were different. we then computed a similarity score between the target action and each of these selected actions by means of the cosine of their use representations. we ranked these scores and selected as our language decoys the two with the highest similarity. this way, we obtained language-based decoys that are semantically very similar to the target action, but are on a different topic and do not share any noun with it. vision-based decoys for each datapoint, we randomly selected a number of datapoints from the entire data that had the following criteria: 1) their action belonged to the same topic cluster of the target one; 2) their action did not share any verb with the target action, i.e., their code1 and code3 were different. we then ranked these actions with respect to their use similarity with the target one, and selected as our vision-based decoys the two with the lowest score. this way, we obtained visionbased decoys that are about the same topic of the target action; at the same time, they do not share any verbs with it and are semantically different. ## experiments to test the importance of combining information from the two modalities and the independent contribution of either modality, we experiment with 3 settings of the bd2bb task: l, where the target 7 for 22 datapoints it was not possible to find all the decoys, hence they were discarded during the creation of the dataset. action among the 5 candidates has to be guessed based on the intention only; v, where only the image is provided; lv, where both the image and the intention are provided. for each setting of the task, we evaluate the performance of (1) a simple baseline trained from scratch on the task; (2) a state-ofart transformer-based pretrained model fine-tuned on the task; (3) the same transformer-based model trained from scratch on the task. moreover, results by models are compared to (4) human performance. ## results results by both models and humans are reported in multimodal integration is the key. the overall best-performing model in bd2bb is lxmert lv (62.2%), which outperforms the other pretrained models, i.e., roberta l (56.2%) and lxmert v (59.2%). on the one hand, this shows that disposing of both modalities is beneficial to perform the model accuracy @xmath0.0 ± 0.4 44.6 ± 0.8 baseline l 45.3 ± 0.9 45.9 ± 0.9 baseline v 45.8 ± 0.8 46.1 ± 0.8 baseline lv 48.6 ± 0.9 49.0 ± 0.9 scratch roberta task. this is in line with the results by human participants, who achieve the highest accuracy in the multimodal setting (79% vs. 50% of l and 72.3% of v). on the other hand, the finding that lxmert v surpasses roberta l (+3%) confirms that the image provides more information compared to the intention. this, again, is consistent with human results, where the gap between v and lv (-7%) is much smaller compared to that between l and lv (-29%). for humans, this visual advantage is likely due to (ms-coco) images depicting complex events that elicit a broad range of aspects related to people's experience of the world. as for the models, it confirms that lxmert, thanks to its massive pretraining, is effective in extracting fine-grained information from images. models are far from humans. humans achieve around 80% accuracy ('best' 82%) on the multimodal version of the task. this is a high result, in line with previous work with a similar setup (consider, e.g., swag, where 'expert' human accuracy is around 85% with 4 choices, i.e., chance level at 25%; @xcite . at the same time, the non-perfect human accuracy reveals that the benchmark is challenging due to the careful selec-tion of plausible decoys. compared to humans, the best-performing lxmert lv achieves much lower results (-17%), which indicates that bd2bb is challenging and far from being solved. since the gap between best-performing models and human participants in unimodal settings is smaller (-13% in v and -6% in l), the biggest computational challenge lies in the integration of complementary information from different modalities. pretrained is better. pretrained models neatly outperform the baseline in all the versions of the task foot_5 and, more interestingly, also all their counterparts trained from scratch. as can be seen in ## analysis best models' errors we perform an analysis on the errors made by the 3 pretrained models to check whether they fall more often into the languagebased or vision-based decoys. to do so, we focus on each model's best run, and compute the proportion of wrong predictions in the test set that belong to one or the other decoy type. for comparison, a model that makes modality-balanced wrong predictions should fall into language-/visionbased decoys 50% of the times. quite surprisingly, roberta l has only a moderate bias toward language-based decoys: in fact, only 60.2% of its errors are of this type. as for lxmert v , no bias at all is observed toward the vision-based decoys (48.6%). finally, the best-performing lxmert lv is shown to be halfway between these models, with only a slight preference for language-based (55.1%) over vision-based decoys (44.9%). in figure 5 , we report two cherrypicked examples where lxmert lv either correctly predicts the target action (left) or choses a wrong one, in this case a vision-based decoy (right). it is worth mentioning that these two cases are challenging: for i: if i am in the mood to act silly, i will. . . if i don't like this, i will. . . l: attend a dinner like this man holding a gift sit next to the woman on the bench l: buy him a cake and invite his friends to party get my face painted t: act silly with this man and eat cake avert my eyes from the man who looks silly v: help my child cut their cake teach him how to tie a tie v: have cake with soldiers wear a costume and march in a parade both of them, human annotators were able to pick the correct action only in the multimodal version of the task-but neither in l nor in v. as can be seen, in the leftmost example the model does a good job in combining complementary information from language and vision. in the rigthmost one, instead, it picks an action that is very much plausible based on the image, but not in presence of the given intention containing a negation (don't). taken together, these analyses indicate that no simple strategies can be exploited by models to detect and rule out decoy types. language-and vision-based decoys are equally challenging, and combining complementary information is needed to solve the task. hard test to explore the robustness of the pretrained models, we check how well they perform on a subset of the test set where several features of the samples were unseen in training. in particular, neither the image nor the intention were seen in training; moreover, the target action could be seen as a decoy but never as the target. in in line with the standard test set, lxmert lv still outperforms the unimodal models, though its drop in performance (-4%) is more pronounced compared to them (-1/2%). this suggests that part of the advantage of the multimodal system over model accuracy humans hard @xmath1.1 ± 1.6 56.5 lxmert v 56.9 ± 0.8 73.9 lxmert lv 58.3 ± 2.7 78.3 the unimodal ones is due to its fine-tuning. indeed, pretraining on its own is not enough to properly combine complementary information from the intention and the image. finally, since humans do not perform worse in these samples, the performance gap with lxmert lv increases to ∼ 20%. ## conclusion inspired by real-life communicative contexts where language and vision are non-redundant, we proposed a novel benchmark to challenge models combine complementary multimodal information. this is a crucial ability that, we believe, our benchmark will contribute push further. in particular, recently proposed universal multimodal encoders can greatly benefit from relatively small but challenging resources as is bd2bb, which can be used to shed light on model abilities and help developing architectures which exhibit more human-like skills. here, we evaluated lxmert and showed that it struggles to achieve results that are comparable to those by humans. in the future, we plan to evaluate other multimodal encoders on it, and to contribute to the development of better multimodal systems.