Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning (2023)

Abstract: The robustness of multimodal deep learning models to realistic changes in the input text is critical for applicability on important tasks such as text-to-image retrieval and cross-modal entailment. To measure robustness, several existing approaches edit the text data, but without leveraging the cross-modal information present in multimodal data. Such information from the visual modality, such as color, size, and shape, provides additional attributes that users can include in their inputs. Thus, we propose cross-modal attribute insertions as a realistic perturbation strategy for vision-and-language data that inserts visual attributes of the objects in the image into the corresponding text (e.g., “girl on a chair” to “little girl on a wooden chair”). Our proposed approach for cross-modal attribute insertions is modular, controllable, and task-agnostic. We find that augmenting input text using cross-modal insertions causes state-of-the-art approaches for text-to-image retrieval and cross-modal entailment to perform poorly, resulting in relative drops of ~15% in MRR and ~20% in F1 score, respectively. Crowd-sourced annotations demonstrate that cross-modal insertions lead to higher quality augmentations for multimodal data than augmentations using text-only data, and are equivalent in quality to original examples. We release the code to encourage robustness evaluations of deep vision-and-language models: https://github.com/claws-lab/multimodal-robustness-xmai

## introduction the ability to model the interaction of information in vision and language modalities powers several web applications -text-to-image search @xcite , summarizing multimodal content @xcite , visual question answering @xcite ## cross-modal attribute insertions our approach for augmenting text in multimodal data involves identifying objects in the image that are also mentioned in the text, and inserting words similar to their attributes in the text at relevant places. an overview of our approach (xmai) is depicted in figure 2 . we denote paired multimodal units as (i, t ), where i represents the input image and t is the text corresponding to that image. our goal is to transform t into t ′ such that the text includes relevant information from i while effectively highlighting the target model's vulnerabilities. our method to infuse object attributes in the text can be broken into four parts: (a) object and attribute detection in i, (b) bert-based [mask] prediction in t while ensuring (c) similarity of inserted tokens with detected object attributes, and (d) enforcing dissimilarity between modified text t ′ and i to obtain robustness estimates of multimodal models. object and attribute detection: for each image i we use a pre-trained bottom-up attention model @xcite to extract objects and their associated attributes. the bottom-up attention model identifies objects and corresponding attributes with a one-to-one mapping. we use these objects and attributes to modify t , by intro-ducing masks (i.e., the [mask] token), in front of the mentions of the objects in t . however, a strict matching criterion would ignore similar objects or alternatively named objects in t . to address this, whenever the text does not have any direct object matches we use a parts of speech (pos) tagger to identify nouns that could represent objects in the image. these identified nouns are compared to objects using cosine similarity between the word embeddings. if the cosine similarity between a noun t and a detected object in i is above some threshold, t, then a [mask] token is placed before that noun. overall, this step ensures that insertions are made only for objects in t that are seen in the corresponding image i to obtain t ′ . mask prediction: next, we aim to fill in the [mask] tokens with contextually relevant object attributes. to do so, we use the pre-trained language model bert @xcite . we sample top-k predictions from the bert model based on probability scores that also meet the following criteria: the predicted word should not be a stop word and should not exist in the 3-hop neighborhood of the current @xcite . furthermore, since t may contain more than one [mask] token, we carry out this process sequentially for each @xcite to utilize newly introduced contexts. following this process, we obtain k candidate insertions that are contextually relevant for each of the identified objects in t that also exists in i. in the next step, to maintain cross-modal relevance, we consider the similarity of these candidate attributes with the attributes detected in i. ## experiments we study the effect of cross-modal attribute insertions on two multimodal tasks: text-to-image retrieval (i.e., retrieving images for a textual description) and cross-modal entailment (i.e., predicting the relationship between textual hypothesis and visual premise). ## results and analysis recall that our primary goal is to use xmai to obtain a complementary and novel set of text augmentations that can highlight the vulnerabilities of multimodal models. to this end, we contrast the performance of the models under investigation on the original and the modified examples, and quantify the relevance of the modified text with respect to the original text and image. we recruit human annotators to compare the quality of the augmentations generated using our approach with (i) the ones generated using the most competitive baseline, and (ii) the original text. robustness of multimodal models: relevance of augmentations: tables 1 and 2 show that xmai produces augmentations t ′ that maintain high-relevance with the original text t and the image i, in terms of sim t -t ′ and sim i-t ′ . it is interesting to note that the bleu scores for augmentations generated by xmai are notably lower than that for the baseline augmentations. on the contrary, meteor scores show that xmai's augmentations are "closer" to the original texts compared to most baselines. xmai's poor bleu scores can be largely attributed to bleu's tendency to penalize novel insertions severely compared to removals or replacements, as it is a precisionbased metric @xcite . 3 in appendix a.3 ( we thus attribute the poor performance of xmai in terms of bleu scores to bleu's inability to handle insertions appropriately. this is further substantiated by the human assessments. ## conclusion a robust understanding of vision-and-language data is crucial for powering several applications. we propose cross-modal attribute insertions -i.e., adding the visual attributes to text, as a new variation that is likely in multimodal data and to which multimodal models should be robust. our approach produces novel augmentations that are complementary to existing methods that model text-only data, and are preferred over them by human annotators. using our augmentations we effectively highlight the vulnerabilities of state-of-the-art multimodal models for text-to-image retrieval and cross-modal entailment. in the future, we aim to empirically study the effect of including xmai augmented data in task-specific training sets and expand to a broader set of multimodal tasks and metrics.