Semantically Distributed Robust Optimization for Vision-and-Language Inference (2022)

Abstract: Analysis of vision-and-language models has revealed their brittleness under linguistic phenomena such as paraphrasing, negation, textual entailment, and word substitutions with synonyms or antonyms. While data augmentation techniques have been designed to mitigate against these failure modes, methods that can integrate this knowledge into the training pipeline remain under-explored. In this paper, we present SDRO , a model-agnostic method that utilizes a set linguistic transformations in a distributed robust optimization setting, along with an ensembling technique to leverage these transformations during inference.Experiments on benchmark datasets with images (NLVR ^2 ) and video (VIOLIN) demonstrate performance improvements as well as robustness to adversarial attacks.Experiments on binary VQA explore the generalizability of this method to other V&L tasks.

## introduction the left and right image contains the same number of sailboats sailing on the water. ## sisp sentence transformations this section describes the generation of semanticspreserving (sp) and semantics-inverting (si) statements. sisp transforms are implemented using spacy @xcite . dataset statistics and additional visualizations are in the appendix. noun synonym/antonym: we extract nouns (subjects and objects) with dependency parsing, and find two nearest (synonyms) or farthest (antonyms) neighbors in the glove space @xcite using a threshold of 0.55. ## experiments datasets. for all datasets, given images/videos and natural language text as input, the system is expected to predict a binary class label. nlvr 2 (suhr et al., 2019) contains ∼86k, 7k, 7k samples for training, development, and testing respectively. each sample in nlvr 2 consists of a pair of images (from search engines) and a sentence (crowdsourced). violin @xcite ) contains video clips from popular tv shows and movies along with subtitles and crowd-sourced statements. vio-lin contains 76k, 9.5k, 9.5k samples for training, validation and testing. vqa yes/no consists of image-question-answer triplets from vqa-v2 dataset @xcite . while vqa-v2 consists of multiple question and answer types, we focus on the subset of questions with binary yes/no answers (∼38% of vqa-v2). evaluation metrics. we use two evaluation metrics: (1) clean accuracy: accuracy on the i.i.d. benchmark test set, and (2) sisp accuracy: average performance on sisp transformations of the test set. since sisp transformations are automated and can be noisy (sec 3.1), evaluation on the sisp test set can be considered a proxy for robustness. ## related work adversarial training (at) has been studied under a game-theoretic @xcite and minmax setup @xcite . @xcite use at to adversarially augment image classification datasets and show improved domain generalization for digit classification. wong and kolter (2020); @xcite modify at for realworld adversaries beyond norm-bounded perturbations. at has been used for text classification with lstms @xcite and for pretraining transformer-based models by adding labelpreserving adversarial perturbations to embeddings of word tokens @xcite @xcite contrastive examples have been explored, collected from humans @xcite , negative mining @xcite , or synthetic generation @xcite @xcite . robustness in v&l has been explored for vqa, such as performance under prior probability shift @xcite and domain adaptation @xcite , along with robustness for implied questions @xcite and novel compositions @xcite , and robustness to logical connectives (including negation) @xcite . @xcite have shown that many v&l, image classification, and sentiment analysis models are sensitive to image editing. there has been a recent effort of model-in-the-loop dataset collection to guide humans to create harder vqa samples @xcite . robustness in nlp: generation of sp adversarial examples @xcite @xcite , and approaches to defend against word substitution @xcite have been explored in natural language processing tasks. evaluation datasets have also been proposed for textual entailment that are manually crafted @xcite or templatebased @xcite @xcite . our method uses automated linguistically-informed si and sp transforms for both training and inference. ## discussion on ensembling coefficients. while designing our ensembling approach, we used α = 0.5, i.e., equal contribution from the original output and the average of all outputs for transformed samples. this choice is generic and does not rely on datasetor model-specific characteristics of sisp accuracy. while treating α as a hyperparameter and tuning it on validation datasets could lead to further gains, our intuitive choice of α = 0.5 is effective by itself. on si samples. tables 3 , 4 , 5 show that existing models perform well on sp transforms, implying that equivalent semantics are captured in transformer-based models. however, these models fail on si samples resulting in a close-to-random (50%) average sisp accuracy. while images perturbed with noise, blur, weather, or digital artifacts @xcite retain semantics (an image of a "cat" remains a cat after perturbation), minimal changes to text inputs, such as a single word changing from "sitting" to "standing" or "not sitting", inflict large changes in meaning. we hope that future work on design of v&l evaluation criterion along the si axis, could benefit from our findings. while we generated si and sp text for vli tasks, the idea could be extended to design sisp transformations for images, by operating at object-level instead of pixel-level on combination of at and sdro. we show that combining at with sdro can improve vli performance and incorporate domain knowledge into the training process, such as semantic knowledge that often exists in natural language or linguistic rules. this is explicitly observed with villa, which is pre-trained and fine-tuned using standard adversarial training @xcite . when fine-tuned with sdro, villa+sdro further improves compared to uniter+sdro. the combination of standard adversarial training, (which accounts for local adversaries inside a norm-ball) and sdro, (which accounts for linguistic adversaries and contrastive examples, typically outside the norm-ball as shown in figure 2 ) could lead to improved generalization in many other v&l tasks. on differentiability. linguistic transformations are not differentiable and prohibit gradient-based solutions to the inner maximization in sdro. however, most v&l tasks would benefit from the incorporation of semantic knowledge into the optimization framework. through sdro, we show that explicitly choosing the argmax over a pre-defined set of transformations leads to model-agnostic improvements for binary classification tasks in v&l. more sophisticated methods may emerge in the future to address non-differentiability by leveraging proximal point or trust-region methods @xcite or interval bound propagation @xcite , to incorporate semantic knowledge into adversarial training.