Can Pre-training help VQA with Lexical Variations? (2020)

Abstract: Rephrasings or paraphrases are sentences with similar meanings expressed in different ways. Visual Question Answering (VQA) models are closing the gap with the oracle performance for datasets like VQA2.0. However, these models fail to perform well on rephrasings of a question, which raises some important questions like Are these models robust towards linguistic variations? Is it the architecture or the dataset that we need to optimize? In this paper, we analyzed VQA models in the space of paraphrasing. We explored the role of language & cross-modal pre-training to investigate the robustness of VQA models towards lexical variations. Our experiments find that pre-trained language encoders generate efficient representations of question rephrasings, which help VQA models correctly infer these samples. We empirically determine why pre-training language encoders improve lexical robustness. Finally, we observe that although pre-training all VQA components obtain state-of-the-art results on the VQA-Rephrasings dataset, it still fails to completely close the performance gap between original and rephrasing validation splits.

## introduction visual question answering (vqa) @xcite is an image conditioned question answering task which has gained immense popularity in vision & language community. since the introduction of the vqa challenge 2 , there has been significant progress in the field of vqa, where new model architectures and training techniques are closing the gap between the model and oracle accuracy on benchmarking datasets like vqa2.0 @xcite . a majority of models obtained higher gains 1 the work was done prior to joining amazon. 2 https://visualqa.org/challenge.html figure 1 : example from vqa-rephrasings dataset @xcite . the answers are obtained using pythia @xcite where green text refers to correct answer and red text refers to wrong answer. by introducing semantically rich visual features @xcite , efficient attention schemes @xcite , and advance multimodal fusion techniques @xcite . however, to deploy these state-of-the-art vqa models into real-world settings, the models must be robust to linguistic variations that originate from interactions with real users. recently, @xcite showed that state-of-the-art vqa models @xcite are extremely sensitive to the lexical variations which result in a significant performance drop on the vqa test datasets when the questions are replaced with their rephrases. figure 1 shows the shift in confidence scores of answers for a rephrasing of the original question. to handle these scenarios, they provided a model-agnostic cyclic-consistency (cc) approach that generates question rephrases on the fly during training, which makes the underlying vqa model lexically robust. the best-reported model with their approach achieves 56.59% vqa accuracy on question rephrasings. nevertheless, all the models that @xcite experimented with their cc framework in-corporate an rnn based language encoder. recently, transformer-based models @xcite led to immense improvements in the whole nlp task spectrum @xcite . multiheaded self-attention, the core of transformer architecture, encodes the relationship of a word with its neighbors in several different representational subspaces, thus making these representations robust to linguistic variations. since existing datasets expose vqa models to a small subset of the language distribution, it leads to incorrect inference when the model receives rephrasings of the original question. although training on large datasets may overcome the problem, however, building such extensive annotated datasets is time-consuming & cost-intensive. pretrained models like ulmfit @xcite , bert @xcite , and gpt @xcite have improved performances on various nlp tasks @xcite in this paper, we study the impact of using pretraining methods to make vqa models linguistically robust. our contributions are summarized as follows: ## background in this section, we explain the building blocks of our experiments in this study. sbert @xcite 3 is a bert-based language encoder that generates semantically rich sentence embeddings. it uses siamese and triplet networks @xcite to finetune bert @xcite , which is 3 https://github.com/ukplab/sentence-transformers a pre-trained transformer encoder trained on large amounts of monolingual data. it obtains state-ofthe-art results on common semantic textual similarity and transfer learning tasks. butd @xcite foot_0 uses a gru to encode input questions and uses them to attend image roi features, enabling region-based attention to generate the answer. butd is the base architecture for many other vqa architectures like pythia @xcite and ban @xcite . lxmert (tan and bansal, 2019) is a visionlanguage cross-modality pre-training framework. in contrast to single modality pre-training like bert, lxmert focuses on vision-language interactions, which helps to understand better visual contents, language semantics, and the relationship between them. it contains three transformer encoders, namely an object relationship encoder, a language encoder, and a cross-modality encoder, pre-trained using five different vision-language tasks. it must be noted that lxmert is just a placeholder for transformer-based vqa architectures to investigate if a model architecture plays any role in improving lexical robustness. ## discussion since pre-trained language models like bert are trained on large and diverse data, it is generally hypothesized that such models are very robust to linguistic variations. our results show that pre-trained language encoders like sbert indeed improve the performance of rep split by 5.42% relative to a gru encoder; however, it still underperforms by 9.37% relative to semantically similar ori questions, modeled by a gru encoder. we observed a similar trend with task-specific multimodal pretraining as well, where lxmert (c) struggles to close the relative performance gap of about 10% between rep and ori. in this work, we show that pre-training indeed improves the linguistic robustness of vqa models while simultaneously revealing the limitations of pre-trained language encoders for standard tasks. ## conclusion and future work in this paper, we show that pre-trained language encoders, like sbert, produce semantically similar embeddings for multiple rephrases of a sentence by latching on keywords, thus making vqa models robust to lexical variations. combining crossmodal pre-training with transformer-based vqa architectures obtains state-of-the-art results on the vqa-rephrasings dataset. in the future, we plan to investigate the factors that prevent closing the accuracy gap between ori & rep despite using extensive cross-modal pretraining. further, we will study why some answer classes like number benefits the most from pretraining while others achieve significantly less relative performance gains.