Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models (2021)

Abstract: A method for creating a vision-and-language (V&L) model is to extend a language model through structural modifications and V&L pre-training. Such an extension aims to make a V&L model inherit the capability of natural language understanding (NLU) from the original language model. To see how well this is achieved, we propose to evaluate V&L models using an NLU benchmark (GLUE). We compare five V&L models, including single-stream and dual-stream models, trained with the same pre-training. Dual-stream models, with their higher modality independence achieved by approximately doubling the number of parameters, are expected to preserve the NLU capability better. Our main finding is that the dual-stream scores are not much different than the single-stream scores, contrary to expectation. Further analysis shows that pre-training causes the performance drop in NLU tasks with few exceptions. These results suggest that adopting a single-stream structure and devising the pre-training could be an effective method for improving the maintenance of language knowledge in V&L extensions.

## introduction pre-trained vision-and-language (v&l) models improve the performance of tasks that require an understanding of the v&l grounding, including visual question answering @xcite , referring expression comprehension @xcite , and image-text matching (itm) @xcite . recent v&l tasks, such as multimodal reading comprehension @xcite @xcite and dialogue @xcite @xcite , require a deeper nlu as well as the grounding. extending pre-trained language models (lms) is an option for those tasks as this allows v&l models to inherit language knowledge from their source lms. the typical extending consists of visual pre-training and structure such as the stream type; the single-stream inserts vision tokens into the input sequence of the lm, and the dual-stream uses another sequence for early visual encoding. one of the remaining challenges is to understand how such extensions affect the pre-trained language knowledge. for example, @xcite proposed the dual-stream model where part of the goal was to protect the learned lms. the authors focused on evaluation with v&l tasks and did not evaluate their models with language-only tasks. @xcite evaluated the extent of language knowledge loss in the single/dual-stream models against the source lm using language-only tasks. however, the difference between single-stream and dualstream models was unclear because the pre-training was also different in their models. in this paper, we investigate the effect of visual extensions in v&l models on language-only tasks foot_0 . @xcite proposed a framework to unify transformer-based v&l models and compared some single/dual-stream models in the same setup. based on their work, our study shows how these structural differences affect the performance of nlu using the glue @xcite tasks. in our experiments, fine-tuning of pre-trained v&l models shows that both single/dual-stream models perform worse than the source lm and that single-stream models perform slightly better than dual-stream models. further, we fine-tune the models created by only structural modifications without pre-training. we observe that the single/dual modification alone has little effect on the glue scores, indicating the performance degradation is primarily caused by pre-training. we also see how the v&l models changed from the source lm by analyzing the changes in the model parameters and the problem sets that each model can solve. our results suggest that it would be more effective to adopt a single stream, and devise pre-training strategies for maintaining language knowledge. ## controlled v&l models in this section, we describe the pre-trained v&l models used in our experiments. @xcite proposed a framework for v&l models that consider a sequence of tokens in sentences as language information, and a sequence of recognized object regions as visual information. in their framework, they reproduced five existing models, visual-bert @xcite , uniter @xcite , vl-bert @xcite , vilbert @xcite , and lxmert @xcite , and made their controlled versions by modifying some parts for a fairer and easier comparison. we use these controlled versions. ## discussion and conclusion the number of v&l model works that focus on both v&l tasks and language-only tasks has increased @xcite @xcite . @xcite proposed a v&l neural architecture and trained it on a language model in a visual context. they demonstrated that their architecture outperforms its equivalent trained on language alone in perplexity and stated that language is inseparable from its physical context. although it is not clear whether methods that improve the perplexity of language modeling can also apply to maintain the performance of downstream tasks, the strategy of improving models with reference to human cognition would be an important direction. more recently, @xcite achieved better performance on language-only tasks than their base model with pre-training on three types of corpora (text, image, and image-text pairs) at the same time. @xcite reported that adding separated extractors for vision and language on top of a single-stream encoder can help maintain language knowledge. in this paper, we fine-tuned v&l models extended from a language model (lm) to an nlu benchmark to compare their nlu performance. we used five v&l models, including single-stream and dual-stream models, pre-trained in the same setup. the benchmark scores of those models decreased compared with their source lm. we also found that the single-stream models tended to retain (slightly) more language knowledge than the dual-stream models, and that the main cause of the drop in the nlu tasks can be pre-training. our observations suggest that adopting a single stream and devising pre-training strategies could be effective, at least for preserving the language knowledge.