How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input? (2022)

Abstract: Current language models have been criticised for learning language from text alone without connection between words and their meaning. Consequently, multimodal training has been proposed as a way for creating models with better language understanding by providing the lacking connection. We focus on pre-trained multimodal vision-and-language (VL) models for which there already are some results on their language understanding capabilities. An unresolved issue with evaluating the linguistic skills of these models, however, is that there is no established method for adapting them to text-only input without out-of-distribution uncertainty. To find the best approach, we investigate and compare seven possible methods for adapting three different pre-trained VL models to text-only input. Our evaluations on both GLUE and Visual Property Norms (VPN) show that care should be put into adapting VL models to zero-shot text-only tasks, while the models are less sensitive to how we adapt them to non-zero-shot tasks. We also find that the adaptation methods perform differently for different models and that unimodal model counterparts perform on par with the VL models regardless of adaptation, indicating that current VL models do not necessarily gain better language understanding from their multimodal training.

## introduction having models learn language from text alone has been criticised based on several aspects, from fundamental arguments about how language works @xcite to findings on lack of certain information in text @xcite . to train language models on more sources than text is therefore a proposed direction for creating language models with better language understanding @xcite . these models would then become multimodal, with the capability to process both text and information from other modalities. the multimodal models of interest in this work are vision-and-language (vl) models that have been trained on images and their corresponding captions or visual questions @xcite @xcite @xcite . these models are performant on several image-text tasks such as image captioning and vqa, while there also is an increased interest for evaluating how their natural language understanding is influenced by their multimodal training @xcite . it is however tricky to investigate the pure natural language understanding of the aforementioned vl models, since their language processing is conditioned on visual features. for certain investigations, we may simply wish to evaluate the models on textonly domains, while these models have not been developed for this purpose. if we do not attend to the issue of accurately adapting vl models to text-only domains we risk evaluating them out-ofdistribution and fail to accurately measure their natural language understanding capabilities. different methods for adapting vl models to a text-only input have already been tried and we have some results on the natural language understanding capabilities of these models @xcite . however, no systematic search for the best way to adapt vl models to a textonly input has been performed and it is unclear how well the vl models work with the previously proposed adaptations. if we wish to continue the search for better natural language understanding in multimodal models, we should ensure that we evaluate them in the best way possible. in this work, we search for the best method for adapting existing vl models to a text-only input, as illustrated in figure 1 . 1 with the adaptations in place, we can then compare the vl models to their unimodal text-only counterparts. this will complement already existing results on the natural language understanding capabilities of vl models and the effect of multimodal training. the contributions of our work are as follows: ## models we investigate adaptations to text-only input for the three multimodal models clip-bert, lxmert and visualbert. we also compare their results with those of a baseline bert-base model and flava. the models are further described below and an overview of them can be found in for each of the multimodal models, we also describe how to make the model function without visual input. this is later used in some of the adaptations we evaluate, described in section 3. all models evaluated in this work except for clip-bert are provided by the huggingface library @xcite . the pre-trained model weights for all models except for clip-bert are also provided by this library. the clip-bert weights are found in our public repository. ## adaptations to text-only input there are several ways to adapt a vl model to a text-only input. in this work we investigate and compare seven possible adaptations, as described below. two of the adaptations described here have already been used for investigating the linguistic capabilities of vl models @xcite . common for all adaptations is that their intended use is for evaluation of a pre-trained vl model (encoder) on text-only input. when we refer to the word adaptation we refer to the adaptation of a vl model to text-only input. the adaptions can be grouped into three different categories based on how they are implemented. for the first category, we simply remove the visual input to the vl model (sections 3.1 and 3.2). for the second category, we provide the model with visual features that are constant and can be viewed as fillers, (sections 3.3 to 3.6). for the third category, we provide the model with visual features predicted from text (section 3.7). ## evaluation methods to assess the performance of our text-only adaptations, we firstly evaluate them on the glue benchmark, described in section 4.1. these evaluations will give us results on how well the models and their adaptations work for general natural language understanding tasks. this benchmark has been used by both @xcite and @xcite to evaluate natural language understanding capabilities of bert and vl models. furthermore, to assess the performance of the adaptations on text domains that are more focused on visual concepts, we perform evaluations on vpn, further described in section 4.2. this will provide us with results on tasks the vl models potentially are more attuned to, complementing the general glue results. ## results we report the evaluation results for clip-bert, lxmert and visualbert with the seven potential adaptions to text-only input in figure 2 . we also report the results for our four bert-base baselines. for glue we report the macro-averaged score over the glue tasks. the score for each task is measured using its corresponding predefined metric described by @xcite . for vpn evaluation we report the mean average precision (map) score averaged over each concept and relation per query template. we also report the evaluation results on glue for each task and model for the best performing adaptation measured by average glue score in we format our discussion around a set of statements that can be made with respect to the results of this work, as follows. model performance on glue is more sensitive to pre-training than to adaptation model performance on glue varies insignificantly between different adaptations for each model in figure 2a . trained-lxme trained-lxme trained-wikipe default no-visual-featu no-visual-featu avg-visual-feat zero-image-vis zeroed-visual-f finetuned-lxm finetuned-wiki imagined-visua .60 0.65 0.70 0.75 0.80 0.85 score trained-lxmert trained-lxmert-scratch trained-wikipedia default no-visual-features-finetuned-lxmert no-visual-features-finetuned-wikipedia avg-visual-features zero-image-visual-features zeroed-visual-features finetuned-lxmert-visual-features finetuned-wikipedia-visual-features imagined-visual-features 0.55 0.60 0.65 0.70 0.75 0.80 0.85 score bert-base clip-bert lxmert visualbert model trained-lxmert trained-lxmert-scratch trained-wikipedia default no-visual-features-finetuned-lxmert no-visual-features-finetuned-wikipedia avg-visual-features zero-image-visual-features zeroed-visual-features finetuned-lxmert-visual-features finetuned-wikipedia-visual-features imagined-visual-features the clip-bert performance varies with less than 0.01 score points between adaptations and the visu-albert performance with at most 0.02 score points between 'no-visual-features-finetuned-wikipedia' and 'zeroed-visual-features'. the lxmert performance also has a performance difference of at most 0.02 score points between 'default' and 'zeroimage-visual-features'. the largest performance difference on the glue benchmark can be observed between models, where lxmert and bert-base trained from scratch on lxmert data perform significantly worse in comparison to the other models. most likely, this is due to that the models were not initialized from bert weights and consequently were not tuned to more general language usage. a possible explanation for why the adaptation methods seem to matter so little for the glue results is that the benchmark is not zero-shot. the fine-tuning performed on each task might provide a sufficient of signal to the model for it to adapt to the unimodal domain. lastly, our results on glue for lxmert and visualbert differ from those obtained by @xcite . especially with respect to lxmert for which we observe a significant performance difference compared to the other vl models, while the same cannot be observed in the results by @xcite . however, this should not raise any concerns about the robustness of the results, since we evaluated the original released models, while @xcite vpn score bert-base 49.1 ± 13.2 flava 30.7 ± 6.9 clip-bert 48.2 ± 4.2 lxmert 42.8 ± 10.8 visualbert 38.1 ± 9.1 ## related work as previously mentioned, @xcite have already looked at the language understanding capabilities of vl models, while they only looked at one way of adapting these models to a text-only input and only evaluated on glue. @xcite also look at the language understanding capabilities of vl models by evaluating the linguistic representations of visualbert and com-pare them to a bert-base model that has been trained on the same text data. in contrast to their work, we investigate several vl models and evaluate their performance on language generation tasks. @xcite , hessel and lee (2020), thrush et al. (2022) and frank et al. (2021) also perform extensive evaluations of several vl models such as lxmert and visualbert. in contrast to our work, they primarily focus on the vl performance of the models, and do not consider the model performance on text-only input. tan and bansal (2020) introduce a new method for enriching the textual representations of a model by training on visual information. their method results in a model that can be directly applied to text-only tasks and outperforms its standard bert model counterpart on glue. this method provides a parallel research avenue compared to adapting vl models to text-only input. ## conclusions we have investigated and compared seven possible adaptations of clip-bert, lxmert and visualbert to text-only input by evaluation on glue and visual property norms. we can conclude that care should be put into adapting these pre-trained vl models to text-only input for better performance on zero-shot tasks, while the choice of adaptation method seems to be less impactful on tasks coupled with fine-tuning sets. finally, we have observed that a unimodal lm has a performance on text-only tasks that is better or comparative to that of its vl model counterparts, regardless of how these counterparts were adapted to text-only input. seemingly, improved pure text capabilities are not guaranteed from simply training a model on arbitrary multimodal tasks. this agrees with and solidifies previous research results on vl models.