Visually-augmented pretrained language models for NLP tasks without images (2023)

Abstract: Although pre-trained language models (PLMs) have shown impressive performance by text-only self-supervised training, they are found lack of visual semantics or commonsense. Existing solutions often rely on explicit images for visual knowledge augmentation (requiring time-consuming retrieval or generation), and they also conduct the augmentation for the whole input text, without considering whether it is actually needed in specific inputs or tasks. To address these issues, we propose a novel **V**isually-**A**ugmented fine-tuning approach that can be generally applied to various PLMs or NLP tasks, **W**ithout using any retrieved or generated **I**mages, namely **VAWI**. Experimental results show that our approach can consistently improve the performance of BERT, RoBERTa, BART, and T5 at different scales, and outperform several competitive baselines on ten tasks. Our codes and data are publicly available at https://github.com/RUCAIBox/VAWI .

## introduction recent years have witnessed the success of pretrained language models (plms) @xcite , such as gpt-3 @xcite and t5 @xcite , in a variety of natural language process (nlp) tasks. since these plms are mostly trained on text-only corpus via self-supervised pre-training, they have been shown lack of visual commonsense @xcite and real-world knowledge @xcite . as a result, plms can't well solve visually related language tasks foot_0 , e.g., answering the color and size of common things, especially those requiring complex commonsense knowledge. to alleviate this problem, existing works mainly enhance plms by infusing visual information. typ-ically, given a text input, these studies firstly augment the visual information from retrieved or generated images about the input and then leverage their visual representations to improve plms on nlp tasks. such an approach leads to visuallyaugmented pre-trained language models (valms), where they adopt either visually-augmented pretraining @xcite or visually-augmented fine-tuning @xcite . despite the effectiveness, there are two major shortcomings in these methods. first, these methods often rely on pre-learned complementary retrievers or generators, and also require time-consuming inference to retrieve or generate proper images that are paired with the input. the above costly conditions largely limit the applicability of these approaches. second, the retrieved or generated images are inevitable to involve irrelevant or redundant visual information. if simply integrating them, the original text representations might be affected.increasing evidence shows that the visual information is not always useful for nlp tasks @xcite , and sometimes leads to performance degradation. considering these issues, we aim to develop a more efficient and effective way to visually augment the plms and the solution is twofold: such a way is simple and efficient: we can only keep the text encoder of a vl-ptm to produce the visually-aligned representations of texts, getting rid of the complicated image retrieval or generation process. it is widely recognized that there is a large semantic gap between different modalities @xcite . our method can alleviate this issue to some extent since the visual augmentations are derived from the text representation itself. to this end, in this paper, we propose a general visually-augmented fine-tuning approach to improving plms for nlp tasks without images, namely vawi. our approach consists of three ingredients, namely visually-hungry words extraction, visual knowledge augmentation, and visuallyenhanced fine-tuning. given the text input from a nlp task, we first extract the visually-hungry words (vh-words) from the input sentence. as the annotations of vh-words are generally unavailable, we propose three strategies to automatically extract the vh-words, relying on the syntax trees, attention distributions of vl-ptms, and an adaptive learnable module, respectively. then, based on the extracted vh-words, we leverage the text encoder of clip @xcite (being fixed in our approach), a vl-ptm that has been pre-trained on millions of text-image pairs, to encode the vhwords for obtaining their visually-aligned representations. finally, we infuse the visually-aligned representations into plms, and consider the general and parameter-efficient fine-tuning strategies for small and large plms, respectively. to verify the effectiveness of our framework vawi, we test it on four plms (i.e., bert, bart, roberta, and t5) at different scales (i.e., 110m, 340m, 3b), and conduct extensive experiments in natural language understanding, commonsense reasoning, and text generation tasks. ex-perimental results show that our vawi can boost the performance of these plms significantly, i.e., 3.11%, 2.54%, and 2.16% absolute improvements on the commonsenseqa task using roberta-base, roberta-large, and t5-3b, respectively. besides, vawi can outperform (or be on par with) several competitive baselines that adopt complicated visually-augmented methods. ## method in this section, we firstly introduce the task setting, and then describe our proposed visual augmentation approach for infusing visual knowledge into plms during fine-tuning. ## conclusion in this paper, we proposed a general visuallyaugmented fine-tuning approach that can be applied to a variety of plms and nlp tasks, without using any retrieved or generated images, namely vawi. specifically, we first identified and extracted the visually-hungry words (vh-words) from input text via a token selector, where three different methods have been proposed, including syntax-, attention-and learning-based strategies. then, we adopted a fixed vl-ptm text encoder to generate the visually-augmented representations of these vh-words. as it has been pre-trained by visuallanguage alignment tasks on the large-scale corpus, it is capable of injecting visual semantics into the aligned text representations. finally, we transformed the visually-aligned features into visuallyaugmented features by reformulation layer based on vh-words, and inserted them into plms to enrich the visual semantics of word representations in plms. experimental results on 10 nlp tasks show that our approach can consistently improve the performance of bert, roberta, bart, and t5 at different scales, and outperform several competitive baselines significantly. besides, the visual prompts of our framework can also be used for parameter-efficient tuning, which can boost the performance of large language models, such as t5-3b.