Visual Prompt Tuning for Few-Shot Text Classification (2022)

Abstract: Deploying large-scale pre-trained models in the prompt-tuning paradigm has demonstrated promising performance in few-shot learning. Particularly, vision-language pre-training models (VL-PTMs) have been intensively explored in various few-shot downstream tasks. However, most existing works only apply VL-PTMs to visual tasks like image classification, with few attempts being made on language tasks like text classification. In few-shot text classification, a feasible paradigm for deploying VL-PTMs is to align the input samples and their category names via the text encoders. However, it leads to the waste of visual information learned by the image encoders of VL-PTMs. To overcome this drawback, we propose a novel method named Visual Prompt Tuning (VPT). To our best knowledge, this method is the first attempt to deploy VL-PTM in few-shot text classification task. The main idea is to generate the image embeddings w.r.t. category names as visual prompt and then add them to the aligning process. Extensive experiments show that our VPT can achieve significant improvements under both zero-shot and few-shot settings. Importantly, our VPT even outperforms the most recent prompt-tuning methods on five public text classification datasets.

## introduction pre-training models have achieved great success across a variety of tasks in recent years. pretraining language models (plms) like bert @xcite , gpt @xcite and their variants @xcite @xcite firstly appeared as the milestones in the ai field. they brought huge boost to natural language processing (nlp) tasks, such as text classification @xcite , named entity recognition (ner) (jia * zhiwu lu is the corresponding author. @xcite , and text generation @xcite . in computer vision, large-scale pre-training models (e.g., bit @xcite and vit @xcite ) became popular as in nlp. with convolutional neural networks or transformers @xcite as the backbones, they were shown to be effective on a wide range of visual downstream tasks (e.g., image classification, object detection, and semantic segmentation). more recently, inspired by these pre-training models in nlp and computer vision, vision-language pre-training models (vl-ptms) have been intensively explored @xcite @xcite @xcite . they achieve excellent performance in cross-modal tasks like image-text retrieval, visual question answering (vqa), and image caption. besides, they also show great potential in single-modal tasks @xcite . these achievements clearly declare the power of large-scale pre-training models. with @xcite demonstrating astonishing zero-shot and few-shot ability, researchers are encouraged to explore the potential of large-scale pre-training models in fewshot learning. recently, prompt-tuning has been widely used in few-shot tasks as a paradigm for deploying pre-training models. compared with prompt-tuning, the performance of the traditional fine-tuning paradigm has apparent drawbacks when only few training samples are available @xcite . plm based prompt-tuning methods (e.g. prefix-tuning @xcite , p-tuning @xcite , adapet @xcite ) have shown their effectiveness and robustness on nlp tasks. meanwhile, vl-ptm based prompt-tuning methods like coop @xcite , clip-adapter @xcite and cpt @xcite apply vl-ptms to few-shot visual tasks including few-shot image classification and visual ground-figure 1 : zero-shot results on five public text datasets. the cross-modal model brivl @xcite is shown to outperform the comparably-sized singlemodal model roberta-large, indicating that the visual information may bring benefits to textual tasks. in this work, we propose a novel method named visual prompt tuning (vpt) for few-shot text classification. it is a prompt-tuning method designed to apply vl-ptm in few-shot text classification. to make use of the visual understanding ability of vl-ptm, we design a visual prompt generation module based on model inversion (see figure 2 ), which can obtain sound visual representations of the categories offline as visual prompts. in the classification process, we still adopt the standard prompt-tuning pipeline, using the text encoder of vl-ptm as backbone. specifically, we append learnable soft prompt to the front of category names' text embeddings. we then add visual prompts to the obtained embeddings of the corresponding categories and conduct the text classification by computing the cosine similarity scores between the input embeddings and the summed (both visual and textual) category embeddings. in addition, to make extensive evaluation, we collect five public available datasets for chinese text classification, which cover a diverse set of data domains including news, emotions, types of app, and specialized subjects. our main contributions are three-fold: (1) to the best of our knowledge, this is the first work on introducing vl-ptm into few-shot nlp tasks. particularly, the importance of vl-ptm has been successfully shown in few-shot text classification. ## methodology in this section, we give the details of the proposed vpt. the overall architecture is shown in figure 2 . our main idea is to improve the performance of text classification by utilizing visual information in help of vl-ptm. specifically, vpt deploys model inversion of vl-ptm to generate visual representations of category names and then add them to the text embeddings of the corresponding category names. we first describe the overall framework of vpt in sec. 3.1, followed by a detailed description of visual prompt generation in sec. 3.2. finally, the training objective is presented in sec. 3.3. ## conclusion we propose a novel prompt-based method termed visual prompt tuning (vpt) for deploying vl-ptm like brivl in few-shot text classification. the main component of our proposed vpt is a visual prompt generation module based on model inversion of vl-ptm. extensive experimental results on five benchmark datasets demonstrate that our proposed vpt achieves the new state-of-theart in few-shot text classification. the ablation study and visualization results further show the ef-fectiveness of our proposed vpt. in our ongoing research, we will apply our proposed vpt to other few-shot nlp tasks.