CLIPT ext: A New Paradigm for Zero-shot Text Classification (2023)

Abstract: While CLIP models are useful for zero-shot vision-and-language (VL) tasks or computer vision tasks, little attention has been paid to the application of CLIP for language tasks. Intuitively, CLIP model have a rich representation pre-trained with natural language supervision, in which we argue that it is useful for language tasks. Hence, this work bridge this gap by investigating a CLIP model for zero-shot text classification. Specifically, we introduce CLIPText, a novel paradigm for zero-shot text classification, which reformulates zero-shot text classification into a text-image matching problem that CLIP can be applied to. In addition, we further incorporate prompt into CLIPText (Prompt-CLIPText) to better derive knowledge from CLIP. Experimental results on seven publicly available zero-shot text classification datasets show that both CLIPText and Prompt-CLIPText attain promising performance. Besides, extensive analysis further verifies that knowledge from CLIP can benefit zero-shot text classification task. We hope this work can attract more breakthroughs on applying VL pre-trained models for language tasks.

## introduction understanding various modalities is one of the core goals of artificial intelligence. to achieve this, vision-and-language (vl) tasks such as visual question answering @xcite and image caption @xcite have emerged, aiming to test a system's ability to understand the semantics of both the visual world and natural language. recently, clip @xcite , a cross-modality model pre-trained with 400m noisy image-text pairs collected from the internet, has gained remarkable success on various vl tasks. in addition, clip shows strong zero-shot transfer capabilities on over 30 different existing computer vision (cv) datasets (e.g., image classification @xcite and object detection @xcite , 2021b)). in addition to success on cv tasks, various works begin to explore transferring knowledge of clip to other vl modality tasks. for example, @xcite demonstrate that serving clip as a strong visual encoder can benefit vl tasks in both pre-training and fine-tuning stage. @xcite prove that clip can be considered as a strong few-shot learner for vl tasks by providing a comprehensive empirical study on visual question answering and visual entailment @xcite . nevertheless, while significant recent progress has been made in applying clip to other vl and cv modality tasks, the same success has not yet been achieved in language tasks. in this work, we argue that clip was pre-trained with natural language supervision, which should be capable of helping language tasks. motivated by this, this work aims to close this gap by studying this research question: can clip benefit language task? to this end, we provide a comprehensive investigation on zero-shot text classification task, aiming to studying how to transfer clip's zero-shot ability into the language task. specifically, this work presents cliptext, a novel paradigm for zero-shot text classification. the key insight is that cliptext reformulates zero-shot text classification into a text-image matching problem, so that directly applying clip to zero-shot text classification can be achieved. as shown in fig. 1 , cliptext consists of procedure with two steps: (i) label mapping and (ii) inference. specifically, the label mapping step is used for mapping text classification label into a corresponding image, so that the text-image pairs can be constructed. then, the inference step passes the generated textimage pairs into clip model, and the label with the highest alignment score is regarded as the prediction result. in addition, inspired by recent progress in prompt methods in natural language processing @xcite @xcite @xcite , we further present prompt-cliptext by adding an additional semantic prompt word at the beginning of the text in cliptext, enabling model to better infer language knowledge from clip. compared with previous methods, our method has the following advantages. first, some prior work @xcite require additional nli dataset to further train their zero-shot classification model. in contrast, our framework is capable of making full use of the powerful zero-shot capability of the clip without any extra pre-training. second, we present a innovate perspective for zero-shot text classification, which can naturally leverage the additional vision information inferred from clip to benefit language tasks. third, our framework is model-agnostic without any specific network design, thereby it can be easily extended to other vl pre-trained model. we first evaluate our approaches on the standard zero-shot text classification benchmark @xcite . experimental results show that cliptext and prompt-cliptext achieves superior performance. in addition, we further evaluate cliptext on other four publicly available zero-shot text classification datasets to verify the generalization of cliptext and prompt-cliptext. in summary, contributions of this work are: • to our knowledge, this is the first work to investigate how to transfer zero-shot capabilities of clip into language tasks. we hope this work will spur more researchers to rethink the role of vl model for language tasks; in addition, we further propose prompt-cliptext to better infer knowledge from clip to zero-shot text classification; to promote the further research, codes are will be publicly available at https://github.com/ lightchen233/cliptext . ## model this section illustrates how to solve the zero-shot text classification task with clip (see cliptext ( §3.1) and prompt-cliptext ( §3.2)). ## related work in this sectin, we discuss the related work of zeroshot text classification task and application of clip. ## conclusion in this work, we studied how to transfer knowledge from clip into zero-shot text classification. to this end, we introduced a novel paradigm, cliptext and prompt-cliptext, for zero-shot text classification by reformulating it as a text-image matching problem. experimental results demonstrated that clip can be a good zero-shot learner for text classification. to the best of our knowledge, this is the first work to apply clip for zero-shot text classification task. we hope that our work will motivate further research on transferring knowledge from vl model (e.g., clip) to language tasks.