Prompt-based Zero-shot Text Classification with Conceptual Knowledge (2023)

Abstract: In recent years, pre-trained language models have garnered significant attention due to their effectiveness, which stems from the rich knowledge acquired during pre-training. To mitigate the inconsistency issues between pre-training tasks and downstream tasks and to facilitate the resolution of language-related issues, prompt-based approaches have been introduced, which are particularly useful in low-resource scenarios. However, existing approaches mostly rely on verbalizers to translate the predicted vocabulary to task-specific labels. The major limitations of this approach are the ignorance of potentially relevant domain-specific words and being biased by the pre-training data. To address these limitations, we propose a framework that incorporates conceptual knowledge for text classification in the extreme zero-shot setting. The framework includes prompt-based keyword extraction, weight assignment to each prompt keyword, and final representation estimation in the knowledge graph embedding space. We evaluated the method on four widely-used datasets for sentiment analysis and topic detection, demonstrating that it consistently outperforms recently-developed prompt-based approaches in the same experimental settings.

## introduction numerous studies have achieved great success in applying supervised natural language processing (nlp) techniques to address a plethora of nlp applications, including text classification @xcite , natural language inference @xcite and neural machine translation @xcite . however, achieving high accuracy with deep learning models for textual data analysis necessarily requires a large amount of manually annotated samples, which is both time-consuming and labour-intensive. to address the issues in low-resource settings, considerable attention has been paid to the pretrained language models (plms), such as gpt-3 <@xmath0. @xcite , bert @xcite , and roberta @xcite , due to their superior performances on knowledge transfer. the model pre-training stage typically involves language modelling tasks, i.e., word prediction based on the context of the input. extensive investigations, e.g., knowledge probing, on plms show that they have a certain capacity to store both linguistic and relational knowledge from large-scale corpora of general domain data @xcite . in recent years, the paradigm of nlp has been shifted from "pre-train and fine-tune" to "pre-train and prompt" @xcite , to fully exploit these plms in a gradient-free manner and effectively mitigate the gap between pre-training tasks and downstream tasks for the extreme zero-shot scenario @xcite . specifically, in the prompt-based approaches @xcite @xcite , each sample in nlp tasks can be wrapped into cloze-style questions with their corresponding templates, prompting the plms to generate the targeted output to solve the problem. for example, in a binary sentiment analysis task (shown in figure 1 ), the text "no apparent joy" is transformed to the prompt-augmented input "no apparent joy. it @xmath1.", where @xmath2. this text will then be labelled as positive or negative according to the predicted words. most existing works utilize a verbalizer to provide the translations from the predicted vocabulary to the label space in a specific task @xcite . however, these approaches are subject to two significant limitations: (i) by only considering a limited set of pre-defined label words filled in the masked position, some potentially relevant or useful words in the certain domain could be ignored, hindering the model's capacity to generalize; and (ii) the pretraining data of plms may contain biases that are reflected in the model's predictions on downstream tasks @xcite . some works propose calibration strategies to adjust the distribution of prior probabilities @xcite , which requires access to a large amount of data in specific datasets for true estimation. in this work, we propose a framework to perform prompt-based zero-shot text classification with conceptual knowledge and overcome the above limitations. the proposed framework includes promptbased keyword extraction, weight assignment to each keyword in the meaningful semantic space, and final representation estimation. specifically, in the weight assignment component, by leveraging the contextual relationships captured by simcse @xcite , a powerful contrastive learning model, we refine the probabilities of each keyword being filled in the masked position from the language prompt to mitigate the bias. additionally, in the final representation, we integrate structured factual data provided by the knowledge graphs (kgs) to include a wider range of semantic relationships between entities in a given domain. by combining their strengths, the proposed framework enables more informed predictions and a richer understanding of the underlying domain. in the experiment, we strictly follow the "label-fully-unseen" setting proposed by @xcite for evaluation. we employ four widely-used text classification datasets and compare the proposed framework with several recently-developed prompt-based approaches under the same experimental settings. the result indicates that our proposed framework brings significant improvement to the model performance. ## related works language prompt has been introduced to elicit knowledge from plms to solve different nlp tasks, which was inspired by a series of works related to prompt-based approaches, including gpt-3 @xcite and pet @xcite . however, one issue under the zero-shot set-ting identified by @xcite is the lack of domain adaptation. they performed prompt-aware continual pre-training based on adaptively retrieved data for better performance on text classification tasks. to widen the coverage of label words, @xcite incorporated external knowledge bases for the verbalizer construction, which greatly improved the stability. the above-mentioned works used hand-crafted prompt templates, particularly designed by humans for various nlp tasks. while they are carefully constructed, the process requires a considerable amount of human effort. several automatic prompting techniques were introduced to automatically select a prompt based on the input provided to the plms. @xcite suggested to employ a pre-trained text-to-text transformer, t5 @xcite , for candidate template generation. the best language prompt can be derived after the evaluation of each candidate template. @xcite proposed a gradient-based approach to search for a set of impactful tokens as the prompts that can cause significant changes in the model's output. nevertheless, the quality of the automatically generated prompt usually cannot be guaranteed, and this approach lacks sufficient interpretability. besides discrete prompts, research such as (li and liang, 2021) and @xcite presented continuous prompts as prefixes to the input, which are continuous vectors that can be learned based on patterns and structures from the data. this approach avoids the hassle of explicit prompt design while it introduces a large number of new parameters to be optimized. ## methodology we propose a prompt-based approach to tackle the zero-shot text classification problem. the overall framework is shown in figure 2 . we first extract the keywords to summarize the input text with the prompt-based approach. then, we assign weights to these keywords based on their semantic relevance to the overall meaning of the text. the weighted embeddings of all extracted keywords in the knowledge graph (kg) embedding space are aggregated to produce the final representation of the input text. finally, we determine if the text is related to a label in the kg according to their cosine similarity. in the following subsections, we describe the task definition in the extreme zero-shot setting, prompt-based keyword extraction, weight assignment and final representation estimation in the constructed kg embedding space. ## conclusion we proposed a prompt-based framework to tackle the text classification problem in the extreme zeroshot setting. we exploited the plm to extract keywords from input, assigned their weights in the meaningful semantic space and incorporated conceptual knowledge from conceptnet to estimate the final representation. evaluation results showed 4 https://lvdmaaten.github.io/tsne/ that the method reduced the biases of the mlm head and generalized well on two topic detection and two sentiment analysis datasets, outperforming several recently-developed prompt-based approaches.