Boosting Prompt-Based Self-Training With Mapping-Free Automatic Verbalizer for Multi-Class Classification (2023)

Abstract: Recently, prompt-based fine-tuning has garnered considerable interest as a core technique for few-shot text classification task. This approach reformulates the fine-tuning objective to align with the Masked Language Modeling (MLM) objective. Leveraging unlabeled data, prompt-based self-training has shown greater effectiveness in binary and three-class classification. However, prompt-based self-training for multi-class classification has not been adequately investigated, despite its significant applicability to real-world scenarios. Moreover, extending current methods to multi-class classification suffers from the verbalizer that extracts the predicted value of manually pre-defined single label word for each class from MLM predictions. Consequently, we introduce a novel, efficient verbalizer structure, named Mapping-free Automatic Verbalizer (MAV). Comprising two fully connected layers, MAV serves as a trainable verbalizer that automatically extracts the requisite word features for classification by capitalizing on all available information from MLM predictions. Experimental results on five multi-class classification datasets indicate MAV’s superior self-training efficacy.

## introduction the language model has demonstrated impressive results in numerous practical applications by undergoing extensive pre-training using objectives such as masked language modeling and autoregressive language modeling @xcite @xcite . recently, the prompting approach, as proposed by @xcite and @xcite , has played a critical role in addressing the fewshot scenario in various natural language processing tasks @xcite @xcite . @xcite and @xcite introduced a novel prompt-based fine-tuning methodology, known as pet foot_1 and lm-bff foot_2 , which applied prompting to encoder-only models such as bert @xcite and roberta @xcite . their research demonstrated the efficacy of prompt-based fine-tuning in few-shot text classification tasks, as it significantly outperformed standard fine-tuning methods. to address the scarcity of labeled data, semisupervised learning approaches have also been extensively researched by leveraging unlabeled data, which often contain sufficient contextual information @xcite @xcite . recent studies @xcite @xcite have experimentally demonstrated the efficacy of combining prompt-based fine-tuning with self-training. however, existing research on prompt-based self-training has focused on addressing binary classification (e.g., sentiment analysis) or three-class classification (e.g., natural language inference) tasks @xcite . although leveraging unlabeled data has been a crucial aspect of multi-class classification @xcite @xcite , prompt-based self-training approaches for multi-class classification remain under-explored. the challenge in applying conventional promptbased self-training approaches to multi-class classification arises from the verbalizer. in promptbased fine-tuning, a given input sentence x (e.g., "i am happy.") is wrapped with a pre-defined natural language sequence including [mask] token, called template (e.g., "x: it is @xcite ."). subsequently, verbalizer, a mapping from a label word (e.g., "good") to a specific class (e.g., positive), extracts predicted value of the label word from the mlm prediction to generate the final predictive probability distribution. that is, the probability that the label word is predicted at the masked position is regarded as the prediction probability of the class. the main problem is that most of self-training studies construct verbalizers using manually selected single label words. creating such a manual verbalizer necessitates domain knowledge @xcite , which causes the high cost associated with manually selecting label words for numerous classes under the constraint of choosing a label word as a single token. furthermore, extracting only a small amount of limited information corresponding to label words from highdimensional mlm predictions can result in information loss @xcite , leading to considerable performance disparities depending on the selected label word @xcite . to address the limitations of current verbalizers, we propose a novel mapping-free automatic verbalizer (mav) that consists of two fully connected layers (fcls) to automatically extract the necessary vocabulary information for classification by leveraging entire mlm predictions without specifying explicit label words. the proposed methodology mav has the following advantages: (1) it eliminates the need for manual selection of label words. (2) it leverages the entire mlm predictions without information loss. (3) it circumvents the cost issue associated with searching for optimal label words. (4) it can be applied to any multi-class datasets, irrespective of the number of classes. in this study, we conducted few-shot text classification on five multi-class datasets, achieving an average performance improvement of 12.8% over existing self-training methodology. quantitative metrics demonstrated that the proposed mav benefits most from self-training. further analysis revealed that mav is proficient in extracting vocabulary features representative of each class, thereby enhancing the efficacy of prompt-based learning without manual manipulation of the verbalizer. ## conclusion in this research, we proposed a simple and effective verbalizing method, mapping-free automatic verbalizer, to enhance prompt-based self-training performance on various multi-class classification tasks. unlike previous verbalizers which take limited information from mlm prediction and require label engineering, mav is a trainable verbalizer consisting of two fcls that extracts key vocabulary features from mlm predictions and maps them to label space. this approach has the advantage of promoting the efficiency of prompt-based learning without manual verbalizer manipulation or the search for optimal label words. experimental findings not only validate the effectiveness of mav but also highlight its self-training efficacy across all datasets, as supported by quantitative metrics and qualitative analysis.