A Benchmark on Extremely Weakly Supervised Text Classification: Reconcile Seed Matching and Prompting Approaches (2023)

Abstract: Extremely Weakly Supervised Text Classification (XWS-TC) refers to text classification based on minimal high-level human guidance, such as a few label-indicative seed words or classification instructions. There are two mainstream approaches for XWS-TC, however, never being rigorously compared: (1) training classifiers based on pseudo-labels generated by (softly) matching seed words (Seed) and (2) prompting (and calibrating) language models using classification instruction (and raw texts) to decode label words (Prompt). This paper presents the first XWS-TC benchmark to compare the two approaches on fair grounds, where the datasets, supervisions, and hyperparameter choices are standardized across methods. Our benchmarking results suggest that (1) Both Seed and Prompt approaches are competitive and there is no clear winner; (2) Seed is empirically more tolerant than Prompt to human guidance (e.g., seed words, classification instructions, and label words) changes; (3) Seed is empirically more selective than Prompt to the pre-trained language models; (4) Recent Seed and Prompt methods have close connections and a clustering post-processing step based on raw in-domain texts is a strong performance booster to both. We hope this benchmark serves as a guideline in selecting XWS-TC methods in different scenarios and stimulate interest in developing guidance- and model-robust XWS-TC methods.

## introduction recently there has been a significant advancement in the text classification with the emergence of extremely weakly supervised text classification (xws-tc) methods @xcite @xcite 2022; * equal contribution. † corresponding author. 1 github repo at https://github.com/ zihanwangki/x-tc expand seed words prompt a generative lm calibration (optional) park and lee, 2022), which requires no humanannotated datasets. instead, these methods rely on minimal human guidance, such as the names of the classes or instructions describing the classification task. there are two main approaches to xws-tc: one based on matching seed words (seed), and the other on prompting a language model (lm) with instructions (prompt). we give a brief introduction in the following paragraphs, and a more thorough review is in section 3. seed methods for xws-tc rely on a userspecified list of seed words for each class, as well as an unlabeled in-domain corpus. these seed words are then expanded into a larger set of related words for the class through statistical methods @xcite , embedding similarity @xcite , or masked language model predictions @xcite . these related words are used to assign a pseudo-class to each text in the unlabeled corpus through some matching strategy (e.g., assign a text to a class if it contains the related words for that class). the pseudo labels are then used to train a classifier through standard fully-supervised fine-tuning. on the other hand, prompt methods for xws-tc, rely on reformulating text using an instruction template and prompting the language model to generate the likelihoods for each label in the classification task @xcite . for example, in a sentiment classification task, using an instruction template @xmath0. sentiment:, the model generating "happy" or "sad" will help classifiy the sentiment of the text. naive zero-shot prompting considers the highest likelihood label as the answer and recent improvements for more accurate likelihoods include calibration of likelihood scores @xcite @xcite and verbalizers that find more label words to better represent the class @xcite @xcite . both seed and prompt methods have demonstrated strong performance in xws-tc. however, there has been a lack of comprehensive comparison between these two approaches. this is due to the perception that the approaches are unrelated and the lack of standardization in datasets, supervision, and hyperparameter choices across methods. we are motivated to construct a benchmark that fairly evaluates the performance of xws-tc methods. the benchmark consists of 11 datasets covering four domains along with their fine-grained variants and different numbers of classes. in addition, we make an effort to use the same hyperparameters across datasets for the methods, as there should not be a development set to tune the hyperparameters in the xws setting @xcite . our benchmarking results suggest that both seed and prompt approaches are competitive, with no clear winner. seed tends to perform better when both approaches use a similar-sized pretrained model and is more robust and tolerant to changes in human guidance (such as seed words, classification instructions, and label words). on the other hand, prompt methods have the ability to handle more general types of human guidance (such as descriptions of class names, rather than specific words) and do not have a strict requirement for an unlabeled corpus. when the underlying pre-trained language model changes, prompt is more robust and scales better with the language model than seed. we also examine two specific methods from each approach, x-class @xcite and protocal @xcite , which independently proposed a post-processing approach to calibrate the class predictions through clustering on an unlabeled in-domain corpus to improve classification performance. our results show that this subroutine can be a universal booster for both seed and prompt approaches. through this benchmark, we aim to advance the study of xws-tc methods and call for the develop-ment of methods that are robust to different human guidance and language models. we firmly believe that this paper will serve as a guide for selecting the appropriate method in different scenarios and contribute to the advancement of the field. ## background extremely weak supervision in text classification refers to a few high-level human guidance as supervision. this guidance typically is in the form of seed words that describe each class, or an instruction paired with label words that define the task. there are two main approaches for xws-tc: matching seed words (seed) and prompting language models (prompt). ## benchmark in order to establish a benchmark that can accurately evaluate various xws-tc methods, it is essential to consider a range of factors: dataset choices, instructions, label words, hyperparameter control, use of pre-trained language models, metrics and ensure their consistency across all experiments. we will discuss each of these factors in detail in the following sections. ## connections between recent seed and prompt methods while prompt is introduced by the seminal gpt-3 paper (brown et al., 2020) not too long ago, seed has a longer history and can be traced back to early tf-idf retrieval methods @xcite . in recent years, seed methods and prompt methods are exploring similar ideas. seed methods have been leveraging pre-trained language models to better understand the semantics of seed words; for example, by asking the language model to fill in masks @xcite or through means of representation similarities @xcite . prompt methods have been exploring calibration and verbalizers to improve and stabilize its predictions. verbalizer includes a step of finding more label words that better represent the class, which is a similar approach used in seed. we show that a recent representative seed method x-class and two prompt methods, verbalizers and protocal have higher similarities and deeper connections in their design. this is particularly interesting as both directions have been developing independently. in figure 2 , we provide a pipeline of the methods and highlight the similarities. ## conclusions and future work in this work, we introduce a benchmark to qualitatively evaluate different seed and prompt approaches for extremely weakly supervised text classification. through the benchmark, we raise awareness of the existence of seed approaches, that are strong competitors to the more well-known zero-shot prompting (with calibrations). we also experiment on the robustness of these two approaches, and show that seed are more tolerant to the given human guidance changes, however also being more selective to the pre-trained language models. we also analyzed the connections of seed and prompt approaches through the lens of a few representative methods of the two approaches and showed that the methodologies are converging more recently. finally, we also include a study on clustering as a calibration technique that was independently proposed for both approaches , and show that it can be a good performance booster. we envision future work in two directions. the first one would be to understand the source of robustness difference and design a method that can take the best of both worlds (see section 6.4). the other would be to scale up the experiments and test if the conclusions still hold for larger pre-trained language models.