Cross-task Knowledge Transfer for Extremely Weakly Supervised Text Classification (2023)

Abstract: Text classification with extremely weak supervision (EWS) imposes stricter supervision constraints compared to regular weakly supervise classification. Absolutely no labeled training samples or hand-crafted rules specific to the evaluation data are allowed. Such restrictions limit state-of-the-art EWS classification methods to indirect weak labeling techniques that assign unnatural label uncertainty estimates. We present PLAT, a framework that creates weak labels by leveraging recent developments in zero-shot text classification. PLAT employs models trained for sub-tasks other than classification to label documents. Most importantly, PLAT refrains from assigning overly confident weak labels and improves soft-label training performance for downstream classifiers. Classifiers trained with PLAT significantly outperform those trained on weak labels generated by the previous state-of-the-art in extremely weakly supervised text classification.

## introduction we undertake the low-resource task of categorizing an unlabeled set of documents using just candidate category labels. the task is a stricter subtask of weakly supervised text classification -weak labels cannot be obtained even through utilizing a small training set or hand-crafted rules based on domain knowledge. such task formulation mimics a realistic and practical scenario where one has to classify a set of documents into a label from a pre-defined label set, using only class names. following @xcite we call this task classification with extremely weak supervision (ews). due to such additional constraints on sources of supervision, models under ews cannot trivially adapt recent state-of-the-art approaches under regular weak supervision. best-performing methods for classification under ews usually involve mining a set of category-indicative keywords from pre-trained language models @xcite @xcite @xcite . at evaluation time, each document is compared to the keyword set of each label. the weak label for a document is the label with the keyword set most similar to words that constitute the document. this divorcement of feature extraction and label assignment introduces additional noise during weak labeling, causing unnatural assignment of label confidence and oversensitivity to training size @xcite . we overcome such limitations in ews by leveraging pre-trained language models to create weak labels in an end-to-end fashion. most importantly, we eliminate the keyword-collection step in currently popular ews approaches. we employ language models trained on non-classification tasks (textual entailment, next sentence prediction, and multiple-choice question-answering) as weak labelers for classification. our research bridges weaklysupervised noisy-label training with recent developments in prompt-based low-shot text classification @xcite . our framework realizes both the robustness of noisylabel training and the label efficiency of prompting. we use publicly available, off-the-shelf models for each source task in our experiments. our contributions are as follows: ## problems with keyword-matching ews compared to simple supervised classification, ews methods mentioned in section 2.3 introduce two additional steps to the training pipeline: building category-indicative keyword sets and assigning weak labels to unlabeled documents using the built keyword sets. our investigations show that the disjoint nature of such approaches leads to unwanted side effects that degrade weak-label quality. ## plat plat draws inspiration from recent findings in zero-shot language model prompting @xcite @xcite to obtain weak labels for unlabeled documents. in the ews setting, plat leverages source models trained on a single non-classification task to solve classification tasks. plat follows the typical two-step weaklysupervised classification pipeline. in the first phase, plat creates weak labels for classification. in the second phase, a final classifier is trained using the obtained weak labels. the novelty of plat lies in improving the weak labeling phase with cross-task knowledge distillation. we aim to keep the plat framework source-model-agnostic and make the weak labelers hot-swappable, taking advantage of parallel advances in zero-shot nlp. ## experiments and results we qualitatively analyze plat in two aspects: accuracy of generated weak labels, and prediction ## conclusion we present three variants of plat, a framework for text classification under extremely weak supervision. by eliminating keyword-based weak labeling, plat sidesteps the brittle dependence on evaluation set size and hyperparameters found in previous state-of-the art methods. plat is a flexible framework that leverages prompting to generate weak labels with more natural confidence estimates. plat makes no assumptions about the training dynamics of its source models. therefore, evolutions of source models are completely orthogonal to developments in plat. the black-box treatment of its weak labeler models enables the usage of completely unsupervised weak labelers -a potential already demonstrated by plat nsp . we expect future developments in unsupervised solutions to enable even more resource-efficient classification under plat.