Weakly Supervised Text Classification using Supervision Signals from a Language Model (2022)

Abstract: Solving text classification in a weakly supervised manner is important for real-world applications where human annotations are scarce. In this paper, we propose to query a masked language model with cloze style prompts to obtain supervision signals. We design a prompt which combines the document itself and “this article is talking about [MASK].” A masked language model can generate words for the [MASK] token. The generated words which summarize the content of a document can be utilized as supervision signals. We propose a latent variable model to learn a word distribution learner which associates generated words to pre-defined categories and a document classifier simultaneously without using any annotated data. Evaluation on three datasets, AGNews, 20Newsgroups, and UCINews, shows that our method can outperform baselines by 2%, 4%, and 3%.

## introduction text classification is a fundamental task in natural language processing (nlp) with diverse realworld applications such as identifying relevant documents of a case in legal proceedings @xcite , and classifying victim's requests (e.g., food, shelter, and medical aids) on social media platforms during earthquakes @xcite . current state-of-the-art text classification methods @xcite @xcite still need a large number of annotated data. however, in the real world, naturally annotated data are rare and human annotations are expensive. solving the text classification task without using annotated data but exploiting inexpensive supervision signals is worth investigation. in the weakly supervised setting, any annotated document is not accessible, but inexpensive supervision signals such as label surface names or keywords can be used. existing weakly supervised text classification methods @xcite @xcite first used seed keywords to retrieve more keywords, and then created pseudo labels for documents and then train a model in a "standard" supervised learning manner. in previous work, supervision signals are restricted to a small set of keywords from documents contents. recent work shows that prompts can probe knowledge from plms @xcite and the knowledge can provide supervision signals to solve different nlp tasks including relation extraction @xcite , question answering @xcite , and summarization @xcite . for example, @xcite solved the knowledge base completion task by querying an mlm with a prompt "alan turing was born in @xcite ." using prompts to generate supervision signals for text classification is worth exploring. we propose to query an mlm with a prompt which combines the document itself and "this article is talking about @xcite .", and use generated words for the @xcite token as supervision signals. for example, in figure 1 , given a prompt "the radio telescope at arecibo observatory will begin mapping the known galaxy on friday, scientists said. this article is talking about @xcite .", an mlm predicts "astronomy", "galaxies", "radio", "science", and "galaxy" for the @xcite token. these words summarize the topic of the document. hence, they can be used as supervision signals. besides generating signal words, an intuitive approach to obtain supervision signals is by extracting important words from documents. we will compare two types of supervision signals. after obtaining signal words, we need to associate these words to pre-defined categories. we propose a latent variable model (wddc) to learn a word distribution learner and a document classifier simultaneously without using any annotated data. a word distribution learner aims to learn ## related work in this section, we review the related work on querying an mlm with prompts, weakly supervised text classification, zero-shot text classification, and variational methods. querying an mlm with prompts. querying an mlm with cloze style prompts provides a new direction to solve some nlp tasks in an unsupervised manner. @xcite queried an mlm using manually designed prompts to solve a knowledge base completion task. for example, in order to complete the missing entity x in (alan turing, born in, x), they designed a prompt "alan turing was born in [mask]" to query an mlm. the generated word for the [mask] token can be directly used to complete the missing fact. by querying language models, some nlp tasks such as relation extraction @xcite , question answering @xcite , summarization @xcite could be solved in an unsupervised manner. however, not all nlp tasks can directly use generated words from an mlm in downstream tasks. some tasks such as sentiment analysis and textual entailment @xcite need more steps for inference. for example, in the sentiment analysis task, @xcite ) used annotated data to train a classifier that links generated words to pre-defined categories. our work does not require annotated data for inference. weakly supervised text classification. in the weakly supervised text classification task, any labeled documents are not allowed, but label sur-face names or limited word-level descriptions of each category can be used. dataless @xcite used explicit semantic analysis (esa) vectors @xcite to represent label name and documents. predictions are based on the label-document similarity. recently, @xcite @xcite @xcite @xcite trained neural text classifiers in an weakly supervised manner. they generated pseudo labels for documents to pre-train a neural classifier and then performed self-training on unlabeled data for model refinement. lotclass @xcite is relevant to our work because they also used pre-trained language models. they used a lm to retrieve a set of semantically correlated words for each class, and then fine-tuned the lm to predict these words. finally, they performed self-training on unlabeled data. our work is different from lot-class because we obtain supervision signals by querying an mlm with cloze style prompts and we propose a latent variable model to learn document classifier rather than using the self-training procedure. prboost @xcite is also relevant to our work because they also use prompts to generate weak labels. prboost first generated rules by using a small amount of labeled data, then asked human annotators to select high-quality rules to generate week labels. finally, they trained a new model in a self-training manner. our work is different from prboost because our method associates predicted words with labels in an unsupervised manner while prboost maps prompting based rules to labels by involving human feedback. zero-shot text classification. in zero-shot learning settings, the classes covered by training instances and the classes we aim to classify are disjoint. zero-shot learning text classification methods @xcite @xcite generalized seen classes to unseen classes by learning semantic relationships between classes and documents via embeddings or semantic knowledge sources. however, zero-shot learning still requires annotated data for the seen classes training. we cannot apply zero-shot learning methods to weakly supervised settings where no annotated document is available. ## methodology in this section, we first introduce how to obtain supervision signals from an mlm and document itself, and then we introduce a latent variable model to learn a word distribution learner and a document classifier simultaneously. ## experiments in this section, we show the empirical performance of our method on the text classification task. ## conclusion to solve the weakly supervised classification task, we propose to query a masked language model with cloze style prompts to obtain supervision signals. we design a prompt which combines the document itself and "this article is talking about @xcite ." the predictions for the "[mask]" token are considered as supervision signals because they summarize the content of documents. we propose a latent variable model (wddc) to learn word distributions given pre-defined categories and a neural document classifier simultaneously without using any annotated data. evaluation on three datasets shows that our method can outperform weakly supervised learning baselines.