Text Classification Using Label Names Only: A Language Model Self-Training Approach (2020)

Abstract: Current text classification methods typically require a good number of human-labeled documents as training data, which can be costly and difficult to obtain in real applications. Humans can perform classification without seeing any labeled examples but only based on a small set of words describing the categories to be classified. In this paper, we explore the potential of only using the label name of each class to train classification models on unlabeled data, without using any labeled documents. We use pre-trained neural language models both as general linguistic knowledge sources for category understanding and as representation learning models for document classification. Our method (1) associates semantically related words with the label names, (2) finds category-indicative words and trains the model to predict their implied categories, and (3) generalizes the model via self-training. We show that our model achieves around 90% accuracy on four benchmark datasets including topic and sentiment classification without using any labeled documents but learning from unlabeled data supervised by at most 3 words (1 in most cases) per class as the label name.

## method in this section, we introduce lotclass with bert @xcite as our backbone model, but our method can be easily adapted to other pretrained neural lms. ## discussions the potential of weakly-supervised classification has not been fully explored. for the simplicity and clarity of our method, (1) we only use the bert-base-uncased model rather than more advanced and recent lms; (2) we use at most 3 words per class as label names; (3) we refrain from using other dependencies like back translation systems for augmentation. we believe that the performance will become better with the upgrade of the model, the enrichment in inputs and the usage of data augmentation techniques. applicability of weak supervision in other nlp tasks. many other nlp problems can be formulated as classification tasks such as named entity recognition and aspect-based sentiment analysis @xcite . sometimes a label name could be too generic to interpret (e.g., "person", "time", etc). to apply similar methods as introduced in this paper to these scenarios, one may consider instantiating the label names with more concrete example terms like specific person names. ## conclusions in this paper, we propose the lotclass model built upon pre-trained neural lms for text classi-fication with label names as the only supervision in three steps: category understanding via label name replacement, word-level classification via masked category prediction, and self-training on unlabeled corpus for generalization. the effectiveness of lotclass is validated on four benchmark datasets. we show that label names is an effective supervision type for text classification but has been largely overlooked by the mainstreams of literature. we also point out several directions for future work by generalizing our methods to other tasks or combining with other techniques.