X -Class: Text Classification with Extremely Weak Supervision (2021)

Abstract: In this paper, we explore text classification with extremely weak supervision, i.e., only relying on the surface text of class names. This is a more challenging setting than the seed-driven weak supervision, which allows a few seed words per class. We opt to attack this problem from a representation learning perspective—ideal document representations should lead to nearly the same results between clustering and the desired classification. In particular, one can classify the same corpus differently (e.g., based on topics and locations), so document representations should be adaptive to the given class names. We propose a novel framework X-Class to realize the adaptive representations. Specifically, we first estimate class representations by incrementally adding the most similar word to each class until inconsistency arises. Following a tailored mixture of class attention mechanisms, we obtain the document representation via a weighted average of contextualized word representations. With the prior of each document assigned to its nearest class, we then cluster and align the documents to classes. Finally, we pick the most confident documents from each cluster to train a text classifier. Extensive experiments demonstrate that X-Class can rival and even outperform seed-driven weakly supervised methods on 7 benchmark datasets.

## introduction weak supervision has been recently explored in text classification to save human effort. typical forms of weak supervision include a few labeled documents per class @xcite , a few seed words per class @xcite @xcite , and other similar open-data @xcite .though much weaker than a fully annotated corpus, these forms still require non-trivial, corpusspecific knowledge from experts. for example, nominating seed words requires experts to consider their relevance to not only the desired classes but also the input corpus; to acquire a few labeled documents per class, unless the classes are balanced, one needs to sample and annotate a much larger number of documents to cover the minority class. in this paper, we focus on extremely weak supervision, i.e., only relying on the surface text of class names. this setting is much more challenging than the ones above, and can be considered as almost-unsupervised text classification. we opt to attack this problem from a representation learning perspective-ideal document representations should lead to nearly the same result between clustering and the desired classification. recent advances in contextualized representation learning using neural language models have demonstrated the capability of clustering text to domains with high accuracy @xcite . specifically, a simple average of word representations is sufficient to group documents on the same topic together. however, the same corpus could be classified using various criteria other than topics, such as locations and sentiments. as visualized in figure 1 , such class-invariant representations separate topics well but mix up locations. therefore, it is a necessity to make document representations adaptive to the user-specified class names. we propose a novel framework x-class to conduct text classification with extremely weak supervision, as illustrated in figure 2 . firstly, we estimate class representations by incrementally adding the most similar word to each class and recalculating its representation. following a tailored mixture of class attention mechanisms, we obtain the document representation via a weighted average of contextualized word representations. these representations are based on pre-trained neural language models, and they are supposed to be in the same latent space. we then adopt clustering methods (e.g., gaussian mixture models) to group the documents into k clusters, where k is the number of desired classes. the clustering method is initialized with the prior knowledge of each document assigned to its nearest class. we preserve this assignment so we can easily align the final clusters to the classes. in the end, we pick confident documents from each cluster to form a pseudo training set, based on which, we can train any document classifier. in our implementation, we use bert as both the pre-trained language model and the text classifier. compared with existing weakly supervised methods, x-class has a stronger and more consistent performance on 7 benchmark datasets, despite some of them using at least 3 seed words per class. it is also worth mentioning that x-class has a much more mild requirement on the existence of class names in the corpus, whereas existing methods rely on the variety of contexts of the class names. our contributions are summarized as follows. ## preliminaries in this section, we formally define the problem of text classification with extremely weak supervision. and then, we brief on some preliminaries about bert @xcite , attention @xcite and gaussian mixture models. problem formulation. the extremely weak supervision setting confines our input to only a set of documents d i , @xmath0, ..., n} and a list of class names c j , @xmath1, ..., k}. the class names here are expected to provide hints about the desired classification objective, considering that different criteria (e.g., topics, sentiments, and locations) could classify the same set of documents. our goal is to build a classifier to categorize a (new) document into one of the classes based on the class names. seed-driven weak supervision requires carefully designed label-indicative keywords that concisely define what a class represents. this requires human experts to understand the corpus extensively. one of our motivations is to relax this burdensome requirement. interestingly, in experiments, our proposed x-class using extremely weak supervision can offer comparable and even better performance than the seed-driven methods. bert. bert is a pre-trained masked language model with a transformer structure @xcite . it takes one or more sentences as input, breaks them up into word-pieces, and generates a contextualized representation for each word-piece. to handle long documents in bert, we apply a sliding window technique. to retrieve representations for words, we average the representations of the word's word-pieces. bert has been widely adopted in a large variety of nlp tasks as a backbone. in our work, we will utilize bert for two purposes: (1) representations for words in the documents and (2) the supervised text classifier. attention. attention mechanisms assign weights to a sequence of vectors, given a context vector @xcite . it first estimates a hidden state @xmath2, c) for each vector h j , where k is a similarity measure and c is the context vector. then, the hidden states are transformed into a distribution via a softmax function. in our work, we use attentions to assign weights to representations, which we then average them accordingly. gaussian mixture model. gaussian mixture model (gmm) is a traditional clustering algorithm @xcite . it assumes that each cluster is generated through a gaussian process. given an initialization of the cluster centers and the co-variance matrix, it iteratively optimizes the point-cluster memberships and the cluster parameters following an expectation-maximization framework. unlike k-means, it does not restrict clusters to have a perfect ball-like shape. therefore, we apply gmm to cluster our document representations. ## our x-class framework as shown in figure 2 , our x-class framework contains three modules: (1) class-oriented document representation estimation, (2) document-class alignment through clustering, and (3) text classifier training based on confident labels. ## experiments we conduct extensive experiments to show and ablate the performance of x-class. ## x-class for hierarchical classification there are two straightforward ways to extend x-class for hierarchical classification (1) x-class-end: we can give all fine-grained class names as input to x-class and conduct classification in an endto-end manner; and (2) x-class-hier: we can first give only coarse-grained class names to x-class and obtain coarse-grained predictions. then, for each coarse-grained class and its predicted documents, we further create a new x-class classifier based on the fine-grained class names. we experiment with hierarchical classification on the nyt-small dataset, which has annotations for 26 fine-grained classes. we also introduce weshclass @xcite , the hierarchical version of westclass, for comparison. lotclass is not investigated here due to its poor coarsegrained performance on this dataset. the results in ## related work we discuss related work from two angles. weakly supervised text classification. weakly supervised text classification has attracted much attention from researchers @xcite @xcite . the general pipeline is to generate a set of document-class pairs to train a supervised model above them. most previous work utilizes keywords to find such pseudo data for training, which requires an expert that understands the corpus well. in this paper, we show that it is possible to reach a similar, and often better, performance on various datasets without such guidance from experts. a recent work @xcite also studied the same topic -extremely weak supervision on text classification. it follows a similar idea of @xcite and further utilizes bert to query replacements for class names to find keywords for classes, identifying potential classes for documents via string matching. compared with lotclass, our x-class has a less strict requirement of class names being existent in the corpus, and can work well even when there is only one occurrence (refer to section 4.7). bert for topic clustering. @xcite showed that document representations obtained by an average of token representations from bert preserve domain information well. we borrow this idea to improve our document representations through clustering. our work differs from theirs in that our document representations are guided by the given class names. ## conclusions and future work we propose our method x-class for extremely weak supervision on text classification, which is to classify text with only class names as supervision. x-class leverages bert representations to generate class-oriented document presentations, which we then cluster to form document-class pairs, and in the end, fed to a supervised model to train on. we further set up benchmark datasets for this task that covers different data (news and reviews) and various class types (topics, locations, and sentiments). through extensive experiments, we show the strong performance and stability of our method. there are two directions that are possible to explore. first, focusing on the extremely weak supervision setting, we can extend to many other natural language tasks to eliminate human effort, such as named entity recognition and entity linking. second, based on the results on extremely weak supervision, we can expect an unsupervised version of text classification, where machines suggest class names and classify documents automatically.