Weakly-supervised Text Classification Based on Keyword Graph (2021)

Abstract: Weakly-supervised text classification has received much attention in recent years for it can alleviate the heavy burden of annotating massive data. Among them, keyword-driven methods are the mainstream where user-provided keywords are exploited to generate pseudo-labels for unlabeled texts. However, existing methods treat keywords independently, thus ignore the correlation among them, which should be useful if properly exploited. In this paper, we propose a novel framework called ClassKG to explore keyword-keyword correlation on keyword graph by GNN. Our framework is an iterative process. In each iteration, we first construct a keyword graph, so the task of assigning pseudo labels is transformed to annotating keyword subgraphs. To improve the annotation quality, we introduce a self-supervised task to pretrain a subgraph annotator, and then finetune it. With the pseudo labels generated by the subgraph annotator, we then train a text classifier to classify the unlabeled texts. Finally, we re-extract keywords from the classified texts. Extensive experiments on both long-text and short-text datasets show that our method substantially outperforms the existing ones.

## introduction text classification is one of the most fundamental tasks in natural language processing (nlp). in real-world scenarios, labeling massive texts is timeconsuming and expensive, especially in some specific areas that need domain experts to participate. weakly-supervised text classification (wtc) has received much attention in recent years because it can substantially reduce the workload of annotating massive data. among the existing methods, the mainstream form is keyword-driven @xcite ; kuipers our method exploits the correlation among keywords by gnn over a keywordgraph, and converts the task of assigning pseudo-labels for unlabeled texts to annotating subgraphs, which leads to much better performance. @xcite @xcite @xcite @xcite , where the users need only to provide some keywords for each class. such class-relevant keywords are then used to generate pseudo-labels for unlabeled texts. keyword-driven methods usually follow an iterative process: generating pseudo-labels using keywords, building a text classifier, and updating the keywords or self-training the classifier. among them, the most critical step is generating pseudo-labels. most existing methods generate pseudo-labels by counting keywords, with which the pseudo-label of a text is determined by the category having the most keywords in the text. however, one major drawback of these existing methods is that they treat keywords independently, thus ignore their correlation. actually, such correlation is important for the wtc task if properly exploited, as a keyword may implies different categories when it co-occurs in texts with other different keywords. as shown in fig. 1 , suppose the users provide keywords "windows" and "microsoft" for class "computer" and "car" for class "traffic". when "windows" and "microsoft" appear in text a, the "windows" means operating system, and text a should be given a pseudo-label of "computer". however, when "windows" meets "car" in text b, the "windows" means the windows of a car and text b should be given a pseudo-label "traffic". with previous simple keyword counting, text a can get a correct pseudo-label, but text b cannot. therefore, treating keywords independently is problematic. in this paper, we solve the above problem with a novel iterative framework called classkg (the abbreviation of classification with keyword graph) where the keyword-keyword relationships are exploited by gnn on keyword graph. in our framework, the task of assigning pseudo-labels to texts using keywords is transformed into annotating keyword subgraphs. specifically, we first construct a keyword graph g with all provided keywords as nodes and each keyword node updates itself via its neighbors. with g, any unlabeled text t corresponds to a subgraph g t of g, and assigning a pseudo-label to t is converted to annotating subgraph g t . to accurately annotate subgraphs, we adopt a paradigm of first self-supervised training and then finetuning. the keyword information is propagated and incorporated contextually during keyword interaction. we design a self-supervised pretext task that is relevant to the downstream task, with which the finetuning procedure is able to generate more accurate pseudo-labels for unlabeled texts. texts that contains no keywords are ignored. with the pseudo-labels, we train a text classifier to classify all the unlabeled texts. and based on the classification results, we re-extract the keywords, which are used in the next iteration. furthermore, we notice that some existing methods employ simple tf-idf alike schemes for reextracting keywords, which makes the extracted keywords have low coverage and discrimination over the unlabeled texts. therefore, we develop an improved keyword extraction algorithm that can extract more discriminative keywords to cover more unlabeled texts, with which more accurate pseudolabels can be inferred. in summary, our contributions are as follows: ## related work here we review the related works, including weakly-supervised text classification and selfsupervised learning. ## conclusion in this work, we propose a novel method for weakly-supervised text classification by exploiting the correlation among keywords. our method follows an iterative paradigm. in each iteration, we first build a keyword graph and the task of assigning pseudo-labels is transformed into annotating keyword subgraphs. to accurately annotate subgraphs, we first train the subgraph annotator with a designed pretext task and then finetune it. the trained subgraph annotator is used to generate pseudo labels, with which we train a text classifier. finally, we re-extract keywords from the classification results of the classifier. our experiments on both long and short text datasets show that our method outperforms the existing ones. as for future work, we will focus on improving the proposed method by new mechanisms and network structures. acknowledgement. this work was supported by alibaba group through alibaba innovative research program.