Contrastive Bootstrapping for Label Refinement (2023)

Abstract: Traditional text classification typically categorizes texts into pre-defined coarse-grained classes, from which the produced models cannot handle the real-world scenario where finer categories emerge periodically for accurate services. In this work, we investigate the setting where fine-grained classification is done only using the annotation of coarse-grained categories and the coarse-to-fine mapping. We propose a lightweight contrastive clustering-based bootstrapping method to iteratively refine the labels of passages. During clustering, it pulls away negative passage-prototype pairs under the guidance of the mapping from both global and local perspectives. Experiments on NYT and 20News show that our method outperforms the state-of-the-art methods by a large margin.

## introduction traditional text classification often categorize into a set of coarse-grained classes, which falls short in real-world scenarios where finer categories emerge. to this end, coarse-to-fine text classification is introduced @xcite , which performs fine-grained classification given only annotation of coarse-grained categories and the coarse-to-fine mapping. then, it finetunes a pre-trained language model for each coarse prototype. 2 however, this two-step method could be sub-optimal. for example, it is vulnerable to the noise which is propagated and accumulated through the pipeline. besides, it requires finetuning and saving a pre-trained language model for each coarse prototype which is heavyweight. to this end, we propose a lightweight bootstrapping method based on contrastive clustering to iteratively refine the labels of passages. 3 to be more specific, the method starts with an epoch of warmup on the weakly-labeled dataset. during warm-up, it pulls away negative passage-prototype pairs under the guidance of the mapping from both global and local perspectives, i.e., coarse inter-cluster and fine inter-cluster perspectives. after the warm-up, the distances between clusters are not significant which causes misclassification. instead of continuing training on the weakly-labeled dataset which might greatly increase the noise (figure 1(b) ), we perform a bootstrapping process which finetunes the model on the selected dataset and updates the selected dataset by the finetuned model alternately. to mitigate the noise, we propose a selection strategy to identify high-quality pairs in terms of similarity and distinction. to further boost our method, we adopt a modified similarity metric from @xcite and use the gloss knowledge to augment the prototype representation. as shown in (figure 1(c )), the resulting clusters are well separated with less noise. our contributions are summarized as follows: ## proposed method this section describes the technical details of the proposed method, starting with the task description. ## related work previous works in weakly supervised text classification have explored different kinds of weak supervision. (1) a set of related keywords. @xcite augment and disambiguate the initial seed words with contextualized and highly labelindicative keywords. @xcite identify keywords for classes by querying replacements for class names using bert and pseudo-labels the documents by heuristics with the selected keywords. contrastive learning @xcite @xcite aims at learning representations by contrasting the positive pairs and negative pairs. in nlp, existing works can be primarily categorized into two distinct streams. unsupervised contrastive learning seeks to contrast grouped or perturbed instances to generate more robust representation of unlabeled textual data @xcite @xcite . on the contrary, supervised contrastive learning @xcite @xcite is label-aware and seeks to create representations for differently labeled data with more discrepancy. our work has shown that supervised contrastive learning incorporating label names, with minimal external knowledge, improves the model's performance in label refinement. ## conclusion in this paper, we study the task of coarse-to-fine text classification. we propose a novel contrastive clustering-based bootstrapping method to refine the label in an iterative manner. experiments on two real-world datasets for coarse-to-fine text classification verify the effectiveness of our method. future work could consider extending this method to other fine-grained decision-making tasks that could potentially benefit from coarse-grained labels, such as various kinds of lexical semantic typing tasks @xcite . another meaningful direction is to consider incorporating other partial-label learning techniques @xcite that are relevant to coarse-to-fine prediction tasks.