C oarse2 F ine: Fine-grained Text Classification on Coarsely-grained Annotated Data (2021)

Abstract: Existing text classification methods mainly focus on a fixed label set, whereas many real-world applications require extending to new fine-grained classes as the number of samples per label increases. To accommodate such requirements, we introduce a new problem called coarse-to-fine grained classification, which aims to perform fine-grained classification on coarsely annotated data. Instead of asking for new fine-grained human annotations, we opt to leverage label surface names as the only human guidance and weave in rich pre-trained generative language models into the iterative weak supervision strategy. Specifically, we first propose a label-conditioned fine-tuning formulation to attune these generators for our task. Furthermore, we devise a regularization objective based on the coarse-fine label constraints derived from our problem setting, giving us even further improvements over the prior formulation. Our framework uses the fine-tuned generative models to sample pseudo-training data for training the classifier, and bootstraps on real unlabeled data for model refinement. Extensive experiments and case studies on two real-world datasets demonstrate superior performance over SOTA zero-shot classification baselines.

## introduction in traditional text classification problems, the label set is typically assumed to be fixed. however, in many real-world applications, new classes, especially more fine-grained ones will be introduced as the data volume increases. one commonly used method is to extend the existing label set to a label hierarchy by expanding every original coarsegrained class into a few new, fine-grained ones, and then assign a fine-grained label to each document. using the directory structure for a set of files in computer as an example (see in figure 1 ), people * jingbo shang is the corresponding author. to accommodate such requirements, in this paper, we introduce a new, important problem called coarse-to-fine grained classification, which aims to perform fine-grained classification on coarsely annotated data without any fine-grained human annotations. there has been prior research on performing text classification using extremely weak supervision, i.e., only label surface names as source of supervision. for example, x-class @xcite learns class-aligned document representations to generate pseudo-labels and lot-class @xcite assumes replacements of label surface names in a sentence are related to the classes and leverages pre-trained language models to extract those words. note that, coarseto-fine setting differs from generic zero-shot text classification in terms of having additional coarse supervision and a pre-conceived label hierarchy, though the final label set is available in either case. and also, coarse-to-fine setting is different from hierarchical classification. we have no supervision for fine-grained labels other than the label names whereas the few shot hierarchical setting can have a few samples for fine-grained labels. therefore, we want to capture the coarse-grained supervision and label hierarchy available to perform fine-grained classification. in this paper, we propose a novel framework c2f as illustrated in figure 2 . in the absence of finegrained human annotations, it uses fine-grained label surface names as weak-supervision signals and leverages pre-trained language models as data generators. similar to previous work, we first generate weak supervision from the whole corpus by assuming label surface names as their respective strong label-indicators. for two iterations, c2f fine-tunes a language model based on weak supervision and trains a classifier based on generated pseudo-training data to refine weak supervision. we observe that raw weak supervision usually has a highly-skewed label distribution, especially at the beginning, because the popularity of the label names varies. since we have no prior knowledge about the underlying label distribution, to avoid significant deviations from that distribution, we opt to draw a balanced, weakly annotated subset through a stratified sampling before any model training. we propose to fine-tune language models in a labelconditioned, hierarchy-aware manner. specifically, we inform the language models with label information by adding the label surface names at the beginning of each document. we further incorporate a regularization objective into the fine-tuning process that captures the constraints derived from the label hierarchy. facilitated by this fine-tuning process, we then generate pseudo-training data for each fine-grained label and train a classifier. next, using this fine-grained classifier's predictions over the coarsely annotated data, we select the samples with a high predicted probability for each respective fine-grained label. we conduct experiments on two real-world datasets containing both coarse and fine-grained labels. the results demonstrate the effectiveness of our framework in leveraging a label hierarchy and a rich pre-trained language model to perform fine-grained text classification with no supervision. via thorough ablation, we isolate separate benefits accrued, initially just from using the labelconditioned, fine-tuned language model in the weak supervision pipeline, and the later incremental benefit once we incorporate our proposed regularization objective into the language model fine-tuning. to the best of our knowledge, we are the first to work on the coarse-to-fine text grained classification, which aims to perform fine-grained classification on coarsely annotated data without any finegrained annotations. it is also worth mentioning that c2f is compatible with almost any generative language model and text classifier. our contributions are summarized as follows: reproducibility. we will release the code and datasets on github foot_0 . ## problem formulation the input of our problem contains: (1) a treestructured label hierarchy t with coarse-grained labels c at the first level and fine-grained labels f as their children. the m coarse-grained classes are named {c 1 , c 2 , . . . , c m }, and k finegrained classes are named as {f 1 , f 2 , . . . , f k }. all these class names are in natural language (e.g., words or phrases) and assumed to be informative; and (2) a collection of n text documents @xmath0, d 2 , . . . , d n } and their corresponding coarse-grained labels {c 1 , c 2 , . . . , c n }. we record the mapping from each fine-grained class to its corresponding coarse-grained parent class as f ↑ : @xmath1. the fine-grained classes in a coarse-grained label are represented by the coarseto-fine mapping f ↓ : @xmath2, where p(•) is the powerset operator, which generates the set of all subsets. in this problem, each coarse class maps to a non-empty subset of fine classes, and all these subsets of fine-classes taken together are mutually non-overlapping and exhaustive. we aim to build a high-quality document classifier from these inputs, assigning a fine-grained class label f @xmath3. ## our c2f framework as visualized in figure 2 , c2f aims to build a text classifier that can assign fine-grained labels to a set of coarsely-annotated documents based on only the label surface names and their hierarchical relations. in the absence of fine-grained human annotations, it uses fine-grained label surface names as weak-supervision signals and leverages a pretrained language model as data generators. following an iterative process, c2f fine-tunes a language model based on weak supervision. this fine-tuned language model is used to generate pseudo training data to train a fine-grained text classifier. based on the classifier's predictions, we select highly probable samples for each fine-grained class and repeat this process for one more iteration by replacing weak supervision with these samples. this bootstrapping increases the quality of weak supervision by eliminating the mislabeled samples and improves the performance of text classifier as we show later in our case studies. our major contributions lie in how to better incorporate the label names and their hierarchical relations into the language model and therefore generate more high-quality psuedo training data. our framework is compatible with any generative language model and we choose gpt-2 @xcite in our implementation. we feed label names to the language model through a labelconditioned formulation. we further incorporate a regularization objective into the fine-tuning process that captures the constraints derived from the label hierarchy. the key components of c2f are discussed in detail in the following sections. ## experiments in this section, we start with introducing datasets, compared methods, and experimental settings. next, we present quantitative evaluation results of c2f together with all compared methods. in the end, we show qualitative studies to analyze different aspects of our c2f framework. ## related work we review the literature about different weakly supervised text classification methods. there are three main sources of weak supervision: (1) a set of representative keywords for each class @xcite @xcite , (2) a few labeled documents @xcite @xcite , (3) label surface names @xcite @xcite . typically, weakly supervised text classification frameworks obtain pseudo-labeled data, train a classifier, and improve the classifier by bootstrapping over unlabeled data. seed-driven frameworks obtain pseudo-labeled data from userprovided seed words. when a few labeled documents are provided as weak supervision, the abovementioned pipeline similarly starts with these as pseudo-labeled data. in this paper, we focus on label surface names as the source of weak supervision. along this line, doc2cube @xcite expands label keywords from label surface names and performs multidimensional document classification by learning dimension-aware embedding; @xcite identify keywords for classes by querying replacements for class names using bert and pseudo-labels the documents by string matching with the selected keywords. @xcite proposed an adaptive representation learning label sentence hockey the rangers' two injury-riddled offense and seven-game effort to defeat the florida panthers on saturday have prompted coach alain vigneault to call on the veteran right wing to start against the capitals. his message to the rookie goalie thomas vanek was that "he was fine today" and the rangers did not have to worry about his status in a game against another talented team. although our method performs text classification using only fine-grained label surface names as supervision, we have coarse-grained annotated data and leverage it to improve fine-grained classification. there are a few methods that perform weakly supervised hierarchical classification @xcite . however, our problem statement is different from hierarchical classification. we have coarse-grained annotated data and our framework utilizes it and label hierarchy to perform fine-grained text classification. recently, @xcite introduced coarse-tofine weakly-supervised multi-label learning problem. however, they assume a few fine-grained labeled documents as supervision whereas we require only label surface names. additionally, our framework is generative in nature i.e. instead of pseudo-labeling the corpus, we generate training data and train the classifier. ## conclusions and future work through this work, we introduced the task of coarse-to-fine grained classification and laid out its significance. next, we showed the promise of incorporating pre-trained language models like gpt-2 into a weak supervision strategy which starts out with just label surface names. finally, we showed a way to attune these models for our task even better, through explicit regularization based on coarsefine label constraints which fall naturally out of our task definition. we outperform multiple sota zero-shot baselines on nyt and 20news, underscoring the utility both of incorporating pre-trained language models as well as task constraints. we believe exploring newer ways of exploiting task agnostic knowledge sources and injecting task constraints into the weakly supervised learning process are promising avenues for future work.