Text Grafting: Near-Distribution Weak Supervision for Minority Classes in Text Classification (2024)

Abstract: For extremely weak-supervised text classification, pioneer research generates pseudo labels by mining texts similar to the class names from the raw corpus, which may end up with very limited or even no samples for the minority classes. Recent works have started to generate the relevant texts by prompting LLMs using the class names or definitions; however, there is a high risk that LLMs cannot generate in-distribution (i.e., similar to the corpus where the text classifier will be applied) data, leading to ungeneralizable classifiers. In this paper, we combine the advantages of these two approaches and propose to bridge the gap via a novel framework, text grafting , which aims to obtain clean and near-distribution weak supervision for minority classes. Specifically, we first use LLM-based logits to mine masked templates from the raw corpus, which have a high potential for data synthesis into the target minority class. Then, the templates are filled by state-of-the-art LLMs to synthesize near-distribution texts falling into minority classes. Text grafting shows significant improvement over direct mining or synthesis on minority classes. We also use analysis and case studies to comprehend the property of text grafting.

## introduction recent research has made rapid progress on extremely weak-supervised text classification (xws-tc) @xcite , limiting the supervision to a brief natural-language description without any annotated samples. for example, text mining-based xws-tc @xcite @xcite @xcite takes only class names or seed words from humans and discovers potential in-class texts following designated heuristics. minority classes are arguably the most challenging part of xws-tc. the class distribution stage 1: potential text mining gather texts with beneficial components to appear in the grafted results. stage 2: template creation mask the components that do not contribute to grafting. text mining text raw noisy yes data synthesis none generated clean hardly text grafting (ours) template grafted clean mostly in real-world datasets is often a long-tailed distribution @xcite , with a non-trivial number of minority classes. these minority classes have a very small number of documents in the raw corpus, therefore, it is difficult to locate the right documents by mining-based methods, leading to noisy pseudo-labels. under extreme circumstances, the mining-based methods may end up with no sample for minority classes. a potential way to address this issue is data synthesis-based xws-tc @xcite , which hopes to generate inclass texts by prompting large language models (llm) @xcite @xcite @xcite with class names or defini-tions. however, such synthesized texts may follow a distribution different from the corpus where the text classifier will be later applied @xcite , which makes the learned text classifier outof-distribution, leading to poor performance. this paper combines the advantages of miningbased and synthesis-based frameworks to propose a new framework, text grafting, which aims to obtain clean and near-distribution weak supervision for minority classes. as specified in figure 1 , text grafting incorporates three stages: (1) potential text mining gathers raw texts with beneficial components to synthesize in-class texts for the target minority class. (2) template creation forms templates by masking the components that do not contribute to the in-class text synthesis. (3) template filling synthesizes in-class texts by filling in the blanks. to identify the words not contributing to the classification, we borrow the marginalization idea from llm reasoning @xcite . we get the probability logit of each word in the raw text by instructing llms (relatively small, specifically gemma @xcite ) to generate with or without the in-class as a requirement. the difference between the two logits represents the potential of each word to appear in the grafted text. as only words with high potential will be left, we use the average potential of top-k% words to represent the text potential score. the bottom-(100 -k)% words will be masked to form the template for data synthesis. we rank the templates by their potential scores and select top-t % templates for the last template-filling stage. finally, these selected templates are filled by prompting a state-of-the-art llm, @xcite . we compare the three mentioned frameworks on various raw corpora to classify different minority classes. the experiment results show text grafting can outperform state-of-the-art text mining and dataset synthesis methods. the ablation study verifies that all stages and the intermediate template contribute to the success of our proposed text grafting. the mask-and-filling scenario also shows its advantage over simple in-context generation, since it forces the llm to incorporate components from the raw texts. we also involve an extreme situation where the target class does not appear in the raw corpus completely. remarkably, text grafting shows its robustness to this extreme situation, indicating its applicability does not require the target class to appear in the raw corpus. this enables text grafting to work on a very small corpus which boosts efficiency. furthermore, we analyze and discuss the property of text grafting. we apply principal component analysis to visualize that the drafted texts are indeed near in-distribution. we also find the grafted texts are near-distribution enough that we do not need to synthesize negative samples as in traditional data synthesis, which reduces the cost. we also conduct a comprehensive hyperparameter analysis of our method. interestingly, we found that the mask ratio is searched to be better set to a high value like 0.75 and the mined template number can be as small as 200. these case studies explore the advantages of text grafting in distribution approximation and its failure when the raw texts are near the distribution of llm generation. we summarize our contributions as follows, • we propose a novel xws-tc framework for minority classes, text grafting, combining the indistribution advantage of text mining and the inclass advantage of data synthesis. text mining is a fundamentak task @xcite for natural language processing. in xws-tc, the text miner follows high-level rules from humans to annotate raw texts, which are used to train the text classifier. a mainstream rule is whether a seed word appears in the raw text @xcite @xcite , categorized as seed methods. another mining way is to prompt language models for logits that reflect the probability of texts falling in classes @xcite , which can be calibrated by several techniques @xcite @xcite . the strong performance of existing text mining methods is highly dependent on the precision of the class-indicative rules @xcite , which is hard to maintain for minority classes. data synthesis @xcite addresses the precision degradation in text mining by directly prompting llms with the label names to generate in-class texts @xcite . with the powerful generative ability of llms, the synthesized texts are generally clean (in-class) for training strong classifiers. however, synthesized texts hold llm-specific patterns, discovered by llm-generated text detectors @xcite . this pattern is hard to be eliminated even with in-context learning @xcite . thus, synthesized texts are generally out-of-domain and consequently fine-tune a weaker classifier on the test set. minority classes widely appear in classification datasets as a result of long-tailed distribution @xcite . for minority classes with supervised annotations, techniques like re-sampling @xcite @xcite and data augmentation @xcite @xcite . however, these methods are applied to unbalanced annotations, which are unavailable under xws. counterfactual augmentation refers to generating annotated data out of the dataset or raw corpus. different from regular augmentation, counterfactual augmentation changes the reference, e.g., label flipping @xcite . counterfactual augmentation is also applied for text-to-text tasks like translation @xcite or summarization @xcite . counterfactual augmentation shares the same requirement for known reference as regular augmentation. this paper explores a counterfactual augmentation method for unannotated raw text under xws. text mining gathers in-class texts with highlevel rules g(x) that can precisely assign x to target class c. example rules include whether x contains words indicating c (seed words) @xcite or x has top confidence to be in c by prompting llms @xcite however, text miners fail in minority classes due to their low proportion in the raw corpus. by running a state-of-the-art text mining method @xcite on ag-news @xcite with class name proportion modified by sampling, we observe the mining precision drops sharply with the decrease of proportion, presented in figure 2 . another concern is the class might be too minor that even no ground truth can be mined from the raw corpus, limiting the precision to 0% no matter how intuitive the mining rule is. data synthesis does not annotate raw texts for classifier fine-tuning but directly prompts llms to generate in-class texts (x ′ ∼ llm(i c )), where i c is an instruction to write a text in class c. with the strong capability of state-of-the-art llms (openai, 2024; meta, 2024), the generated x ′ are highly confident to fall in class. another advantage of data synthesis is the ability of llms to generate negative samples @xcite . however, synthesized texts consist of patterns different from other sources @xcite , which indicates classifiers f (•) fine-tuned by synthesized texts are out-of-domain, consequently weaker in the classification task. ## case study in figure 8 , we depict workflows of text grafting in comparison with in-context generation to illustrate the strength of grafting and possible failure. strength of text grafting is the ability of stateof-the-art llms to fill in hard templates as shown in the first case. while the template is not easy to be grafted into the target "politics" class, the llm comes up with the methodology to synthesize such a text. the text is also more similar in writing style to the original text than the in-context generation, which depicts the benefit from text grafting. failure of text grafting can happen when the corpus does not have a writing style very far from the way that llms can imitate. as shown in the second case, the llm can synthesize the animal question without the intermediate template on the trec corpus @xcite , which reduces the necessity of text grafting. the xws-tc of the minority class "animal" on this corpus also shows a similar performance between data synthesis (f1 @xmath0.88) and text grafting (f1 @xmath1.46), which again emphasizes "near-distribution" to be an essential motivation to use text grafting. ## conclusion and future work we introduced text grafting, a technique to generate in-distribution texts for minority classes using llms. by mining high-potential masked templates from the raw corpus and filling them with state-ofthe-art llms, we achieve significant improvements in classifier performance on minority classes. our analysis and case studies demonstrate the effectiveness of text grafting in enhancing text synthesis for minority classes. future work will concentrate on improving the precision of template mining and the extension of text grafting to other tasks like information extraction.