Incubating Text Classifiers Following User Instruction with Nothing but LLM (2024)

Abstract: In this paper, we aim to generate text classification data given arbitrary class definitions (i.e., user instruction), so one can train a text classifier without any human annotation or raw corpus. Recent advances in large language models (LLMs) lead to pioneer attempts to individually generate texts for each class via prompting. In this paper, we propose Incubator, the first framework that can handle complicated and even mutually dependent classes (e.g., " TED Talk given by Educator " and " Other "). Specifically, our Incubator is a fine-tuned LLM that takes the instruction of all class definitions as input, and in each inference, it can jointly generate one sample for every class. First, we tune Incubator on the instruction-to-data mappings that we obtained from classification datasets and descriptions on Hugging Face together with in-context augmentation by GPT-4. To emphasize the uniformity and diversity in generations, we refine Incubator by fine-tuning with the cluster centers of semantic textual embeddings of the generated samples. We compare Incubator on various classification tasks with strong baselines such as direct LLM-based inference and training data generation by prompt engineering. Experiments show Incubator is able to (1) outperform previous methods on traditional benchmarks, (2) take label interdependency and user preference into consideration, and (3) enable logical text mining by incubating multiple classifiers

## introduction text classification is one of the most fundamental natural language processing (nlp) tasks and plays a vital role in many nlp systems @xcite . traditional supervised text classification fine-tunes models on expensive human annotation @xcite , limiting its usage for lower-source domains. zero-shot text classification reduces manual effort by building classifiers * corresponding author. ## our incubator framework figure 2 offers an overview of our incubator framework, including two stages, (1) instruction-tuning and (2) self-diversification. the instructiontuning stage utilizes the existing resources on the huggingface platform to learn an llm as incubator to generate training data based on user instructions. the self-diversification stage further improves the uniformity and diversity in incubator generation with an auxiliary text embedder and clustering. we now elaborate on the details of these two stages. ## experiments we conduct several experiments to evaluate the performance of our incubator. we include experiments on traditional datasets, and revised datasets with the label "other". we also evaluate the ability of incubator to handle complex personal labels and even ones with conjunctive relationships. ## conclusion and future work in summary, this paper proposes a new framework for model incubation by querying an instructiontuned llm. our model, incubator, is pre-trained on huggingface resources and icl-based augmentation. the incubator is further strengthened by a novel self-diversification technique. we show that incubator can incubate strong classifiers for traditional benchmarks and customized text mining, following instructions. we also include comprehensive analysis to explore the properties of the incubator for deeper insight and better application.