Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM -Generated Training Labels (2024)

Abstract: Computational social science (CSS) practitioners often rely on human-labeled data to fine-tune supervised text classifiers. We assess the potential for researchers to augment or replace human-generated training data with surrogate training labels from generative large language models (LLMs). We introduce a recommended workflow and test this LLM application by replicating 14 classification tasks and measuring performance. We employ a novel corpus of English-language text classification data sets from recent CSS articles in high-impact journals. Because these data sets are stored in password-protected archives, our analyses are less prone to issues of contamination. For each task, we compare supervised classifiers fine-tuned using GPT-4 labels against classifiers fine-tuned with human annotations and against labels from GPT-4 and Mistral-7B with few-shot in-context learning. Our findings indicate that supervised classification models fine-tuned on LLM-generated labels perform comparably to models fine-tuned with labels from human annotators. Fine-tuning models using LLM-generated labels can be a fast, efficient and cost-effective method of building supervised text classifiers.

## introduction supervised text classification often relies on human-labeled text data for training and validation. computational social science (css) researchers frequently use these types of supervised models to classify large quantities of text, ranging from news articles on the internet to government documents @xcite . collecting training and validation labels generated by humans for these tasks, however, is expensive, slow, and prone to a variety of errors @xcite . to address these limitations, prior research suggests utilizing few-shot capabilities of generative large language models @xcite to annotate text data instead of human annotators @xcite . generative llms are faster and cheaper than human annotators and do not suffer from common human challenges such as limited attention span or fatigue. while this approach has its limitations and generative llms do not excel at all text annotation tasks @xcite , prior research illustrates that there are numerous circumstances where generative llms can produce high quality text-annotation labels. 1 although past work suggests llm few-shot annotation is highly effective, it may be cost prohibitive in many settings. research with text data often involves classifying millions of documents or text samples. for example, a recent css article studies a data set of 6.2 million tweets labeled on four dimensions @xcite , a task that would have cost nearly @xmath08,990 @xmath1124 @xmath20.01 per 1k input tokens and @xmath38, 990 = (1, 000 × 620, 000×0.00001)+(150×620, 000×0.00003). while this is a loose estimate, it illustrates the challenges posed by the marginal per-sample cost of automated llm annotation for large-n css research. using our approach, labeling 1,000 text samples and training a supervised classifier would cost under @xmath40.06 to @xmath515.00 per hour for six tasks per minute), which extrapolates to 360 tasks per hour. under the standard assumption of three mturk workers per task and taking a majority vote, the entire annotation time to label 1,000 tweets would have taken slightly under three hours and cost @xmath615 hourly rate, manually annotating 1,000 text samples would take 12.5 hours and cost approximately $187. table a1 shows a comparison of these costs. not only is automated annotation remarkably faster than human annotators, our procedures introduced here can cost researchers less than 10% the cost of typical alternatives. these efficiency gains are conservative in the sense that they disregard the time to find, hire, and train the annotator. 11