LLM a AA : Making Large Language Models as Active Annotators (2023)

Abstract: Prevalent supervised learning methods in natural language processing (NLP) are notoriously data-hungry, which demand large amounts of high-quality annotated data. In practice, acquiring such data is a costly endeavor. Recently, the superior few-shot performance of large language models (LLMs) has propelled the development of dataset generation, where the training data are solely synthesized from LLMs. However, such an approach usually suffers from low-quality issues, and requires orders of magnitude more labeled data to achieve satisfactory performance. To fully exploit the potential of LLMs and make use of massive unlabeled data, we propose LLMaAA, which takes LLMs as annotators and puts them into an active learning loop to determine what to annotate efficiently. To learn robustly with pseudo labels, we optimize both the annotation and training processes: (1) we draw k -NN examples from a small demonstration pool as in-context examples, and (2) we adopt the example reweighting technique to assign training samples with learnable weights. Compared with previous approaches, LLMaAA features both efficiency and reliability. We conduct experiments and analysis on two classic NLP tasks, named entity recognition and relation extraction. With LLMaAA, task-specific models trained from LLM-generated labels can outperform the teacher within only hundreds of annotated examples, which is much more cost-effective than other baselines.

## introduction large language models (llms) have exhibited remarkable few-shot performance in a wide range of tasks, with only a few demonstrations and welldesigned prompts @xcite @xcite . however, with rapid ad-(c) llmaaa: active llm annotation as supervision. figure 1 : comparing llmaaa with other frameworks. we actively acquire annotations from llm for efficiency, requiring little human effort. on the other hand, training task-specific models (tams) for nlp tasks necessitates extensive amounts of labeled data. due to the superior generative capacity of llms, some researchers attempt to synthesize training data with text generation @xcite , as depicted in figure 1 . however, the generated text usually struggles with low-quality issues and may exhibit domain shifts with test data @xcite . to exploit the abundant unlabeled corpus, an alternative is to employ llms as annotators, which generate labels in a zero-shot or few-shot manner. while this approach seems promising, it is important to acknowledge that llm-generated labels inevitably contain noise, especially when applied to challenging tasks and domain-specific data @xcite . besides, larger models come with heavier expenses, and it is also crucial to reduce the annotation cost when the budget is restricted. to enhance the reliability (i.e. accuracy) of tams' performance as well as to ensure the data efficiency in annotation cost, we propose ll-maaa, an innovative framework that integrates active learning into the llm annotation process, i.e., making llms as active annotators. by exploring different active acquisition strategies, ll-maaa enables the llm to annotate more informative instances that benefit model performance more. to train tams reliably, we optimize both the annotation and training processes within llmaaa framework. firstly, we employ prompt engineering techniques to enhance llms' performance by (1) selecting k-nn samples from a demonstration pool as in-context examples, and (2) building finelevel descriptions aligned with natural language for unnatural labels (e.g., category labels in the re task). the valuable contextual information helps improve the quality of llm annotations substantially. during training, we adopt the automatic reweighting technique @xcite to assign learnable weights to the silver foot_0 training samples. this strategy allows the model to prioritize more informative and representative samples while simultaneously reducing the impact of noisy annotations. we evaluate llmaaa on two practical nlp tasks: named entity recognition (ner) and relation extraction @xcite . experiments show that: (1) with small-scale gold data (~100 examples) serving for demonstration and validation, the trained tams can outperform their teacher llms within hundreds of silver samples via llmaaa; (2) our approach is significantly more data efficient compared to prevalent data generation methods, which usually require large-scale synthetic training data (size varying from 10k to 200k, @xcite . these results confirm the potential of llmaaa as a practical and cost-efficient solution to make llms as good annotators. the tams created through our framework offer advantages in terms of task-specific performance, data privacy, and inference costs, which release the capacity of llms for real-world productivity. we summarize our contributions as follows: dataset synthesis supervised learning methods in nlp are often limited by high-quality annotated data. to address the bottleneck, researchers have explored techniques to synthesize training … input: soccer -late goals give japan win over syria. output: [{"span": "japan", "type": "loc"}, {"span": "syria", "type": "loc"}] ## llm as active annotator to exploit llms' superior few-shot performance and leverage abundant unlabeled data, we attempt to take llm as annotator and train task-specific models for inference. an ideal process should be both efficient and reliable: we want to learn tams robustly with minimal llm-generated labels. concretely, our solution is to make llms as active annotator. as shown in figure 2 , ll-maaa comprises three key components: (1) an llm annotator that generates pseudo labels of given data, (2) an active acquisition mechanism for efficient data selection, and (3) an automatic reweighting technique to ensure robust learning with noisy labels. llmaaa iterates the three stages to gradually produce stronger tams. ## tasks we instantiate llmaaa with two tasks: named entity recognition (ner) and relation extraction ## analysis 6.1 llmaaa with different annotators to guarantee the universal effectiveness of ll-maaa, we further investigate the performance with other llm annotators, i.e. @xcite and @xcite . due to budgetary considerations, we opt to restrict our experiments to ontonotes. the precision, recall and f1 score are shown in ## conclusion in this work, we propose llmaaa, a framework that uses llms as active annotators to address the challenges of data scarcity in nlp tasks. with active learning strategies, llmaaa allows llms to label more informative samples that promote tams performance efficiently. we also optimize for reliability within the framework, which uses prompt engineering techniques and automatic reweighting to improve annotation quality and to reduce the impact of noisy labels, respectively. experiments on ner and re tasks demonstrate the effectiveness of llmaaa. the evaluation results highlight the efficiency and reliability of llmaaa. trained with just hundreds of llm-annotated samples, tams are able to outperform their teacher llms substantially. besides, llmaaa is also much more efficient compared to prevalent data generation methods, which usually require orders of magnitude more synthetic training data. these findings reveal that llmaaa offers a cost-effective, privacy-ensured, yet well-performing solution to apply llms in practical scenarios.