P rog G en: Generating Named Entity Recognition Datasets Step-by-step with Self-Reflexive Large Language Models (2024)

Abstract: Although Large Language Models (LLMs) exhibit remarkable adaptability across domains, these models often fall short in structured knowledge extraction tasks such as named entity recognition (NER). This paper explores an innovative, cost-efficient strategy to harness LLMs with modest NER capabilities for producing superior NER datasets. Our approach diverges from the basic class-conditional prompts by instructing LLMs to self-reflect on the specific domain, thereby generating domain-relevant attributes (such as category and emotions for movie reviews), which are utilized for creating attribute-rich training data. Furthermore, we preemptively generate entity terms and then develop NER context data around these entities, effectively bypassing the LLMs’ challenges with complex structures. Our experiments across both general and niche domains reveal significant performance enhancements over conventional data generation methods while being more cost-effective than existing alternatives.

## introduction recently, large language models (llms) have showcased their impressive capabilities across various nlp downstream tasks, including question answering, mathematical reasoning, and code generation ( we introduce proggen, a new method that leverages the self-reflexive capabilities of llms to progressively synthesize ner datasets. our method, in contrast to traditional approaches that directly generate synthetic ner datasets, operates through a sequential process involving multiple steps. to overcome the challenges posed by llms' limited ability to handle complex structures, we initially focus on generating entity terms, which are then employed for constructing the ner dataset. this step-by-step generation process allows for more effective dataset creation in new domains. in addition to progressively generating ner datasets, we follow several recent studies to leverage llms' ability to self-reflect (shinn et al., 2023; madaan et al., 2023) from two perspectives. first, instead of employing simple classconditional prompts, we leverage llms to generate ## method our workflow is succinctly depicted in figure 1 . at the outset, we engage llms in a sequential process to generate a variety of ner samples. unlike traditional methods that rely on domain-specific, unlabeled corpora, our approach is more pragmatic and resource-efficient. we utilize only a handful of examples to prompt llms to produce sentences along with their corresponding named entity annotations. this method is exemplified in ## experiments we assess the effectiveness of our framework via downstream model performance on generated ner datasets. ## discussion scaling diversity alone plateaus using generated dataset size as a rough proxy of sample diversity, we plot its relationship with downstream model performance in figure 3 . we compare simple prompt with the top-performing diversify y variants where we re-generate 3k valid samples. we take 6 subsets of sample sizes [300, 600, 900, 1.5k, 2.1k, 3k] where we enforce the smaller subsets are contained in the larger subsets. we then train ner models as in §4.2. we observe that substantially increasing dataset size doesn't lead to much performance gains across the board, confirming our speculation that sample diversity alone has diminishing returns ( §4.3.1). ## conclusion we present proggen, an innovative framework that leverages llms to generate diverse and accurate task-specific datasets via requirement sampling and self-reflection for low-resource ner. experimental results on 4 datasets encompassing various domains demonstrate the advantages of our data generation approach compared to llm few-shot in-context learning both in downstream model performance and cost-efficiency. our research highlights the critical role of entity diversity in improving dataset quality and demonstrates the effectiveness of incorporating llm self-correction alongside manual feedback to enhance annotation accuracy. further analysis on the generated datasets reveals persistent biases in llm entity classification and that annotating accurately is a major bottleneck to close the gap with traditional supervised learning methods.