Pushing the Limits of Low-Resource NER Using LLM Artificial Data Generation (2024)

Abstract: Named Entity Recognition (NER) is an important task, but to achieve great performance, it is usually necessary to collect a large amount of labeled data, incurring high costs. In this paper, we propose using open-source Large Language Models (LLM) to generate NER data with only a few labeled examples, reducing the cost of human annotations. Our proposed method is very simple and can perform well using only a few labeled data points. Experimental results on diverse low-resource NER datasets show that our proposed data generation method can significantly improve the baseline. Additionally, our method can be used to augment datasets with class-imbalance problems and consistently improves model performance on macro-F1 metrics.

## introduction named entity recognition (ner) is a natural language processing (nlp) task that classifies named entities in text, such as persons, organizations, locations, and more. ner is an important problem to solve and has a lot of use cases, such as relation extraction @xcite , question answering @xcite , and a lot of other applications @xcite @xcite . most work on ner needs access to a large number of labeled examples @xcite @xcite to perform well. using distant supervision can alleviate the problem @xcite @xcite , but it usually needs domain expertise or a complex pipeline, and the results are also sensitive to the existence of a clean validation example @xcite . even recent work on few-shot ner usually assumes the existence of a dataset on the source task, which can be used to adapt to novel * equal contribution. recently, llm have demonstrated impressive capabilities in performing various tasks @xcite @xcite and generating data for tasks like classification @xcite , but their application on data augmentation in ner task is still limited. one approach use llm to paraphrase text @xcite , but this still lacks the necessary diversity. another approach utilized entities from a predefined list to create ner data using llm @xcite . we believe that this approach has several limitations, including the need for an iterative prompt engineering and a substantial volume of entity list, which may not be practical. in this study, we tackle few-shot and lowresource ner by leveraging llm without using a source task or unlabeled data. for costeffectiveness and applicability, we utilize the gptq-quantized sota open-source model mixtral @xcite . we show that our simple method boosts performance on various low-resource datasets. notably, we observe a substantial 19-point increase in f1 score when using 1% data of the wnut and tweebank datasets. additionally, our method improves macro-f1 scores on resource-scarce, classimbalanced datasets, with a 4-point average increase on crossner. ## ner data generation using llm in this section, we introduce our method to enhance ner performance in low-data scenarios. the core concept involves generating sentences from entities randomly selected from a list, masking the sentence, subsequently using these to update the entity list. during data generation, we store all entities available in the dataset as e, with e t representing all entities of a specific type. the process begins by initially generating sentences given certain entities, as explained in section 2.1, and then diversifying the entity list by masking entities and regenerating sentences, as detailed in section 2.2. ## experiment and result we utilize distill-bert-cased (68m parameters) @xcite 1 , which employs transformer layers @xcite , as our backbone model. since bert model representation is token-wise, we pool the representation by summing all tokens belonging to one word, ensuring the output has the same length as the label. a linear head is added to classify each word in a sentence. we train the model using the adam optimizer (kingma and ba, 2014) with a learning rate of 2e-5. training consists of 300 iterations, with each iteration randomly sampling 32 data points from the dataset. the maximum number of entities sampled n is set to 9. the evaluation score used is the micro-f1 score, unless specified otherwise. to enhance the reproducibility and wider usability of our work, we employ the 4-bit version of mixtral-8x7b-instruct-v0.1 (46.7b parameters) 2 as the llm and set the temperature to 1. ## conclusion we propose a simple method for generating new data for ner tasks using only few examples. our approach is simple and leverages open-source llm, making it more affordable and cost-effective to reproduce. the method demonstrates performance improvement across various datasets and topics. additionally, we showcase the application of our ner data generation methods on classimbalanced datasets, highlighting significant performance improvements on such datasets.