Improving Low-Resource Named Entity Recognition via Label-Aware Data Augmentation and Curriculum Denoising (2021)

Abstract: Deep neural networks have achieved state-of-the-art performances on named entity recognition(NER) with sufficient training data while they perform poorly in low-resource scenarios due to data scarcity. To solve this problem we propose a novel data augmentation method based on pre-trained language model (PLM) and curriculum learning strategy. Concretely we use the PLMto generate diverse training instances through predicting different masked words and design atask-specific curriculum learning strategy to alleviate the influence of noises. We evaluate the effectiveness of our approach on three datasets: CoNLL-2003 OntoNotes5.0 and MaScip of which the first two are simulated low-resource scenarios and the last one is a real low-resource dataset in material science domain. Experimental results show that our method consistently outperform the baseline model. Specifically our method achieves an absolute improvement of3.46% F1 score on the 1% CoNLL-2003 2.58% on the 1% OntoNotes5.0 and 0.99% on the full of MaScip.

## introduction named entity recognition (ner) is a fundamental natural language processing (nlp) task aiming to identify the names of people, places, organizations, and proper nouns in texts, which supports a wide range of downstream applications @xcite . the current state-of-the-art methods for ner rely on abundant training data. however, manual annotation is expensive, which limits the effectiveness of the model, especially in bio-medicine and material chemistry domains @xcite . many studies have investigated ner in low-resource scenarios, by transferring pre-trained language representations on self-supervised or rich-resource domains to target domains @xcite . others use the knowledge base to semi-automatically label extra data for training @xcite . nevertheless, these methods usually require huge expertise knowledge to obtain good performance. data augmentation has been proven effective to alleviate data scarcity in many nlp tasks, including machine translation @xcite ), text classification @xcite , question answering @xcite , etc. @xcite . however, most existing studies focus on sentence-level tasks, which generate sentences via word replacement, swap, and deletion @xcite or generative models @xcite . different from these sentence-level nlp tasks, ner predict entities on the token level. that is, for each token in the sentence, ner models predict a label indicating whether the token belongs to a mention and which entity type the mention has. therefore, applying transformations to tokens may also change their labels. due to this difficulty, data augmentation for ner is comparatively less studied. in this work, we propose a novel data augmentation framework for ner in low-resource scenarios, which generates examples with consistent labels and filters noises in the generated data. concretely, our approach contains two complementary components: 1) data augmentation via pre-trained bert, which figure 1 : the input format of different data augmentation methods. da, la, lda correspond to the basic method, label additional method, and label description additional method. as shown in blue, la puts the label at the beginning of the sentence, while lda uses the description of the label. to evaluate the effectiveness of our approach, we conduct experiments in both simulated low-resource scenarios and real-world low-resource scenarios. in the former scenarios, we use two standard ner datasets: @xcite and ontonotes5.0 @xcite , which are randomly sampled to simulate a low-resource scenarios. in the latter scenarios, we use a dataset from the material science domain: mascip @xcite . the results show that our method obtains a significant performance improvement over baseline model (i.e., bi-lstm-crf for ner) our contributions are summarized as follows: experimental results demonstrate the effectiveness of our methods in low-resource scenarios. moreover, our methods can be easily applied to other token-level tasks. this method greatly reduce the requirements on the quality of the dictionaries. @xcite propose a revised fuzzy crf layer to handle tokens with multiple possible labels and a neural model c c l 2 0 2 1 autoner with a new tie or break scheme. @xcite propose domain adaptive finetuning with unlabeled data to reduce the discrepancy between different domains. all of these methods focus on existing resources and do not consider using synthetic data for data augmentation. several works have studied using data augmentation for ner. @xcite train a weak tagger to annotate unlabeled data through weak supervision. @xcite summarize the sentence-level and sentence-pair level data augmentation methods on the nlp tasks and apply some of them to the ner task, including the token replacement, synonym replacement, mention replacement, and shuffle within segments. synonym replacement often relies on external knowledge, e.g. wordnet @xcite , which is a manually designed dictionary that may have low coverage (or not available) for low-resource languages. @xcite propose an augmentation method with language models trained on the linearized labeled sentences. different from the above methods, our approach uses pre-trained bert to predict the masked words, which contains rich contextual information. then the masked words are replaced with the predicted words to generate new synthetic sentences, and the original label sequence remains unchanged. ## conclusion in this paper, we propose a novel framework to generate high-quality synthetic data for low-resource ner. we use pre-trained bert to fully integrate contextual information to generate diverse synthetic sentences, and leverage curriculum learning to denoise synthetic sentences for obtaining higher quality augmented data. our framework shows superior performance in both simulated low-resource scenarios and real-world low-resource scenarios. in the future, we will explore the performance of our framework when customizing label descriptions and on other token-level nlp tasks.