Investigation on Data Adaptation Techniques for Neural Named Entity Recognition (2021)

Abstract: Data processing is an important step in various natural language processing tasks. As the commonly used datasets in named entity recognition contain only a limited number of samples, it is important to obtain additional labeled data in an efficient and reliable manner. A common practice is to utilize large monolingual unlabeled corpora. Another popular technique is to create synthetic data from the original labeled data (data augmentation). In this work, we investigate the impact of these two methods on the performance of three different named entity recognition tasks.

## introduction recently, deep neural network models have emerged in various fields of natural language processing (nlp) and replaced the mainstream position of conventional count-based methods @xcite @xcite . in addition to providing significant performance improvements, neural models often require high hardware conditions and a large amount of clean training data. however, there is usually only a limited amount of cleanly labeled data available, so techniques such as data augmentation and selftraining are commonly used to generate additional synthetic data. significant progress has been made in recent years in designing data augmentations for computer vision (cv) @xcite , automatic speech recognition (asr) @xcite , natural language understanding (nlu) @xcite and machine translation (mt) @xcite in supervised settings. in addition, semisupervised approaches using self-training techniques @xcite have shown promising performance in conventional named entity recognition (ner) systems @xcite @xcite . in this work, the effectiveness of self-training and data augmentation techniques on neural ner architectures is explored. to cover different data situations, we select three different datasets: the english conll 2003 @xcite dataset, which is the benchmark on which almost all ner systems report results, it is very clean and the baseline models achieve an f1 score of around 92.6%; the english w-nut 2017 @xcite dataset, which is generated by users and contains inconsistencies, baseline models get an f1 score of around 52.7%; the germeval 2014 @xcite dataset, a fairly clean german dataset with baseline scores of around 86.3% foot_0 . we observe that the baseline scores on clean datasets such as conll and germeval can hardly be improved by data adaptation techniques, while the performance on the w-nut dataset, which is relatively small and inconsistent, can be significantly improved. ## self-training though, the amount of annotated training data is limited for many nlp tasks, additional unlabeled data is available in most situations. semisupervised learning approaches make use of this additional data. a common way to do this is selftraining @xcite @xcite . at a high level, it consists of the following steps: for the performance of the method it is critical to find a heuristic to select a good subset of the automatically labeled data. the selected data should not introduce too many errors, but at the same time they should be informative, i.e. they should be useful to improve the decision boundary of the final model. one selection strategy @xcite is to calculate a confidence measure for all unlabeled sentences and to randomly sample sentences above a certain threshold. we consider two different confidence measures in this work. the first, hereinafter referred to as c 1 , is the posterior probability of the tag sequence y given the word sequence x: for the second confidence measure, we take into account the normalized tag scores at each position. to get a confidence score for the entire sequence, we take the minimum tag score of all positions. thus, c 2 is defined as follows: ## mlm-based data augmentation instead of using additional unlabeled data, we apply mlm-based data augmentation specifically for ner by masking and replacing original text tokens while maintaining labels. for each masked token x i : there are several configurations that can affect the performance of the data augmentation method: techniques of selecting the tokens to be replaced, the order of token replacement in case of multiple replacement and the criterion for selecting the best tokens from the predicted ones. this section studies the effect of these configurations. ## discussion and future work in this work, we present results of data adaptation methods on various ner tasks. we show that mlm-based data augmentation and self-training approaches lead to improvements on the small and noisy w-nut dataset. for mlm-based data augmentation, we suggest multiple ways of generating synthetic ner data. our results show that even without generating new entity spans we are able to achieve better results. for future work, we would like to incorporate label information into the augmentation pipeline by either conditioning the token predictions on labels or adding additional classification steps on top of the token prediction. another important question is the choice of the mlm and the impact of taskspecific fine-tuning. further investigations into the filtering step should also be carried out. for both self-training and mlm-based data aug-mentation we would like to improve the integration in the training process. the contribution of the original training data to the loss function could be increased or additional data could be weighted by their confidence. finally, we would like to test whether we can combine the two methods to achieve additional improvements.