Robust Self-Augmentation for Named Entity Recognition with Meta Reweighting (2022)

Abstract: Self-augmentation has received increasing research interest recently to improve named entity recognition (NER) performance in low-resource scenarios. Token substitution and mixup are two feasible heterogeneous self-augmentation techniques for NER that can achieve effective performance with certain specialized efforts. Noticeably, self-augmentation may introduce potentially noisy augmented data. Prior research has mainly resorted to heuristic rule-based constraints to reduce the noise for specific self-augmentation methods individually. In this paper, we revisit these two typical self-augmentation methods for NER, and propose a unified meta-reweighting strategy for them to achieve a natural integration. Our method is easily extensible, imposing little effort on a specific self-augmentation method. Experiments on different Chinese and English NER benchmarks show that our token substitution and mixup method, as well as their integration, can achieve effective performance improvement. Based on the meta-reweighting mechanism, we can enhance the advantages of the self-augmentation techniques without much extra effort.

## introduction named entity recognition (ner), which aims to extract predefined named entities from a piece of unstructured text, is a fundamental task in the natural language processing (nlp) community, and has been studied extensively for several decades @xcite @xcite . recently, supervised sequence labeling neural models have been exploited most popularly for ner, leading to state-of-the-art (sota) performance @xcite @xcite . although great progress has been made, developing an effective ner model usually requires a * equal contributions. † corresponding author. large-scale and high-quality labeled training corpus, which is often difficult to be obtained in real-world scenarios due to the expensive and time-consuming annotations by human experts. moreover, it would be extremely serious because the target language, target domain, and the desired entity type could all be infinitely varied. as a result, the low-resource setting with only a small amount of annotated corpus available is far more common in practice, even though it may result in significant performance degradation due to the overfitting problem. self-augmentation is a prospective solution to this problem, which has received widespread attention @xcite @xcite @xcite . the major motivation is to generate a pseudo training example set deduced from the original gold-labeled training data automatically. for ner, a token-level task, the feasible self-augmentation techniques include token substitution @xcite and mixup @xcite , which are deformed at the ground-level inputs and the high-level hidden representations, respectively. nonetheless, there are still some limitations currently for the above token substitution and mixup methods. for one thing, both of them require some specialized efforts to improve their effectiveness due to the potential noise introduced by the selfaugmentation, which may restrict the valid seman-tic representation of the augmented data. for instance, token substitution is typically limited to the named entities in the training corpus @xcite , and the mixup tends to be imposed on the example pairs with small semantic distance gaps @xcite . for another thing, though the two techniques seem to be orthogonal and probably complementary to each other, it remains a potential challenge to effectively and naturally integrate them. in this work, we revisit the token substitution and mixup methods for ner, and investigate the two heterogeneous techniques under a unified meta reweighting framework (as illustrated in figure 1 ). first, we try to relax the previous constraints to a broader scope for these methods, allowing for more diverse and larger-scale pseudo training examples. however, this would inevitably produce some lowquality augmented examples (i.e., noisy pseudo data) in terms of linguistic correctness, which may negatively affect the model performance. to this end, we present a meta reweighting strategy for controlling the quality of the augmented examples and leading to noise-robust training. also, we can naturally integrate the two methods by using the example reweighting mechanism, without any specialization in a specific self-augmentation method. finally, we carry out experiments on several chinese and english ner benchmark datasets to evaluate our proposed methods. we mainly focus on the low-resource settings, which can be simulated by using only part of the standard training set when the scale is large. experimental results show that both our token substitution and mixup method coupled with the meta-reweighting can effectively improve the performance of our baseline model, and the combination can bring consistent improvement. positive gains become more significant as the scale of the training data decreases, indicating that our self-augmentation methods can handle the low-resource ner well. in addition, our methods can still work even with a large amount of training data. the code is available at https: //github.com/lindgew/metaaug4ner . ## our approach in this section, we firstly describe our baseline model. then, we present our self-augmentation methods to enhance the baseline model in the lowresource settings. finally, we elaborate on our meta reweighting strategy, which aims to alleviate the negative impact of the noisy augmented examples caused by the self-augmentation while also elegantly combining these augmentation methods. ## experiments 3.1 settings datasets to validate our methods, we conduct experiments on chinese benchmarks: ontonotes 4.0 @xcite and weibo ner @xcite , as well as english benchmarks: @xcite and ontonotes 5.0 2 (pradhan et al., 2013). the chinese datasets are split into training, development and test sections following @xcite while we take the same data split as @xcite and @xcite on the english datasets. we follow @xcite to use the bioes tagging scheme for all datasets. the detailed statistics can be found in ## related work in recent years, research on ner has concentrated on either enriching input text representations @xcite @xcite or refining model architectures with various external knowledge @xcite @xcite @xcite @xcite . particularly, ner model, with the aid of large pretrained language models @xcite @xcite , has achieved impressive performance gains. however, these models mostly depend on rich manual annotations, making it hard to cope with the low-resource challenges in real-world applications. instead of pursuing a sophisticated model architecture, in this work, we exploit the bilstm-crf model coupled with the pre-trained bert as our basic model structure. self-augmentation methods have been widely investigated in various nlp tasks @xcite @xcite @xcite . the mainstream methods can be broadly categorized into three types: (1) token substitution @xcite @xcite , which performs local substitution for a given sentence, (2) paraphrasing (kumar augmentation examples original [diana] per met [will carling] per at an exclusive gymnasium in [london] loc . positive ts [freddy pinas] per invited [john marzano] per at an available gym in [uk] loc . negative ts [tim henman] per visit [simpson] per at an available room in [nice] loc . positive mixup french 1997 budget due around september 10 -juppe. jewish 1999 deficit due about october 20 m.atherton. negative mixup olympic champion agassi meets karim alami of morocco in the first round. olympic champion nathalie lancien of france also missed the winning attack. many previous studies have explored the example weighting mechanism in domain adaption @xcite @xcite . @xcite and @xcite looked into the example weighting methods for cross-domain tasks. @xcite adapted the maml algorithm @xcite and proposed a meta-learning algorithm to automatically weight training examples of the noisy label using a small unbiased validation set. inspired by their work, we extend the meta example reweighting mechanism to the ner task, which is exploited to adaptively reweight mini-batch augmented examples during training. the main purpose is to mitigate the potential noise effects brought by the self-augmentation techniques, advancing a noiserobust model, especially in low-resource scenarios. ## conclusion in this paper, we re-examine two heterogeneous self-augmentation methods (i.e., ts and mixup) for ner, extending them into more unrestricted augmentations without heuristic constraints. we further exploit a meta reweighting strategy to alleviate the potential negative impact of noisy augmented examples introduced by the aforementioned relaxation. experiments conducted on several benchmarks show that our self-augmentation methods along with the meta reweighting mechanism are very effective in low-resource settings, and still work when enough training data is used. the combination of the two methods can lead to consistent performance improvement across all datasets. since our framework is general and does not rely on a specific model backbone, we will further investigate other feasible model structures.