Improving Model Generalization: A C hinese Named Entity Recognition Case Study (2021)

Abstract: Generalization is an important ability that helps to ensure that a machine learning model can perform well on unseen data. In this paper, we study the effect of data bias on model generalization, using Chinese Named Entity Recognition (NER) as a case study. Specifically, we analyzed five benchmarking datasets for Chinese NER, and observed the following two types of data bias that can compromise model generalization ability. Firstly, the test sets of all the five datasets contain a significant proportion of entities that have been seen in the training sets. Such test data would therefore not be able to reflect the true generalization ability of a model. Secondly, all datasets are dominated by a few fat-head entities, i.e., entities appearing with particularly high frequency. As a result, a model might be able to produce high prediction accuracy simply by keyword memorization without leveraging context knowledge. To address these data biases, we first refine each test set by excluding seen entities from it, so as to better evaluate a model’s generalization ability. Then, we propose a simple yet effective entity resampling method to make entities within the same category distributed equally, encouraging a model to leverage both name and context knowledge in the training process. Experimental results demonstrate that the proposed entity resampling method significantly improves a model’s ability in detecting unseen entities, especially for company, organization and position categories.

## introduction named entity recognition (ner) is a fundamental building block for various downstream natural language processing tasks such as relation extraction @xcite , event extraction @xcite , information retrieval @xcite , question answering @xcite , etc. due to the ambiguous word boundaries and complex composition @xcite , chinese ner task is more challenging compared with english ner. recently, by leveraging upon the pretrained language model (e.g, bert @xcite , etc.), we have witnessed superior performances on chinese ner datasets, including: msra, weibo, ontonotes 4.0 and resume @xcite @xcite . despite the superior performance of the fine-tuned models, we argue that there are two types of data bias that can compromise the model generalization ability. first, we observe that in widely used chinese ner datasets, 50% to 70% entities in test data are seen in the training data. such test data would therefore not be able to evaluate the true generalization ability of a model. second, the datasets are dominated by a few fat-head entities, i.e., entities appearing with particularly high frequency. for example, within the organization category of cluener @xcite , fat-head entity 曼联 (manchester united) appears 59 times, while 法兰克福队 (eintracht frankfurt) occurs only once. as a result, a model might be encouraged to memorize those fat-head entities rather than leveraging context knowledge during training process. the rationale is that given the same entity and diverse contexts, the easiest way for model convergence is to memorize the entity rather than extracting patterns from the diverse contexts. to address these data biases, we first refine each test set by excluding seen entities from it, so as to better evaluate a model's generalization ability. then, we propose a simple yet effective entity rebalancing method to make entities within the same category distributed equally, encouraging a model to leverage both name and context knowledge in the training process. the contributions of this paper are as follows. • we analyze five benchmarking chinese ner datasets and identify two types of data bias that can compromise model generalization ability. in particular, the f1 score has been improved by 12.61% and 37.14% on the organization category of cluener and msra dataset respectively. ## method to improve model's generalization ability in detecting unseen entities, we argue that the model should be trained to leverage both name and context knowledge @xcite . thus, we propose a simple yet effective entity rebalancing algorithm. the main idea is to make the annotated entity equally distributed within the same category. there are two major reasons why the proposed entity rebalancing algorithm works. first, the equal distribution will encourage the model to leverage both name knowledge and context knowledge, since there are no simple statistical cues @xcite to exploit due to uneven distribution. second, different entities within the same category should be interchangeable semantically in most cases, which avoids the train-test discrepancy. the proposed algorithm works as follows. first, rebalance the annotated entity frequency in the training data. let c l denotes the original entity frequency counter of category l. for example, given c @xmath0: 11, e 2 : 1, e 3 : 1}, which means entity e 1 is annotated 11 times, and both e 2 and e 3 are annotated once in the category l, which is very imbalanced. then we turn c l to the balanced entity frequency counter c b l , which is c b @xmath1: 5, e 2 : 4, e 3 : 4}. in c b l , the difference between the maximum and minimum entity frequency is 1 at most. second, replace the fathead entity with randomly sampled entity of the same category, once its accumulated occurrence surpasses the rebalanced frequency in c b l . details are shown in algorithm 1. algorithm 1: entity replacement algorithm foreach sentence in dataset do foreach ent text, ent label in sentence do @xmath2; if c b l [ent text] > 0 then keep ent text as it is; c b l [ent text] -= 1; else sample ent s from c b l if c b l [ent s] > 0; replace ent text with ent s; c b l [ent s] -= 1; ## conclusion and future work in this paper, we take chinese ner as a case study, aiming to improve the model generalization by mitigating the data bias. we first refine each test set by excluding seen entities from it, so as to better evaluate a model's generalization ability. then, we propose an entity rebalancing method to make entities within the same category distributed equally. experimental results show that the proposed entity rebalancing method significantly improves a model's ability in detecting unseen entities. as future work, we will first investigate the generalizability of this study to non-chinese ner. second, we will improve the entity replacement algorithm by leveraging language model so that the replaced entity is more semantically plausible.