Validating Label Consistency in NER Data Annotation (2021)

Abstract: Data annotation plays a crucial role in ensuring your named entity recognition (NER) projects are trained with the right information to learn from. Producing the most accurate labels is a challenge due to the complexity involved with annotation. Label inconsistency between multiple subsets of data annotation (e.g., training set and test set, or multiple training subsets) is an indicator of label mistakes. In this work, we present an empirical method to explore the relationship between label (in-)consistency and NER model performance. It can be used to validate the label consistency (or catches the inconsistency) in multiple sets of NER data annotation. In experiments, our method identified the label inconsistency of test data in SCIERC and CoNLL03 datasets (with 26.7% and 5.4% label mistakes). It validated the consistency in the corrected version of both datasets.

## introduction named entity recognition (ner) is one of the foundations of many downstream tasks such as relation extraction, event detection, and knowledge graph construction. ner models require vast amounts of labeled data to learn and identify patterns that humans cannot continuously. it is really about getting accurate data to train the models. when end-to-end neural models achieve excellent performance on ner in various domains @xcite @xcite @xcite , building useful and challenging ner benchmarks, such as conll03, wnut16, and scierc, contributes significantly to the research community. data annotation plays a crucial role in building benchmarks and ensuring nlp models are trained with the correct information to learn from @xcite @xcite . producing the necessary annotation from any asset at scale is a challenge, mainly because of the complexity involved with annotation. getting the most accurate labels demands time and expertise. label mistakes can hardly be avoided, especially when the labeling process splits the data into multiple sets for distributed annotation. the mistakes cause label inconsistency between subsets of annotated data (e.g., training set and test set or multiple training subsets). for example, in the conll03 dataset @xcite , a standard ner benchmark that has been cited over 2,300 times, label mistakes were found in 5.38% of the test set @xcite . note that the stateof-the-art results on conll03 have achieved an f1 score of ∼ .93. so even if the label mistakes make up a tiny part, they cannot be negligible when researchers are trying to improve the results further. in the work of wang et al., five annotators were recruited to correct the label mistakes. compared to the original test set results, the corrected test set results are more accurate and stable. however, two critical issues were not resolved in this process: i) how to identify label inconsistency between the subsets of annotated data? ii) how to validate that the label consistency was recovered by the correction? another example is scierc @xcite ) (cited ∼50 times) which is a multi-task (including ner) benchmark in ai domain. it has 1,861 sentences for training, 455 for dev, and 551 for test. when we looked at the false predictions given by sciie which was a multi-task model released along with the scierc dataset, we found that as many as 147 (26.7% of the test set) sentences were not properly annotated. (we also recruited five annotators and counted a mistake when all the annotators report it.) three examples are given in ## related work ner is typically cast as a sequence labeling problem and solved by models integrate lstms, crf, and language models @xcite @xcite . another idea is to generate span candidates and predict their type. span-based models have been proposed with multitask learning strategies @xcite . the multiple tasks include concept recognition, relation extraction, and co-reference resolution. researchers notice label mistakes in many nlp tasks @xcite @xcite . for instance, it is reported that the bottleneck of the pos tagging task is the consistency of the annotation result @xcite . people tried to detect label mistakes automatically and minimize the influence of noise in training. the mistake re-weighting mechanism is effective in the ner task @xcite . we focus on visually evaluating the label consistency. ## conclusion we presented an empirical method to explore the relationship between label consistency and ner model performance. it identified the label inconsistency of test data in scierc and conll03 datasets (with 26.7% and 5.4% label mistakes). it validated the label consistency in multiple sets of ner data annotation on two benchmarks, conll03 and scierc.