Identifying Incorrect Labels in the C o NLL -2003 Corpus (2020)

Abstract: The CoNLL-2003 corpus for English-language named entity recognition (NER) is one of the most influential corpora for NER model research. A large number of publications, including many landmark works, have used this corpus as a source of ground truth for NER tasks. In this paper, we examine this corpus and identify over 1300 incorrect labels (out of 35089 in the corpus). In particular, the number of incorrect labels in the test fold is comparable to the number of errors that state-of-the-art models make when running inference over this corpus. We describe the process by which we identified these incorrect labels, using novel variants of techniques from semi-supervised learning. We also summarize the types of errors that we found, and we revisit several recent results in NER in light of the corrected data. Finally, we show experimentally that our corrections to the corpus have a positive impact on three state-of-the-art models.

## introduction the english-language portion of the conll-2003 shared task @xcite (henceforth @xcite is one of the most widely-used benchmarks for named entity recognition (ner) models. it consists of news articles from the reuters rcv1 corpus @xcite . since its debut, conll-2003 has played a central role in nlp research. over 2300 research papers have cited the original conll-2003 paper 1 . among these works, many are landmark results that have revolutionized the field of natural language processing, including glove embeddings @xcite , bert embeddings @xcite , conditional random fields @xcite , and bidirectional lstm models @xcite . the conll-2003 corpus continues to be used in ner research. the papers with code website (paper with code, 2020), which tracks state-of-the-art f1 scores foot_0 for this corpus, currently (as of july 2020) shows 43 results from 2016 through 2019 that improved this metric. while researchers have relied heavily on the conll-2003 corpus as a source of ground truth, few have paid attention to the corpus itself. errors in the corpus could potentially mislead and even divert the course of future research. recent work has pointed out that improper benchmarking can have significant impact on evaluating machine learning algorithms @xcite . the fact that @xcite and @xcite found many errors while examining parts of the corpus is even more alarming. a detailed examination of the corpus has become imperative. in this paper, we present our work on correcting labeling errors in the conll-2003 corpus. section 2 gives an overview of the corpus itself, the high-level process we followed, and related work. section 3 describes how we used a novel form of semi-supervised labeling to identify potentiallyincorrect labels. sections 4 and 5 describe how we examined and categorized the flagged labels. and sections 6 and 7 describe how we created a corrected version of the corpus and reevaluated past results. ## overview the conll-2003 corpus contains news articles from a subset of the reuters rcv1 corpus @xcite . entities are tagged using an extended version of the tagging policy from the message understanding conference @xcite (muc), with the addition of a new tag misc to cover entities not mentioned in muc's labeling rules. the data consists of text files in which each line holds information about one token. associated with each token are tags in insideoutside-begin (iob) format @xcite . the files, eng.train, eng.testa, and eng.testb, contain the train , dev , and test folds of the corpus, respectively. ## automated labeling we did not set out to relabel the conll-2003 corpus. when we started looking at this corpus, our intent was to identify entity mentions that older models are not able to extract, but that state-of-theart models are able to extract. we had hoped to use this information to drive continued improvements to these models. ## hand labeling each of our three ensembles produced two lists of labels: one list of labels that were in the corpus but not in the model outputs; and a second list of labels that were in the model outputs but not in the corpus. overall, we produced six lists of potentially-incorrect labels. four of these lists spanned the entire corpus, while the remaining two (from the original contest entries) only spanned the test and dev folds. we proceeded to examine these lists by hand, looking at each flagged label in the context of the target document. we focused on the labels where there was a strong agreement between the models in each ensemble. we started out by examining the labels where all models agreed, then moved onto the labels where all models but one agreed, and so on. as we progressed to labels with less agreement among models, the fraction of flagged labels that was actually incorrect decreased. when this fraction dropped below 20 percent, we stopped going through the ordered list of flagged labels. for each list of potentially-incorrect labels, one member of our team examined the labels, and a second member of our team audited the decisions that the first member had made. in total, we made 12 passes (3 ensembles × 2 sets of labels × 2 human reviewers) of manual review over the train and test folds of the corpus and 8 passes over the test fold. when we found that a label was incorrect, we coded the type of error and the required correction so that the error could be corrected automatically later on. we divided errors into several categories: appendix 9.1 shows examples of each error type. the data set that we have published as a companion to this paper (see section 2.1.1) includes complete lists of the errors that we found, both before and after manual review. ## incorrect labels identified in total, we examined 3182 labels our ensembles had flagged in the three folds of the corpus. we considered any label where fewer than 7 models agreed with the corpus label to be "flagged". of these labels, 1274 came from the test fold, 854 came from the dev fold, and 1054 came from the train fold; accounting for 22.6%, 14.3%, and 4.5% of their folds, respectively. as we noted in section 3.3, our models had significantly higher f1 scores on the train fold, both with and without cross-validation. because model outputs were closer to the corpus labels, the ensembles flagged fewer labels on this fold. however, the fraction of these labels that were actually incorrect was higher than that on the other folds: 34% versus 23%. of the errors correctly flagged, 184 were found by the ensemble composed of the original entrants' results; 641 were flagged by our custom models; and 275 errors were found by custom models with cross-validation. 372 of these errors were correctly flagged by two or more approaches. while ex-amining the affected documents, we found 470 additional errors in the vicinity of flagged errors. figure 3 shows the distribution of errors broken down by error type and source. the most frequent error type we found in the corpus was a tag type error, accounting for 48% of errors in total. both type errors were least frequent. our bert-based models found a higher fraction of sentence type errors, largely because these models were able to express spans that cross sentence boundaries. the entrants' outputs in our first ensemble, being constrained by the iob file format, were physically incapable of expressing a span that crosses a sentence boundary. we also suspect that many of these older models operated on one sentence at a time, while the document context feeding our bert embeddings could span multiple sentences. had we been aiming to maximize the f1 scores of our bert-based models, we would have postprocessed the outputs of these models to split spans along sentence boundaries. this lack of postprocessing led to a decrease in f1 score, but it enabled us to find more errors. the distribution of error types remained relatively constant across folds, with one exception: sentence errors accounted for a much larger fraction in the train fold -26% of errors, as opposed to the 8% and 10% rates in the dev and test folds, respectively. ## corrected conll-2003 corpus after identifying incorrect tags, spans and sentence boundaries, we created a corrected version of the original conll-2003 corpus, which we refer to as the corrected conll-2003 corpus. we used the text extensions for pandas library to parse the original corpus and extract tokens and spans for each entity. we created data files containing all of the vetted corrections from our hand labeling of ensemble outputs. we wrote a script that applies all of these corrections to the conll-2003 corpus, producing a corrected version of the corpus. for information on how to obtain the code and data necessary to recreate our corrected corpus, as well as all the experiment code for this paper, see section 2.1.1. ## experimental evaluation in this section, we first re-evaluate the entries from the original competition against the corrected test fold of the corpus. we then re-evaluate the metrics of three state-of-the-art ner models from recent literature on the corrected corpus. ## conclusion and future work the conll-2003 corpus is highly influential in named entity recognition (ner) research. it has been used for benchmarking many landmark ner models and has been continuing to play a critical role in recent research. in this paper, we took a closer look at the conll-2003 corpus and identified a number of errors. we used a semi-supervised method to identify these errors and then systematically corrected them. the primary contribution of this paper is the creation of a more error-free version of the conll-2003 corpus, which can potentially be used to evaluate past ner models more accurately and make future benchmarking more reliable. indeed, as our experiments on three recent state-of-the-art ner models have shown, our corrections to the corpus have a positive impact on these models: when evaluated on our corrected test fold, all three models trained on our corrected corpus outperformed their counterparts trained on the original corpus by a non-negligible margin. we firmly believe that benchmarking corpora are the lighthouses for research, and improving the quality of benchmarking corpora is of utmost importance in guiding the research community. we hope that others can replicate the process we applied to this corpus on other key corpora, and in doing so, improve the utility of these vital resources.