C lean C o NLL : A Nearly Noise-Free Named Entity Recognition Dataset (2023)

Abstract: The CoNLL-03 corpus is arguably the most well-known and utilized benchmark dataset for named entity recognition (NER). However, prior works found significant numbers of annotation errors, incompleteness, and inconsistencies in the data. This poses challenges to objectively comparing NER approaches and analyzing their errors, as current state-of-the-art models achieve F1-scores that are comparable to or even exceed the estimated noise level in CoNLL-03. To address this issue, we present a comprehensive relabeling effort assisted by automatic consistency checking that corrects 7.0% of all labels in the English CoNLL-03. Our effort adds a layer of entity linking annotation both for better explainability of NER labels and as additional safeguard of annotation quality. Our experimental evaluation finds not only that state-of-the-art approaches reach significantly higher F1-scores (97.1%) on our data, but crucially that the share of correct predictions falsely counted as errors due to annotation noise drops from 47% to 6%. This indicates that our resource is well suited to analyze the remaining errors made by state-of-the-art models, and that the theoretical upper bound even on high resource, coarse-grained NER is not yet reached. To facilitate such analysis, we make CleanCoNLL publicly available to the research community.

## introduction the conll-03 corpus is arguably the most well-known and utilized dataset for named entity recognition (ner). conll-03 is typically the crucial benchmark to evaluate new ner approaches @xcite @xcite , language models @xcite , andmost recently -instruction-tuned llms for their few-and zero-shot abilities @xcite ). in cleanconll, we leverage the entity link to the wikipedia article of the company to determine that this entity is indeed of type organization (org). further, our entity linking annotation makes more immediately transparent to human inspection why this entity is labeled as org. however, prior works have found significant data quality and consistency issues: @xcite identified a share of 3.8% of all labels as incorrect, with particularly high error rate in the test split (7.5% error rate). similarly, @xcite found errors in 5.4% of all sentences of the test split. qualitatively, various prior works noted inconsistencies where different labels are given to the same entity throughout the corpus, affecting for instance the names of sports teams when referred to by their geographical name @xcite , sports leagues @xcite , names of deities and religious contexts @xcite , and the names of ministries @xcite . further, various prior works noted segmentation errors in which token-, entity-and even sentenceboundaries are incorrectly set @xcite . we argue that these inconsistencies impair the usefulness of conll-03 for benchmarking stateof-the-art (sota) ner approaches, since such approaches generally report f1-scores that lie near (or even above) its estimated noise level. for instance, we find in this paper that nearly half of errors made by a state-of-the-art ner model are in fact correct predictions that are falsely counted as errors due to noisy annotation. we argue that given the maturity of research into high-resource ner, the research community requires a dataset with highly consistent, nearly noise-free labels in order to correctly estimate the performance of state-of-the-art models and to better understand their remaining errors. adding and leveraging entity links. to address this issue, we present cleanconll, a revised version of conll-03 that significantly improves annotation quality and consistency. our main idea is to introduce a second layer of annotation to the corpus that adds entity linking information. entity linking is the task of disambiguating named entities to a unique id in a knowledge base, typically the wikipedia page representing a specific entity. as figure 1 illustrates with an example sentence, adding entity links serves two purposes: first, entity links allow us to define automatic consistency checks on ner labels, since prior work found that ner labels can be (partially) derived from wikipedia page ids @xcite . second, we find that the disambiguating information provided by entity links makes ner labels more immediately transparent to human inspection, thus making it easier for annotators to spot mistakes and discuss disagreements. contributions. in more detail, our contributions are the following: we make the resource publicly available and encourage the community to use and further improve its quality. while this version has less noise than the original corpus, our own analysis (sec. 3.4) and prior work @xcite found that large numbers of errors still remain. merging aida. for initial corpus creation, we take advantage of the fact that aida @xcite , a well-known entity linking dataset, shares the same text base as conll-03. since the entity links in aida are expert-annotated by a group distinct of the original conll-03 annotators, they provide a layer of high quality annotation of an arguably more complex task. however, merging these two corpora was not straightforward since there are discrepancies both in terms of sentence boundary definitions as well as the coverage of entity annotations. we found that of all 34,941 entities in the reiss conll version, 78.9% have an entity link in aida, while the remaining 21.1% (7,369 entities) do not. we found three main reasons for lack of coverage: (1) rare entities that at time of labeling had no corresponding wikipedia article are unlabeled in aida. the definition of what constitutes a named entity differs in some instances, with conll-03 for instance labeling the span "boeing 707" as entity, while aida links only "boeing". deriving ner labels from entity links. we derive ner labels from wikipedia url links by leveraging wikidata foot_2 , a manually curated knowledge graph that connects entities (represented by wikipedia pages) through a set of relations. we leveraged two of these relations: (1) instance_of, which connects an instance to the conceptual class it instantiates, (2) subclass_of, which connects a conceptual class to a more general conceptual class. refer to figure 2 for an example of this: the mention "china" is linked to the entity "china_national_football_team", which is a wikidata instance of "national association football team", which in turn is a subclass of "national sports team". with these two relations, we automatically select conceptual classes for each linked entity in our corpus. for the most common of these classes, we manually defined a mapping to conll-03 entity types. for instance, we manually defined that the wikidata category "sports team" maps to the entity type org. a full description of our mappings is provided in appendix a.1. using these mappings, we automatically mapped 4,163 (74.4%) of the 5,595 unique wikipedia labels in aida to their ner type. however, as we prioritized precision over recall in the development of our manual mappings, 1,432 (25.6%) could not be classified automatically because either none or multiple of our heuristics took effect. manual annotation. these remaining 1,432 entity links were manually assigned ner types. this annotation was carried out separately for all labels by two experts who were provided with guidelines as well as the respective wikipedia and wikidata information. the inter-annotator agreement was 94.9%, cases of disagreement were subsequently resolved in discussion. using the resulting mappings, we updated all ner labels in the corpus to be consistent with the entity link. ## evaluation we aim to get an understanding of annotation quality of cleanconll. to this end, we discuss its properties (sec. 3.1) and perform a manual evaluation of 100 sentences to assess and compare annotation quality with earlier variants (sec. 3.2-3.4). ## release of cleanconll we make cleanconll publicly available to the research community through a github repository. we follow conll-03 in releasing only annotations and a script to merge them with the reuters corpus that can be obtained free of charge for research purposes foot_6 to produce the annotated corpus, see figure 6 for an excerpt. annotation layers. we release three new layers of annotation: (1) our entity linking annotations, (2) the final cleanconll ner tags, and (3) our ner tags after phase 2, i.e. before reverting adjectival affiliations to misc. we release the latter as we believe some researchers might find typeresolved adjectival affiliations useful, and further discuss advantages and disadvantages of both ner label versions in appendix a.4. ## related work as previously mentioned, other works have pointed out poor annotation quality in conll-03 @xcite @xcite . most relevant for our work are @xcite and @xcite , who both published their own corrected versions of the corpus. @xcite propose a method to improve training on noisy datasets via lowering the weights of potentially incorrect instances. they use their method to identify errors, but only focus on the test set. they release a manually corrected test set called conll++. similarly, @xcite use majority voting in an ensemble of trained models to identify potential errors and release a manually corrected version of conll-03. several other works also touch the topic of label error detection or model error classification in conll-03 @xcite @xcite . @xcite do not correct annotation errors but create a completely new dataset which they also call conll++ (not to be confused with the corrections by @xcite ), that emulated conll-03 label usage and distribution as well as text domain, but consists of newer articles. our annotation effort differs in that we (1) first integrate a layer of entity linking information for consistency checking and to assist manual annotation, and (2) use trained models for cross-checking over multiple iterations during our manual reannotation process. our evaluation indicates that cleanconll has significantly higher annotation quality than all previous versions. ## conclusion we presented cleanconll, a corrected version of the classic ner benchmark conll-03, with updated and more consistent ner labels. our approach adds an additional layer of entity linking annotation to disambiguate entities whenever possible to their wikipedia page. we leverage this layer to both identify inconsistencies in annotation, and to make labeling decisions transparent to human inspection. further, we apply an approach of cross-checking throughout our manual annotation process as additional consistency check. through our relabeling effort, we updated 7% of all labels of conll-03, and added 5.5% new entity links over the base aida resource. our eval-uation strongly indicates that overall annotation quality and consistency is significantly improved in cleanconll. in sec. 3.2, we estimated a noise share of only around 1% for our sample, whereas conll-03 had an estimated noise share of 10%. we found that current models reach significantly higher f1-scores on our dataset (sec. 4.1), and that the share of correct predictions incorrectly classified as errors drops from 47% to 6% (sec. 4.2). these findings indicate the validity of our relabeling approach, and the usefulness of clean-conll for training and evaluating state-of-the-art ner models. to enable the research community to do so, we make our new layers of annotation publicly available through an open source repository modeled after the ud treebanks. we hope that active participation by the research community will help us to continuously further improve annotation quality, with the ultimate goal of an entirely "noise-free" ner dataset.