Do C o NLL -2003 Named Entity Taggers Still Work Well in 2023? (2023)

Abstract: The CoNLL-2003 English named entity recognition (NER) dataset has been widely used to train and evaluate NER models for almost 20 years. However, it is unclear how well models that are trained on this 20-year-old data and developed over a period of decades using the same test set will perform when applied on modern data. In this paper, we evaluate the generalization of over 20 different models trained on CoNLL-2003, and show that NER models have very different generalization. Surprisingly, we find no evidence of performance degradation in pre-trained Transformers, such as RoBERTa and T5, even when fine-tuned using decades-old data. We investigate why some models generalize well to new data while others do not, and attempt to disentangle the effects of temporal drift and overfitting due to test reuse. Our analysis suggests that most deterioration is due to temporal mismatch between the pre-training corpora and the downstream test sets. We found that four factors are important for good generalization: model architecture, number of parameters, time period of the pre-training corpus, in addition to the amount of fine-tuning data. We suggest current evaluation methods have, in some sense, underestimated progress on NER over the past 20 years, as NER models have not only improved on the original CoNLL-2003 test set, but improved even more on modern data. Our datasets can be found at https://github.com/ShuhengL/acl2023_conllpp .

## introduction the progress of natural language processing (nlp) is typically measured using performance metrics like accuracy or f 1 score on public test sets. for instance, the top line in figure 1 shows the steady improvement of selected models on the conll-2003 english named entity recognition (ner) test set @xcite over the course of 15 years @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite as measured by published f 1 scores. figure 1 : progress on ner from 2005-2020, as measured using published f 1 scores on the @xcite and conll++ (data from 2020) english ner test set. the gap between the two grows smaller as time passes by, showing improved generalization of models developed over time. however, these scores are all calculated using the same publicly available test set, which raises several questions. one concern is how much of this progress is actually due to adaptive overfitting, i.e. over-estimating performance by reusing the same test set, as opposed to genuine improvement @xcite @xcite . in addition, there is also the issue of temporal drift as training data ages, which can negatively impact performance on modern data @xcite @xcite . performance degradation is a significant concern in applications that use ner, such as text deidentification @xcite , relation extraction @xcite , linking entities to a knowledge base @xcite , etc. however, continuously annotating, training, and evaluating new models on new data is not always possible. ner models that are trained on decadesold data and evaluated on heavily-used public development and test sets may struggle to perform well on modern data, which highlights the need to consider these factors when assessing performance. we report the publication time of the articles, the numbers of four different types of entities, the number of tokens, unique tokens and average number of tokens per sentence. 1 to understand how well ner works when models have been developed over 20 years using the same dataset, we created a new test set called conll++. we closely modeled conll++ after the conll-2003 test set, using news articles from 2020 instead of 1996, as in the original dataset. we carefully controlled for other variables, making results on the two datasets as comparable as possible, with the exception of the time frame. an example of an annotated sentence from conll++ is shown below: ambassador o to o the o united i-org nations i-org : o linda i-per thomas-greenfield i-per using conll++, we conduct an empirical study of more than 20 ner models that were trained on the original conll-2003 training split. our analysis shows that different models can have very different generalization when moving to modern data. simply comparing the performance of models on the conll-2003 test set does not tell the whole story of progress on ner over the past 20 years. similar to the findings of @xcite on the imagenet dataset @xcite , we do not observe evidence of widespread overfitting on conll-2003. on average, each point of f 1 improvement on the conll-2003 test set translates to a larger improvement on conll++ (see figure 2 ), suggesting overall improvements on the original dataset between 2003-2020 are mostly not due to overfitting. rather, most performance deterioration appears to be caused by temporal misalignment @xcite . suprisingly, for some models (e.g. roberta and t5), we find no evidence of performance degradation at all, despite the fact they are fine-tuned on a 20-year-old public dataset. we conduct an extensive analysis, which suggests that model size, architecture, amount of fine-tuning data, and pretraining corpus are all important factors for generalization in ner. name architecture reference corpus corpus time ∆f 1 (%) ∆rank bilstm-crf glove+rnn+crf lample et al. (2016) 5 wp till 2014 -20.25 -7 bilstm-cnn glove+rnn chiu and nichols (2016) 6 wp till 2014 -15.09 0 stanford nlp crf finkel et al. (2005) ---13.25 0 scibert bert beltagy et al. (2019) ss till 2019* -8.94 +2 bilstm-cnn-crf glove+rnn+crf ma and hovy (2016) 7 wp till 2014 -6.52 -1 bilstm-crf-elmo elmo+rnn+crf peters et al. (2018) 1b till 2011* -5.72 -7 flair glove+flair+rnn+crf akbik et al. (2018) wp till 2014* -5.57 -7 1b till 2011* stanza flair+rnn qi et al. (2020) 1b till 2011* -5.19 -8 pooled flair glove+flair+rnn+crf akbik et al. (2019) wp till 2014* -4.65 -6 1b till 2011* mbert bert devlin et al. (2019) wp 01/2001-2018* -4.22 -1 gigabert bert lan et al. (2020a) g5 01/2009-12/2010 -3.90 0 wp till 2019* os 11/2018 albert base bert lan et al. (2020b) bp till 2018* -3.61 +3 albert xxl bert lan et al. (2020b) bp till 2018* -2.22 +4 bert large bert devlin et al. (2019) bc till 2015* -2.01 +3 wp 01/2001-2018* xlm-roberta base roberta conneau et al. (2020) cc till 2020* -0.90 +5 t5 large transformer raffel et al. (2020) c4 04/2019 -0.59 +11 roberta large roberta liu et al. (2019) bp till 2018* +0.64 +3 cn 09/2016-02/2019 ow till 2019* st till 2018* t5 3b transformer raffel et al. (2020) c4 04/2019 +0.67 +1 longformer base roberta beltagy et al. (2020) bp till 2018* +1.00 +5 rp till 2019 st till 2018* rn 12/2016-03/2019 news-roberta base roberta gururangan et al. (2020) rp till 2019 +1.06 +5 rn 12/2016-03/2019 luke large roberta+easa † yamada et al. (2020) rp till 2019 +1.10 0 wp till 12/2018 inter-rater agreement: we find that the conll++ annotations closely follow the style of the original dataset. when considering labels in the conll-2003 test set as gold, our manual reannotation achieves a 95.46 f 1 score. 4 the second author's annotation, when considering the first author's as gold, receives a 96.23 f 1 score on overlapping articles. the token-level cohen's kappa between the two authors is 97.42, which can be considered almost perfect agreement @xcite . ## experimental setup we select models with a variety of architectures and pre-training corpora and fine-tune these models to study how different factors affect generalization. none of the models used any pre-training data that temporally overlap with conll++, eliminating the possibility of articles in conll++ appearing in any pre-training corpus. a list of all models and their implementation details can be found in scripts for fine-tuning flair and elmo are adapted from @xcite . 5 other recurrent neural network (rnn) models are trained using various github repositories (see footnotes 6, 7 and 8). we fine-tune the bert and roberta models with the huggingface transformers library @xcite , except luke with allennlp @xcite . t5 is fine-tuned to conditionally generate ner tags around entities (e.g. <@xmath13. a hyperparameter search is conducted for each model. we follow the recommended search space for a model if available in its publication. {8, 16, 32} and {1e-5, 2e-5, 3e-5, 5e-5} are used for most searches for batch sizes and learning rates respectively. appendix b provides more details on the hyperparameter search. we train models on the conll-2003 training set for 10 epochs, and use the dev set to select the best epoch and other hyperparameters for evaluation. each model is evaluated five times with different random seeds on the conll-2003 test set and on conll++ to obtain the average f 1 . in as we can see in figure 2 and in the following sub-sections, we evaluate the impact of a number of factors on generalization. in §5, we attempt to disentangle to what extent the observed performance drops on conll++ are caused by temporal deterioration, or adaptive overfitting. ## what causes the performance drop observed for some models? models in ## related work how well pre-trained lms adapt to data from future time periods has undergone extensive study. temporal degradation has been found to be a challenge for many tasks, including language modeling @xcite , ner @xcite @xcite , qa @xcite , entity linking (zaporojets et al., 2022), and others @xcite . all of this work has found that the performance of lms degrades as the temporal distance between the training data and the test data increases, sometimes called "temporal misalignment" @xcite . in contrast to the prior work, we study performance deterioration on a dataset that has been heavily used to develop ner models over a period of 20 years, and conduct extensive experiments that aim to disentangle the effects of aging training sets from those due to heavy test reuse. most closely related to our work is @xcite , who analyzed a recently created twitter ner dataset (rijhwani and preotiuc-pietro, 2020) over the period 2014-2019, and found no performance deterioration when using robertabased representations. we build on this line of work by carefully measuring performance deterioration of models trained on the conll-2003 dataset when evaluated on modern data. we analyze which factors are necessary for an ner model trained on a 20-year-old dataset to generalize well to modern data. furthermore, the large 20-year gap helps us focus on not only temporal deterioration, but also if the extensive test reuse leads to adaptive overfitting. we present evidence in support of the hypothesis that most performance degradation is due to temporal drift and not adaptive overfitting. prior work has attempted to mitigate temporal degradation, mostly through continuously updating lms with new data @xcite @xcite . @xcite explored this idea but found that temporal adaptation is not as effective as fine-tuning on the data from whose time period the dataset is drawn. in addition, catastrophic forgetting @xcite can also be a problem when updating the lms. @xcite found that applying knowledge distillation @xcite based approaches to continual learning can mitigate catastrophic forgetting, while improving the temporal generalization of lms. @xcite proposed to train the lms with an additional temporal objective by conditioning on the year of data, and found that this effectively mitigated catastrophic forgetting. @xcite created a lifelong benchmark for continuous training and evaluating lms. ## conclusion and future directons in this paper, we evaluate the generalization of ner models using conll++, a conll-style annotated ner test dataset with data from 2020. we conduct experiments on more than 20 models and find that models exhibit different generalizability. surprisingly, we find that generalizability is not affected by adaptive overfitting, but rather by temporal drift. to achieve better generalization, we need the combination of four factors: a modern transformer-based architecture, a large number of parameters, a large amount of fine-tuning data and a temporally closer pre-training corpus to the test set. we find that our progress on developping ner taggers is largely successful, showing not only good performance on individual test set but also good generalization on new data. this allows conll-2003 taggers to still work in 2023. future research can focus on ways to mitigate temporal drift. investigation on attributes of pretraining or fine-tuning corpora that causes temporal drift, such as change of entities mentioned, different usage of language, etc., can also shed light on the more specific impacts from temporal drift, thereby inspiring new and better ways to mitigate it. we hope that our work provides insights on factors affecting generalization and how to mitigate the negative impact, and calls for more research on this everlasting problem of generalization in the nlp community.