S pan NER : Named Entity Re-/Recognition as Span Prediction (2021)

Abstract: Recent years have seen the paradigm shift of Named Entity Recognition (NER) systems from sequence labeling to span prediction. Despite its preliminary effectiveness, the span prediction model’s architectural bias has not been fully understood. In this paper, we first investigate the strengths and weaknesses when the span prediction model is used for named entity recognition compared with the sequence labeling framework and how to further improve it, which motivates us to make complementary advantages of systems based on different paradigms. We then reveal that span prediction, simultaneously, can serve as a system combiner to re-recognize named entities from different systems’ outputs. We experimentally implement 154 systems on 11 datasets, covering three languages, comprehensive results show the effectiveness of span prediction models that both serve as base NER systems and system combiners. We make all codes and datasets available: https://github.com/neulab/spanner , as well as an online system demo: http://spanner.sh . Our model also has been deployed into the ExplainaBoard platform, which allows users to flexibly perform a system combination of top-scoring systems in an interactive way: http://explainaboard.nlpedia.ai/leaderboard/task-ner/ .

## introduction the rapid evolution of neural architectures @xcite @xcite and large pre-trained models @xcite not only drive the state-of-the-art performance of many nlp tasks @xcite to a new level but also change the way how researchers formulate the task. for example, recent years have seen frequent paradigm shifts for the task of named entity recognition (ner) from token-level tagging, which conceptualize ner as a sequence labeling (seqlab) task @xcite @xcite @xcite @xcite @xcite @xcite , to span-level prediction (spanner) @xcite @xcite @xcite , which regards ner either as question answering @xcite , span classification @xcite @xcite , and dependency parsing tasks @xcite . however, despite the success of span predictionbased systems, as a relatively newly-explored framework, the understanding of its architectural bias has not been fully understood so far. for example, what are the complementary advantages compared with seqlab frameworks and how to make full use of them? motivated by this, in this paper, we make two scientific contributions. we first investigate what strengths and weaknesses are when ner is conceptualized as a span prediction task. to achieve this goal, we perform a fine-grained evaluation of spanner systems against seqlab systems and find there are clear complementary advantages between these two frameworks. for example, seqlab-based models are better at dealing with those entities that are long and with low label consistency. by contrast, spanner systems do better in sentences with more out-of-vocabulary (oov) words and entities with medium length ( §3.3). secondly, we reveal the unique advantage brought by the architectural bias of the span prediction framework: it can not only be used as a base system for named entity recognition but also serve as a meta-system to combine multiple ner systems' outputs. in other words, the span prediction model play two roles showing in fig. 1: (i) as a base ner system; and (ii) as a system combiner of multiple base systems. we claim that compared with traditional ensemble learning of the ner task, spanner combiners are advantageous in the following aspects: experimentally, we first implement 154 systems on 11 datasets, on which we comprehensively evaluate the effectiveness of our proposed span prediction-based system combiner. empirical results show its superior performance against several typical ensemble learning algorithms. lastly, we make an engineering contribution that benefits from the practicality of our proposed methods. specifically, we developed an online demo system based on our proposed method, and integrate it into the ner leaderboard, which is very convenient for researchers to find the complementarities among different combinations of systems, and search for a new state-of-the-art system. ## span prediction for ne recognition although this is not the first work that formulates ner as a span prediction problem @xcite @xcite @xcite , we contribute by @xcite exploring how different design choices influence the performance of spanner and (2) interpreting complementary strengths between seqlab and spanner with different design choices. in what follows, we first detail span prediction-based ner systems with the vanilla configuration and proposed advanced featurization. ## span prediction for ne re-recognition the development of ensemble learning for ner systems, so far, lags behind the architectural evolution of the ner task. based on our evidence from §3.3, we propose a new ensemble learning framework for ner systems. ## experiment 5.1 base systems to make a thorough evaluation of spanner as a system combiner, as illustrated in tab. 3, we first implement 10 seqlab based systems that cover rich combinations of popular neural components. to be fair for other system combination methods, we also include two spanners as base systems. to reduce the uncertainty, we run experiments with multiple trials and also perform the significant test with wilcoxon signed-rank test @xcite at @xmath0.05. ## implications and future directions co-evolution of nlp systems and their combiners systems for nlp tasks (e.g., ner model) and their combiners (e.g., ensemble learning for ner) are developing in two parallel directions. this paper builds the connection between them and proposes a model that can be utilized as both a base ner system and a system combiner. our work opens up a direction toward making the algorithms of nlp models and system combination co-evolved. the unified idea can be applied to other nlp tasks, and some traditional methods like reranking in syntactic parsing can be re-visited. for example, we can formulate constituency parsing @xcite as well as its re-ranking @xcite as a span prediction @xcite problem, which is be unified and parameterized with the same form. combinaboard it has become a trend to use a leaderboard (e.g., paperwithcode 10 ) to track current progress in a particular field, especially with the rapid emergence of a plethora of models. leaderboard makes us pay more attention to and even obsess over the state-of-the-art systems @xcite . we argue that leaderboard with an effective system combination (dubbed combinaboard) feature would allow researchers to quickly find the complementarities among different systems. as a result, the value of a worse-ranked model still could be observed through its combined results. in this paper, we make the first step towards this by releasing a preliminary combinaboard for the ner task http://spanner.sh . our model also has been deployed into the explainaboard @xcite platform, which allows users to flexibly perform system combination of top-scoring systems in an interactive way: http://explainaboard. nlpedia.ai/leaderboard/task-ner/ 10 https://paperswithcode.com/