Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning (2021)

Abstract: Recent advances in Named Entity Recognition (NER) show that document-level contexts can significantly improve model performance. In many application scenarios, however, such contexts are not available. In this paper, we propose to find external contexts of a sentence by retrieving and selecting a set of semantically relevant texts through a search engine, with the original sentence as the query. We find empirically that the contextual representations computed on the retrieval-based input view, constructed through the concatenation of a sentence and its external contexts, can achieve significantly improved performance compared to the original input view based only on the sentence. Furthermore, we can improve the model performance of both input views by Cooperative Learning, a training method that encourages the two input views to produce similar contextual representations or output label distributions. Experiments show that our approach can achieve new state-of-the-art performance on 8 NER data sets across 5 domains.

## introduction pretrained contextual embeddings such as elmo @xcite , flair @xcite and bert @xcite have significantly improved the accuracy of named entity recognition (ner) models. recent work @xcite @xcite found that including document-level contexts of the target sentence in the input of contextual embeddings methods can further boost the accuracy of ner models. however, there are a lot of application scenarios senate democrats eliminated the nuclear option when they had the majority a few years ago , over republican objections . president obama called for eliminating the legislative filibuster last month , which could occur if democrats retake the senate . some republicans say it s time to undo a wrong committed by reid . senate republicans are considering using the nuclear option to end a potential democratic filibuster and confirm neil gorsuch to the supreme court . senate republicans deployed the nuclear option on wednesday to drastically reduce the time it takes to confirm hundreds of president trump s nominees . label: group label: non entity input sentence: retrieved texts: eliminated president option majority republican some senate . some republican filibuster democrats a senate republicans the democratic some republicans some republicans wrong committed republicans nuclear democratic gorsuch republicans wednesday president figure 1: a motivating example from wnut-17 dataset. the retrieved texts help the model to correctly predict the named entities of "democrats" and "republican". in which document-level contexts are unavailable in practice. for example, there are sometimes no available contexts in users' search queries, tweets and short comments in various domains such as social media and e-commerce domains. when professional annotators annotate ambiguous named entities in such cases, they usually rely on domain knowledge for disambiguation. this kind of knowledge can often be found through a search engine. moreover, when the annotators are not sure about a certain entity, they are usually encouraged to find related knowledge through a search engine @xcite . therefore, we believe that ner models can benefit from such a process as well. in this paper, we propose to improve ner models by retrieving texts related to the input sentence by an off-the-shelf search engine. we re-rank the retrieved texts according to their semantic relevance to the input sentence and select several top-ranking texts as the external contexts. consequently, we concatenate the input sentence and external contexts together as a new retrieval-based input view and feed it to the pretrained contextual embedding module, so that the resulting semantic representations of the input tokens can be improved. the token representations are then fed into a crf layer for named entity prediction. a motivating example is shown in figure 1 . moreover, we consider utilizing the new input view to improve model performance with the original input view that does not have external contexts. this can be useful in application scenarios when external contexts are unavailable or undesirable (e.g., in time-critical scenarios). to this end, we propose cooperative learning (cl) that encourages the two input views to produce similar predictions. we propose two approaches to cl which minimize either the l 2 distances between the token representations of the two input views or the kullback-leibler (kl) divergence between the prediction distributions of the two input views during training. our experiments show that including the retrieved external contexts can significantly improve the accuracy of ner models on 8 ner datasets from 5 domains. with cl, the accuracy of the ner models with both input views can be further improved. our approaches outperform previous state-of-the-art approaches in each domain. the contributions of this paper are: ## framework to further improve the model, we use cooperative learning to train a unified model that is strong in both input views. with cl, the model is additionally constrained to be consistent in the internal representations or the output distributions of both input views. the architecture of our framework is shown in figure 2 . ## analysis we use the wnut-17 dataset in the analysis. ## conclusion in this paper, we propose to improve the ner model's accuracy by retrieving related contexts from a search engine as external contexts of the inputs. to improve the robustness of the models when no external contexts are available, we propose cooperative learning. cooperative learning adds constraints between two input views over either the token representations or label distributions of both input views to be consistent. empirical results show that our approach significantly outperforms the baseline models and previous state-of-the-art approaches on the datasets over 5 domains. we also show the effectiveness of cooperative learning in a semi-supervised training manner.