The Role of Global and Local Context in Named Entity Recognition (2023)

Abstract: Pre-trained transformer-based models have recently shown great performance when applied to Named Entity Recognition (NER). As the complexity of their self-attention mechanism prevents them from processing long documents at once, these models are usually applied in a sequential fashion. Such an approach unfortunately only incorporates local context and prevents leveraging global document context in long documents such as novels, which might hinder performance. In this article, we explore the impact of global document context, and its relationships with local context. We find that correctly retrieving global document context has a greater impact on performance than only leveraging local context, prompting for further research on how to better retrieve that context.

## introduction named entity recognition (ner) is a fundamental task in natural language processing (nlp), and is often used as a building block for solving higherlevel tasks. recently, pre-trained transformerbased models such as bert @xcite or luke @xcite showed great ner performance and have been able to push the state of the art further. these models, however, have a relatively short range because of the quadratic complexity of selfattention in the number of input tokens: as an example, bert @xcite can only process spans of up to 512 tokens. for longer documents, texts are usually processed sequentially using a rolling window. depending on the document, this local window may not always include all the context needed to perform inference, which may be present at the global document level. this leads to prediction errors @xcite : in ner, this often occurs when the type of an entity cannot be inferred from the local context. for * these authors contributed equally. in the novel, this prior knowledge comes from the fact that a human reader can recall previous mentions of elantris, even at a very long range. a sequentially applied vanilla transformer-based model, however, might make an error without a neighboring sentence clearly establishing the status of elantris as a city. while some works propose to retrieve external knowledge to disambiguate entities @xcite , external resources are not always available. furthermore, external retrieval might be more costly or less relevant than performing document-level context retrieval, provided the document contains the needed information, which depends on the type of document. therefore, we wish to explore the relevance of document-level context when performing ner. we place ourselves at the sentence level, and we distinguish and study two types of contexts: we seek to answer the following question: is local context sufficient when solving the ner task, or would the model obtain better performance by retrieving global document context? to answer this question, we conduct experiments on a literary ner dataset we improved from its original version @xcite . we release the annotation process, data and code necessary to reproduce these experiments under a free license foot_0 . ## conclusion and future work in this article, we explored the role of local and global context in named entity recognition. our results tend to show that, for literary texts, retrieving global document context is more effective at enhancing ner performance than retrieving only local context, even when using relatively simple retrieval heuristics. we also showed that a re-ranker model using simple document-level retrieval heuris-tics could obtain significant ner performance improvements. overall, our work prompts for further research in how to accurately retrieve global context for ner.