Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset (2023)

Abstract: While recent pre-trained transformer-based models can perform named entity recognition (NER) with great accuracy, their limited range remains an issue when applied to long documents such as whole novels. To alleviate this issue, a solution is to retrieve relevant context at the document level. Unfortunately, the lack of supervision for such a task means one has to settle for unsupervised approaches. Instead, we propose to generate a synthetic context retrieval training dataset using Alpaca, an instruction-tuned large language model (LLM). Using this dataset, we train a neural context retriever based on a BERT model that is able to find relevant context for NER. We show that our method outperforms several retrieval baselines for the NER task on an English literary dataset composed of the first chapter of 40 books.

## introduction named entity recognition (ner), as a fundamental building block of other natural language processing (nlp) tasks, has been the subject of a lot of researchers' attention. while pre-trained transformerbased models are able to solve the task with a very high f1-score @xcite , they still suffer from a range limitation caused by the quadratic complexity of the attention mechanism in the input sequence length. when applied to long documents such as entire novels, this range limitation prevents models from using global document-level context, since documents cannot be processed as a whole. however, document-level context is useful for entity disambiguation: for ner, lacking access to this context results in performance loss @xcite . to try and solve the range limitation of transformers, a number of authors recently explored "efficient transformers", with a sub-quadratic complexity (see @xcite for a survey). still, long sequences remain challenging @xcite . retrieval-based methods can circumvent these issues, by retrieving relevant context from the document and concatenating it to the input. unfortunately, no context retrieval dataset is available for ner, and manually annotating such a dataset would be costly. this lack of data prevents from using a specialized supervised context retrieval method. meanwhile, instruction learning has very recently seen an increasing focus from the nlp community @xcite . instructions-tuned models are able to solve a variety of tasks (classification, regression, recommendation...) provided they can be formulated as text generation problems. inspired by the progress of these models, we propose to generate a synthetic context retrieval dataset tailored to the ner task using alpaca @xcite , a recently released instruction-tuned large language model (llm). using this synthetic dataset, we train a neural context retriever that ranks contexts according to their relevance to a given input text for the ner task. since applying this retriever on a whole document is costly, we instead use it as a re-ranker, by first retrieving a set of candidate contexts using simple retrieval heuristics and then keeping the best sentences among these. in this article, we study the application of our proposed method on a corpus of literary novels, since the length of novels make them particularly prone to the range limitation of transformers. we first describe the generation process of our synthetic retrieval dataset. we then evaluate the influence of the neural context retriever trained on this dataset in terms of ner performance, by comparing it to unsupervised retrieval methods. we also compare our re-ranker with other supervised re-rankers trained on a manually annotated context retrieval dataset unrelated to literary ner. we explore whether the number of parameters of the llm used for generating our synthetic context retrieval dataset has an influence on the trained neural retriever by comparing alpaca-7b and alpaca-13b, and the influence of the number of candidate sentences to retrieve before re-ranking. additionally, we study the influence of the context window (i.e. context retrieval range) on the performance of the baseline retrieval heuristics and of our neural context retriever. we release our synthetic dataset and our code under a free license 1 . ## method in this section, we start by introducing the document-level context retrieval task for ner. then, since no dataset exists for this task, we detail how we generate a synthetic context retrieval dataset using an instruction-following llm in section 3.2. we describe the neural context retriever that we train using this dataset in section 3.3. figure 1 provides an overview of the process we propose to retrieve context with our neural retriever. finally, we provide details on our dataset in section 3.4. description (all classes) prompt template '{input sentence}' -in the preceding sentence, {entity} is a character. invent a one-sentence description for this character, mentioning their name. example input sentence [one-eye]'s handicap in no way impairs his marvelous hindsight. example generated context one-eye is a wise and mysterious character with a penchant for coming up with invaluable insights after the fact. action (per only) prompt template invent a single sentence depicting the character '{entity}' performing an action, mentioning their name. example input sentence "it's my stomach, [croaker]," example generated context croaker was whistling a jaunty tune as he strolled through the park. movement (loc only) prompt template invent a single sentence depicting a character of your invention going to {entity}. you must mention the name of the character. example input sentence lightning from a clear sky smote the [necropolitan hill]. example generated context the gothic vampire count necropolis ventured to necropolitan hill, his ancient stomping grounds. ## conclusion in this article, we proposed to train a neural context retriever tailored for ner on a synthetic context retrieval dataset. our retrieval system can be used to enhance the performance of a ner model, achieving a gain of around 1 f1 point over a raw ner model and beating unsupervised retrieval heuristics. we also demonstrated results on par or better than re-rankers trained using manually annotated data. as stated in section 7, a limitation of our neural context retriever is that the relevance of each candidate context sentence is estimated separately. to account for potential interactions, we could iteratively add each candidate to the input sentence and rank the remaining candidates with respect to that newly formed input. future works may expand on this article by exploring whether taking these interactions into account can have a positive impact on performance. we also note that our proposed method, generating a retrieval dataset using an instructionstuned llm, could be leveraged for other nlp tasks where global document-level retrieval is useful. the only pre needed to generate such a dataset is the knowledge of which types of context samples can help to improve the performance for a local task, in order to generate positive samples. future works may therefore focus on applying this principle on different tasks.