IITD at S em E val-2023 Task 2: A Multi-Stage Information Retrieval Approach for Fine-Grained Named Entity Recognition (2023)

Abstract: MultiCoNER-II is a fine-grained Named Entity Recognition (NER) task that aims to identify ambiguous and complex named entities in multiple languages, with a small amount of contextual information available. To address this task, we propose a multi-stage information retrieval (IR) pipeline that improves the performance of language models for fine-grained NER. Our approach involves leveraging a combination of a BM25-based IR model and a language model to retrieve relevant passages from a corpus. These passages are then used to train a model that utilizes a weighted average of losses. The prediction is generated by a decoder stack that includes a projection layer and conditional random field. To demonstrate the effectiveness of our approach, we participated in the English track of the MultiCoNER-II competition. Our approach yielded promising results, which we validated through detailed analysis.

## introduction named entity recognition (ner) is a crucial task in natural language processing (nlp), where the goal is to identify and classify named entities in text, such as people, organizations, locations, and products. ner has a wide range of applications, including information extraction, question answering, and sentiment analysis, among others. recently, there has been a growing interest in the development of ner models that can handle ambiguous, complex, and emerging named entities with shorter context. in response to this, the multi-coner challenge was introduced, which aims to evaluate ner models on a diverse range of named entity classes and contexts. the first iteration of the multiconer challenge, multiconer-i, focused on classification into six classes: per, loc, grp, corp, prod, cw @xcite . however, the performance of ner systems on complex entities such as creative works, groups, and products was found to be lesser than on entities such as person @xcite . in response to this, the second iteration of the multiconer challenge, multiconer-ii, was introduced, which aimed to address this issue by providing a fine-grained ner task with 36 different classes @xcite . to make the task more realistic, the organizers added noise to the test dataset. additionally, a new coarse-grained class, medical, was introduced in multiconer-ii, and the corp class was subsumed into grp. the coarse-grained classes were further subdivided into fine-grained entities. multiconer-ii consists of 13 different tracks, including 11 monolingual tracks for languages such as english, spanish, dutch, russian, turkish, korean, farsi, german, chinese, hindi, and bangla, as well as a code-mixed track and a multilingual track @xcite . to tackle the multiconer-ii challenge, we propose an ner system based on the concepts of a single-stage information retrieval (ir) framework. our information retrieval pipeline uses two document corpora, kilt and ms-marco, to retrieve relevant documents. we rely on the bm25 scoring method for document retrieval, and retrieve documents are chunked into small passages so that we can capture the relevant part of the document. to increase the confidence of the prediction, we have used an ensemble of focal loss and conditional random field (crf) loss. our system also incorporates additional contextual information into the sentences to improve the performance of lowperforming classes. through experiments and evaluations, we demonstrate the effectiveness of our approach in handling the multiconer-ii challenge. ## related work named entity recognition is one of the core tasks of nlp. it powers different real-world applications such as chatbots, and document parsers to identify and extract key information. ner is applied in different domains e.g. medical @xcite , ecommerce @xcite and sentiment analysis @xcite to name a few. in the multiconer-i, the dataset for the task has sentences comprised of complex and emerging entities with smaller context @xcite . challenges associated with the low-context ner and complex entities are outlined by @xcite . they have suggested an approach to use the gazetteer-based information for ner. document level context all improves the performance of ner system @xcite @xcite transformer-based models are the cornerstone of ner. to effectively use transformer-based models, researchers have employed transfer learning. in multiconer-i, teams have relied on transformer @xcite based models -bert @xcite , luke @xcite and roberta @xcite as their base encoders. in order to enrich low-context sentences with contextual information, participating teams has relied on knowledge base @xcite , gazetteer @xcite based network to infuse additional information to the model. almost all the participating teams beat the official baseline model. however, ner systems have produced a lower score on complex entities like cw, and grp @xcite . it shows the complexity associated with the task. ## data multiconer-ii is a named entity recognition (ner) task that focuses on fine-grained classification of 36 different classes. the dataset used for this task is multiconer v2, which was introduced in a paper by fetahu et al.. the dataset follows the iob format for annotating entities, and the distribution of instances for english is presented in number of instances dev set 871 train set 16778 test set 249980 in addition to the multiconer v2 dataset, we have used two other datasets, ms-marco (payal bajaj, 2016) and kilt @xcite , for our information retrieval pipeline. ms-marco is a dataset of search queries and passages, and the we use the 2019-20 dump of ms-marco for our system. kilt is a benchmark for knowledge-intensive nlp tasks, and the we use it for our information retrieval pipeline. ## system description the goal of our system is to use the external context that is retrieved through an information retrieval pipeline. to describe our paper, we will formally define our information retrieval (ir) pipeline and named entity recognition (ner) system. we will then provide details about the retrieval framework and ner system. our system relies on the ir pipeline to provide a list of the most relevant passages for a given query. specifically, given a query q, the system retrieves top-k documents {d 1 , d 2 , ..., d k }. each document is then segmented into passages of length l and ranked in terms of their relevance to the query q. the ir system generates a ranked list of passages {p1, p2, ..., pr} for each query q. ner tagging involves predicting one of the tags for each of the tokens in a given sentence s, with words {w 1 , w 2 , ..., w n } and set of tags {t 1 , t 2 , ...t n }. in our model, the query q and the top-n {p1, p2, ..., pn} passages are passed through a transformer model to generate contextual embeddings for the query. the contextual embeddings, along with an appropriate token mask, are then passed to a crf layer @xcite to learn transition probabilities for a bio tagging scheme. the schematic diagram of our system is shown in figure -2 . ## conclusion this paper presents an information retrieval based pipeline for fine-grained ner. the training data has a class imbalance issue. in order to handle the class imbalance, we have introduced focal loss. the model has generated a good performance on clean subset of test data. however, it was not able to produce similar performance on noisy data. the ablation study points out that the model was able to leverage additional context. it also points out that focal loss has enhanced the performance of the model. the use of focal loss can be explored further with masking of context tokens. the retrieval pipeline can be explored further with a query reformulation step to handle the short context and noise in sentences.