IXA /Cogcomp at S em E val-2023 Task 2: Context-enriched Multilingual Named Entity Recognition Using Knowledge Bases (2023)

Abstract: Named Entity Recognition (NER) is a core natural language processing task in which pre-trained language models have shown remarkable performance. However, standard benchmarks like CoNLL 2003 do not address many of the challenges that deployed NER systems face, such as having to classify emerging or complex entities in a fine-grained way. In this paper we present a novel NER cascade approach comprising three steps: first, identifying candidate entities in the input sentence; second, linking the each candidate to an existing knowledge base; third, predicting the fine-grained category for each entity candidate. We empirically demonstrate the significance of external knowledge bases in accurately classifying fine-grained and emerging entities. Our system exhibits robust performance in the MultiCoNER2 shared task, even in the low-resource language setting where we leverage knowledge bases of high-resource languages.

## introduction the research on the named entity recognition field has mainly focused on recognizing entities from news corpora. datasets like conll 2003 @xcite or ontonotes @xcite have been the standard benchmarks for a long time. it is not difficult to find systems that perform over 92% of f1 score on those datasets. however, it is known that actual models perform pretty well on relatively easy entity types (such as person names) but have been struggling to extract more complex entities @xcite . moreover, the evaluation setup usually involves a large overlap on the entities from the train and test sets, overestimating the systems performance. removing such overlap resulted on a large performance drop, consequently showing that models tend to memorize rather than generalize to complex or unseen entities @xcite . motivated by these insights, recent works have developed new and more complex datasets like multiconer @xcite or ufet @xcite that involve complex entity mentions with higher granularity on the type definition. to address these new challenges large language models (llms) are commonly used, with the hope that after being trained on billions of tokens the models become aware of the relevant entities. despite the capacity of the models to memorize information about each individual entity, they face a harsh reality: the knowledge that a model can memorize is limited by the date on which the model was pre-trained. so, new emerging entities can be difficult to categorize for models trained years ago. moreover, there are cases when it is not possible to identify the fine-grained category of an entity just from the context, and thus requires prior knowledge about the entity. for example: george bernard shaw, douglas fairbanks, mary pickford and margaret thatcher are some of the famous guests who stayed at the hotel.. from the previous sentence, it is impossible to categorize george bernard shaw and margaret thatcher as politicians neither douglas fairbanks and mary pickford as artist. we hypothesize that a possible solution for this problem is to allow the models access updated knowledge bases, and use the updated information to infer the correct type of the entity of interest. to overcome the mentioned challenges, in this work we present a ner approach (see figure 1 ) that (1) identifies possible entity candidates by analyzing the input sentence structure, (2) links the candidate to an existing updated knowledge base if possible, and (3) performs the fine-grained classification using the input sentence plus the retrieved information from the kb about the entity. the approach allowed us to perform fine-grained ner with updated world knowledge when possible, and standard ner otherwise. our code is publicly available to facilitate the reproducibility of the results and its use in future research foot_0 . ## related work named entity recognition (ner) @xcite is a core natural language processing task where the goal is to identify entities belonging to predefined semantic types such as organizations, products and locations. since its inception, different approaches have been developed, from statistical machine learning methods @xcite @xcite to the ones based on neural networks @xcite and the mix of both @xcite . recently, the use of richer contextual embeddings computed via transformer models @xcite @xcite have considerably improved the state-of-the-art of the task. nevertheless, these models still have problems detecting and labelling unseen or complex entities @xcite . many datasets @xcite and methods @xcite @xcite have been developed as a result of the difficulty to label complex entities. the multiconer @xcite shared task focuses on detecting semantically ambiguous and complex entities in low-context settings. almost every participant of the task used transformer based models as their backbone model, xlm-roberta @xcite being the most popular. the best two models presented for the task @xcite show that the use of external kbs, such as wikipedia and wikidata, improve their results significantly. meanwhile, other best performing models @xcite @xcite take advantage of different data augmentation methods without relying on external kbs. the gap between the performance of the first two methods was less noticeable than the gap between the second best and the rest of approaches, implying that exploiting external kbs during training and inference is a promising research line. this difference was greater when more complex entities have to be identified. however, the overall results showed that recognizing complex or infrequent entities is still difficult @xcite . in this work we focus on the multiconer2 (fetahu et al., 2023a) dataset, a second version of multiconer @xcite with higher granularity than its predecessor. comparing to the state-of-the-art of the first multiconer task, our approach for this task also uses xlm-roberta for entity boundary detection and entity classification, but differs on how we retrieve information from the external kbs. instead of retrieving relevant text passages to classify a given entity from a query sentence @xcite , or building a gazetteer network from wikidata entities and their derived labels to further enrich token representations @xcite , we directly predict wikidata ids from entity candidates by integrating mgenre (de cao et al., 2022a) as our entity linking module. previous models have already tried to link entities with external kbs @xcite @xcite . however, mgenre allows us to easily access both wikidata entity descriptors and wikipedia articles to increase our candidate's context for entity classification. ## system description our method implements three steps. first we identify entity candidates in the input sentence. then, we link the candidate to an existing knowledge base. finally, we predict the fine-grained category for each entity candidate. ## experimental setup and dataset we use the official multiconer2 dataset @xcite in all tracks to train our entity boundary and entity classification models. to query external knowledge, we utilize the medi-awiki api to retrieve data from the live version of wikipedia/wikidata. multiconer2 consists of two substacks, the monolingual task with 12 monolingual datasets and a multilingual task that contains data from all languages. for 7 languages (en, zh, it, es, fr, pt, sv) some sentences were corrupted with noise, either on context tokens or entity tokens. the results of the shared task are evaluated based on entity-level macro f1 scores, where all label categories contribute equally to the final score, regardless of the number of entities labeled with each category. the dataset consists of 36 pre-defined fine-grained categories which are grouped into 6 coarse-grained categories: medical entities, locations, creative works, groups, persons and products. a description of the hyperparameters used, dataset stats and hardware used in available in the appendix. ## results our team participated in all the 12 monolingual tracks and the multilingual track. more than 35 team participated in the shared task. we achieved the 3rd positions in four tracks. the shared task results computed by the organizers are shown in we also compare our system with the performance of systems from others teams that got the first position in any of the tracks. our system was most competitive in hindi and bangla, as well as the multilingual track. hindi and bangla are by far the languages with the smallest wikipedia's, ranked 59th and 63rd by the number of articles respectively. 2 we attribute this to our decision to retrieve the external knowledge from the english wikipedia/wikidata for all the tracks. because we predict a wikidata id, we are able to retrieve knowledge in any language in which it is available. this enables our system to be used for low-resource languages for which no large, up-to-date knowledge base is available, thanks to being able to link the entities found to a knowledge base in high-resource languages. on the other hand, our system is not as competitive in very high-resource languages, such as english and chinese. ## conclusion we have developed a system that identifies potential entity candidates and leverages external knowledge from an up-to-date knowledge base to classify them into a set of predefined fine-grained categories, addressing the challenges posed by temporal knowledge and unknown entities. our results demonstrate exceptional performance in classifying entities into fine-grained categories, un-derscoring the need for external knowledge in accurately classifying entities in the multiconer dataset. however, our current entity boundary detection step does not incorporate external knowledge, which we plan to improve in the future. furthermore, our system exhibits promising results for low-resource languages where a comprehensive wikipedia is unavailable. by linking the identified entities to a knowledge base in a highresource language, it can be used to process lowresource languages. in future work, we intend to integrate all the steps and insights gained from our current system into a single end-to-end model. although each step performs well when evaluated independently, the current pipeline multiplies errors at each step, which we aim to address in the integrated model. epoch and then select the best performing checkpoint. we train five independent models and then use majority vote as the ensembling strategy at inference time.