MEERQAT - IRIT at S em E val-2023 Task 2: Leveraging Contextualized Tag Descriptors for Multilingual Named Entity Recognition (2023)

Abstract: This paper describes the system we submitted to the SemEval 2023 Task 2 Multilingual Complex Named Entity Recognition (MultiCoNER II) in four monolingual tracks (English, Spanish, French, and Portuguese). Considering the low context setting and the fine-grained taxonomy presented in this task, we propose a system that leverages the language model representations using hand-crafted tag descriptors. We explored how integrating the contextualized representations of tag descriptors with a language model can help improve the model performance for this task. We performed our evaluations on the development and test sets used in the task for the Practice Phase and the Evaluation Phase respectively.

## introduction named entity recognition (ner) @xcite consists in detecting groups of words as named entities from a given sentence and recognizing their type from an available list of entity tags. according to the entity tags list size, the ner task is classified as i) coarse-grained when the list is small (such as names of people, organizations, and locations, as proposed in the conll task @xcite ) or ii) fine-grained for a more extensive list @xcite . additionally, we can distinguish different named entity types, referenced as traditional (person, location) and non-traditional (titles of creative work, such as a book or a song). nontraditional entities, also called complex entities, are difficult to recognize due to problems such as semantic ambiguity @xcite . the semeval 2023 task 2 multilingual complex named entity recognition (multiconer ii) @xcite evaluates ner systems capable of identifying semantically complex and ambiguous named entities in 12 languages (english, spanish, hindi, bangla, chinese, swedish, farsi, french, italian, portuguese, ukrainian, and german) providing multilingual resources to this end. additionally, this task version proposes a finegrained entity taxonomy with more than 30 different entity tags and a complex test set composed of low-context sentences divided into two subsets, named "noisy" and "clean". the "noisy" subset (available for eight languages) includes corrupted sentences that add noise on context tokens or entity tokens to simulate errors. multiconer ii proposes two types of evaluations: 12 monolingual tracks and one multilingual track. a task constraint is that multilingual models can not participate in the monolingual tracks, and similarly, the monolingual models can not participate in the multilingual track. also, a specific dataset is provided for each phase of the evaluation: we adopt hereafter the term development set for the practice phase and the term test set for the evaluation phase. previous work identified complex entities in lowcontext sentences by focusing on augmenting the input query using external resources, training a knowledge base retriever @xcite , or using data-augmentation techniques @xcite . most of these works concentrate on improving the contextualized representations of the query; however, less attention is given to the representations of the entity tags. we argue that contextualized representations of the entity tags can benefit the model's performance, particularly for a finegrained entity taxonomy where some classes are usually underrepresented. our intuition is to associate the input query entities with a clear definition of their entity tags, leading to an adapted representation space where entities sharing the same tag will have closer representations. to this end, we defined rich and relevant contextualized tag descriptors to leverage the model representations for this task. we define a tag descriptor as combining a textual definition of the tag and a hypernymy-based definition to capture the fine-grained taxonomic structure. we created hand-crafted definitions in multiple languages. our system includes two different setups for the training and the test. for the training setup, we finetuned a pre-trained language model (plm), xlm-roberta, computing contextualized representations for the query and the associated entity tag descriptors obtained from the golden annotations. then, we aligned and aggregated both types of representations. the final token representations are then fed into a linear-chain conditional random field (crf) layer @xcite for named entity prediction. for the test setup, we evaluate our model using only the improved plm and the crf layer. we tested our models in 4 monolingual tracks: english, spanish, french, and portuguese. our experiments showed that injecting this knowledge helps enhance the model performance and shows consistent gain w.r.t the task baseline. our systems obtained the rankings 21/34, 12/18, 13/17, and 14/17 for the english, spanish, french, and portuguese tracks, respectively. ## related work broadly known plms, such as bert @xcite and xlm-roberta @xcite , leveraged the performance on multiple ner benchmarks. this improvement has triggered the interest of the nlp research community by finding better setups to improve these models for the ner task. notably, we can observe a significant presence of this type of model in the semeval-2022 task 11: multilingual complex named entity recognition (multiconer) @xcite . the multi-coner task provided a multi-lingual dataset @xcite focused on detecting semantically ambiguous and complex entities in short and low-context settings, which is more complicated than recognizing traditional entities @xcite . based on this first version of the task, the multiconer ii task adds two more challenges by using a fine-grained entity taxonomy and simulating errors in the dataset. adding relevant context to the input phrase with external lexical resources, such as knowledge bases, helped to create improved token representations that positively impacted the performance of the multiconer task @xcite . similarly, previous works proposed pre-training enhanced-plms by fusing knowledge base representations with plms to improve token representations, which has been helpful in related nlp tasks @xcite . however, the main drawback of these approaches is the high computing cost needed to pre-train the models, mainly due to the additional network architectures they contain. another line of work explored the use of plms as knowledge bases to retrieve @xcite and inject facts @xcite of these external resources by translating them into textual utterances and applying to these utterances the pretraining task of the plm. based on these techniques, our approach considers a tag descriptor as the textual representation of a knowledge base fact. we aim to inject and retrieve this information helpfully into a plm under a light training setup. ## data the multiconer 2 dataset @xcite comprises a fine-grained tagset of 33 tags grouped into six coarse-level categories (location, creative work, group, person, product, and medical). the organizers provided 13 dedicated datasets for the 12 monolingual and multilingual tracks with train, valid, and test splits. each dataset follows the conll format, and the tag labels a bio scheme. we generated the tag descriptors using the following procedure. first, we identified each finegrained tag with its corresponding coarse-level class in taxonomy. then, we added definitions with two widely used taxonomies in the research community, wordnet @xcite and ene @xcite , as references. if multiple definitions were available, we selected the one whose hypernym matched the coarse-level name class. we used as a complementary resource the wikipedia category website when the category was found in none of the primary sources. we created the definitions initially in english. then, we used an automatic translation tool to obtain their corresponding spanish, french, and portuguese definitions. finally, we manually curated these translations to keep the orthographic correction and the contextual sense of the translation. category lang. tag descriptor medicine-symptom en a symptom is any sensation or change in bodily function that is experienced by a patient and is associated with a particular disease. a symptom is a medical term. es un síntoma es cualquier sensación o cambio en la función corporal que experimenta un paciente y que se asocia a una enfermedad concreta. un síntoma es un término médico. fr un symptôme est toute sensation ou modification d'une fonction corporelle ressentie par un patient et associée à une maladie particulière. un symptôme est un terme médical. pt um sintoma é qualquer sensação ou mudança na função corporal que é experimentada por um paciente e está associada a uma determinada doença. um sintoma é um termo médico. location-human settlement en a human settlement is community of people smaller than a town. a human settlement is a location. es un asentamiento humano es una comunidad de personas más pequeña que una ciudad. un asentamiento humano es un lugar. fr un établissement humain est une communauté de personnes plus petite qu'une ville. un établissement humain est un lieu. pt um assentamento humano é uma comunidade de pessoas menor do que uma cidade. um assentamento humano é um lugar. figure 1 to improve and adapt the model representations for the task but we rely only on the plm to perform the evaluation. we take as input a sentence composed of n tokens @xmath0, t 2 , ..., t n } for the pre-trained language model and tag-descriptor encoder. the tag-descriptor encoder first expands the input by concatenating the entity mention and the hand-crafted entity tag descriptors; then, it computes their contextualized representations. the tag-descriptor encoder provides such a representation for each entity identified in the input sentence. we align these subtokens representations with the output representation of the pre-trained language model and compute an aggregated representation using the information fusion module. the combined representations are passed to a crf layer that produces the label predictions. ## conclusion this paper describes our approach to improving token representation for the ner task. we evaluated the effectiveness of injecting context-relevant information about tag descriptors to improve the token representations of a plm. the results for the development and test sets showed important improvements over the baseline when considering tag context information. considering that the top-1 model requires extensive computation and textual resources for training and testing, we consider that our proposition, that only requires very limited additional knowledge, can be adopted as a first step towards developing better approaches for low-resource languages. in future work, we will evaluate how to extract better representations from the hierarchical fine-grained taxonomy and adapt this improved version of xlm-roberta as a base model to other approaches, such as the best-performing method of this evaluation.