gar NER at S em E val-2023: Simplified Knowledge Augmentation for Multilingual Complex Named Entity Recognition (2023)

Abstract: This paper presents our solution, garNER, to the SemEval-2023 MultiConer task. We propose a knowledge augmentation approach by directly querying entities from the Wikipedia API and appending the summaries of the entities to the input sentence. These entities are either retrieved from the labeled training set (Gold Entity) or from off-the-shelf entity taggers (Entity Extractor). Ensemble methods are then applied across multiple models to get the final prediction. Our analysis shows that the added contexts are beneficial only when such contexts are relevant to the target-named entities, but detrimental when the contexts are irrelevant.

## introduction multiconer is a shared task @xcite organized as part of the semeval workshop to assess the performance of named entity recognition (ner) systems across multiple languages. the task specifically addresses the recognition of complex named entities in a multilingual setting, such as product names, event names, and creative works. complex named entities often have form variations and may resemble normal language usage (eg. "catch me if you can"), making them difficult to recognize with traditional ner techniques. to accurately classify these entities, domain-specific knowledge and advanced language modelling techniques are often considered necessary components of the systems submitted to this task. the introduction of transformer-based models has resulted in significant advancements in various natural language processing (nlp) tasks, including named entity recognition (ner). the integration of pre-trained embeddings, trained on lengthy texts, has proven to be an effective technique for modelling long-range dependencies and resolving ambiguity in complex named entities within sentences @xcite @xcite . among the systems submitted in multiconer 2022, the winning system was damo-nlp @xcite , which ranked 1st in most languages @xcite . the idea behind their approach is that incorporating a knowledge base system grants ner models a similar advantage as professional annotators, allowing them to retrieve relevant documents and information from the knowledge base to disambiguate challenging samples in the dataset. the underlying principle of incorporating additional context into the original sentences is to enhance the contextual embeddings and thereby improve the contextual representations of the complex entities. in our revised method, we follow the same principle of extracting relevant information and employing long-range dependencies to establish robust token representations. however, a significant difference between our approach and that of the damo-nlp knowledge-based system lies in our proposal for knowledge enhancement without the requirement for a comprehensive knowledge base. the cost of building a knowledge base is significant, especially in a multilingual context, so we propose a streamlined approach in which, given an entity, we use the wikipedia api to gather a summary from relevant wikipedia pages, which is then concatenated with the input sentence. this approach considers the computational resource constraints while striving to maintain an equivalent level of performance for complex named entity recognition tasks. we fine-tune multilingual and monolingual transformer models and then apply ensembling techniques to obtain the final predictions. our best system results are consistently positioned in the middle of the rankings, and they rank in the top ten at ten language tracks. moreover, our error analysis indicates that the added contexts are only advantageous when they are relevant to the target-named entities, whereas they are disadvantageous when the additional context are irrelevant. ## related work pre-trained transformer-based models @xcite can leverage extensive sources of knowledge, such as wikipedia and social media, to achieve state-of-the-art (sota) performance on various nlp tasks. multilingual transformer like xlm-roberta @xcite and bert multilingual base (mbert) @xcite have undergone extensive training with vast amounts of data drawn from various languages and datasets. their transfer learning capabilities allow them to identify subtle nuances and patterns common across different languages, leading to superior performance compared to previous approaches. however, monolingual transformers trained on a large amount of data from a single language can sometimes outperform multilingual models in different nlu tasks, including named entity recognition (ner). this has been observed in languages like bangla @xcite , portuguese @xcite , and others. @xcite 's google search methodology for information retrieval has been expanded to include the use of pre-existing knowledge sources, such as wikipedia or dbpedia, which are widely used as general-purpose external knowledge resources across a wide range of natural language processing (nlp) tasks @xcite . it has also been widely used in named entity recognition (ner) tasks since they contain huge amounts of entities like their types, attributes, and relationships shown by damo-nlp @xcite . also, incorporating domainspecific knowledge such as dictionary features can improve ner performance @xcite . systematized nomenclature of medicine clinical terms (snomed-ct) @xcite , unified medical language system (umls) have already shown significant enhancement in the performance of ner as demonstrated by umlsbert @xcite . ensemble learning, which combines the strengths of multiple models, is a straightforward approach to address nonlinear relationships and minimize data noise. as a result, it demonstrates significant performance in named entity recognition (ner) (speck and ngonga ngomo, 2014), including prior multiconer tasks @xcite . in the recent multiconer task, several ensembling methods were employed, including merging pre-trained language encoders with data augmentation techniques based on translations of the original training data @xcite , multilingual models (such as xlm-roberta-large and microsoft/infoxlm-large) with a conditional random field (crf) layer @xcite , multilingual and monolingual transformer models @xcite , and transformer combined with reinforcement learning @xcite . these methods achieved significant performance improvements, demonstrating the effectiveness of ensembling in ner tasks. our study expands on recent developments in external knowledge integration and ensemble learning techniques. while knowledge augmentation has been shown to be highly effective in named entity recognition (ner) tasks, the resulting systems tend to be complex and computationally intensive. our work introduces a streamlined approach to knowledge augmentation that is combined with ensemble techniques to improve ner performance. ## dataset description the multiconer dataset has complex ner data with more syntactically challenging scenarios (malmasi et al., 2022a) for 12 languages: english (en), spanish (es), hindi (hi), bangla (bn), chinese (zh), swedish (sv), farsi (fa), french (fr), italian (it), portuguese (pt), ukrainian (uk) and german (de). it also includes a multilingual set, which combines the same monolingual examples from each of the 12 languages @xcite . this dataset involves identifying and classifying various types of named entities in text, such as location, creative work, group, person, product, medical, etc. it is a fine-grained dataset, which means it contains more specific and detailed labels than coarse-grained alternatives. the dataset employs a fine-grained named entity taxonomy, with 36 finegrained types and 6 coarse-grained types. based on various contexts and factors, it can distinguish between different types of named entities. for example, it can distinguish between "scientist" and "athlete", which can be difficult for coarse-grained ner systems. the multiconer dataset is made up of three main sources: wikipedia sentences (lowner), questions (msq-ner), and search queries (orcas-ner). the train set has roughly 16k instances, whereas the dev set has approximately 800 instances. while each of the de, hi, zh, and bn test sets has around 20k instances, each of the remaining languages' test sets has around 250k instances. en, de, it, fr, es, zh, pt, and sv have up to 30% of instances with additional noise. in this dataset, the noise consists of typographical errors caused by different keyboard layouts. the final system has been evaluated based on the performance of both noisy and clean dataset. ## experimental setup we present our approach to enhancing a named entity recognition (ner) model using fine-tuned multilingual and monolingual encoders based transformer models for a token classification task. to facilitate our ner model pipeline, we first tokenize each sentence and convert it into a fixedlength vector of 256 dimensions, which is then fed into the transformer model. the final classification layer generates a probability matrix of 256 × n to predict the class label for each token where n is the number of classes. we only consider the output from actual tokens to determine the loss, disregarding those generated from augmented sentences as we lack gold labels for these extra tokens. during fine-tuning, we utilize a learning rate of 1e -4 and train the model for 7 epochs. furthermore, we fine-tune three models for each language and transformer model, using three different datasets (e.g. no augmentation, entity extractor, and gold entity). while the fine-tuned models with the no augmentation and entity extractor datasets are integrated into our final system, we use the other fine-tuned models with the gold entity dataset for error analysis. during our experiments, we use the multiconer dev set to evaluate our models and split the train set into 80-20 split to create the training and validation sets for finetuning. ## results for our results, we compute macro f1-scores based on token entity predictions. comparison with other systems in we can certainly observe that the addition of noise has an impact on the model's performance, but it is encouraging to see that the drop in languages like fr, pt, and it is not below 60. however, the effects of noise on the zh track are notable as they are much closer to the ones from top participants like damo-nlp and ustc-nelslip foot_0 . moreover, our model shows greater robustness in some tracks (sv, zh, fr) compared with participants who outperform us with no noise 3 . despite this, the en track is significantly affected by the presence of noise. overall in ## discussion monolingual vs multilingual our multilingual models outperform the monolingual ones barring 4 tracks (en, sv, fa, bn) where the multilingual f1 scores are lower than their monolingual counterparts ( ## conclusion and future work in this paper we present our garner system that shows promising results for models trained with a gold entity dataset, which suggests that relevant context can boost performance in complex ner tasks. therefore, we consider that training a custom entity extractor model for our knowledge augmentation step can improve the current version of our system. in order to do so, a simpler dataset could be created by removing the fine-grained categories and transforming the tags from b-x to b and i-x to i. since the current model cannot detect the majority of tags, this is crucial and may improve overall performance. lastly, we consider that adding a conditional random field (crf), as it is common practice in token classification tasks, may be relevant because it avoids illegal span predictions. crf layers could potentially be added on these transformer models to further evaluate ner performance and see if there are any improvements. muril: the muril @xcite ) model is based on the bert @xcite architecture and was trained from scratch using various corpora, including wikipedia foot_3 , common crawl oscar, pmindia @xcite , and dakshina @xcite for 17 indian languages and their transliterated versions. masked language modeling (mlm) and translation language modeling (tlm) are two language modeling objectives that were used to train the model. italian bert: the opus corpora foot_4 collection of texts plus a recent wikipedia dump were used to train the italian bert (schweter, 2020) model, producing a final training corpus of 13 gb and 2,050,057,573 tokens. the same training data from opus plus additional data from the italian section of the oscar corpus foot_5 were used for the xxl italian models, resulting in a final training corpus size of 81gb and 13,138,379,147 tokens. swedish bert: the architecture of bert @xcite serves as the foundation for the swedish bert model @xcite . about 15 to 20 gb of text (200 million sentences and 3 billion tokens) from diverse sources are used to train the model (books, news, government publications, swedish wikipedia, and internet forums). german bert: the bert architecture @xcite , which was first suggested by google, was used to create the german bert language model @xcite . data was gathered from a variety of sources, including oscar @xcite , german-language wikipedia dumps, opus, open legal data foot_6 , and news items to train this model. albeto: albeto @xcite ) is a pre-trained albert model @xcite variation that has only been trained on spanish language corpora. it employs the weight-tied approach, which shares all parameters among the model's layers. there are five sizes available for the albeto models: tiny, base, large, xlarge, and xxlarge. sentencepiece @xcite was used to build a vocabulary of 31k lowercase tokens that was shared by all of these models