NLP eople at S em E val-2023 Task 2: A Staged Approach for Multilingual Named Entity Recognition (2023)

Abstract: The MultiCoNER II shared task aims at detecting complex, ambiguous named entities with fine-grained types in a low context setting. Previous winning systems incorporated external knowledge bases to retrieve helpful contexts. In our submission we additionally propose splitting the NER task into two stages, a Span Extraction Step, and an Entity Classification step. Our results show that the former does not suffer from the low context setting comparably, and in so leading to a higher overall performance for an external KB-assisted system. We achieve 3rd place on the multilingual track and an average of 6th place overall.

## introduction the 2022 multiconer shared task @xcite aimed at addressing the challenges present in processing semantically-ambiguous and complex named entities (ne) in short and lowcontext settings. the top-performing systems made use of existing knowledge in the form of knowledge-bases @xcite and gazeteers @xcite . the organisers of the mul-ticoner task noted, however, that such solutions are not robust against nes which are not present in the existing knowledge or are noisy 1 . such findings are the key motivators behind the 2023 multi-coner shared task @xcite , with a far more fine-grained set of classes and noise introduced into some additional datasets @xcite . we propose splitting the standard named entity recognition (ner) task into two stages: (1) span extraction and (2) entity classification. the reason being that span extraction is closely tied to language and thus the noisiness of the entities or their existence in the knowledge, should not greatly 1 examples of noisy named entities are ones with spelling mistakes and typos. affect its results. following which, the classification of the extracted spans should be simpler, as the spans will have already been extracted. furthermore, we make use of different types of additional contexts based off of the wikipedia dumps for each language @xcite . the details of which can be found in sub-section 3.2. ## related work recent approaches to ner can generally be split into three groups 1. crf-based, 2. linear, and 3. span enumeration. crf-based approaches such as @xcite and @xcite use a bidirectional lstm with a conditional random field (crf) as a sequence tagger, relying on one or a combination of word embeddings, character representations, and transformer representations as input features to the combined lstm-crf decoder. linear approaches such as luoma and pyysalo (2020) and schweter and akbik (2021) rely on finetuning a pretrained transformer architecture with a linear layer to greedily decode the output. the representations passed onto the linear layer can also include word embeddings such as in @xcite . as an alternative to the greedy approach, luoma and pyysalo (2020) combines scores for each sentence from different positions in the input context. finally, approaches such as @xcite ) classify all possible spans within a sentence. these spans are then ranked by their highest score and each span is included in the final output if it does not overlap with a higher scoring span that is already included in the final output. regardless of the method used, most state-ofthe-art models use either a large transformer representation and/or a combination of sources for their word embeddings. @xcite use elmo, bert, and flair embeddings, while @xcite use both word and character embeddings, and @xcite use bert, fast-text, and character embeddings. @xcite propose training a controller to automatically concatenate embeddings from different sources based on the current context. finally, schweter and akbik (2021) rely on xlm-r embeddings and word embeddings from different sources depending on the target language. @xcite on the other hand, train a transformer model that produces contextualised embeddings for words and entities in a sentence, independent of each other. this entity-aware transformer is then tuned on a span enumeration task as described above. from the 2022 multiconer shared task, the two winning systems used distinct strategies. @xcite relied on large-scale retrieval of relevant paragraphs to the target sentence, which were then concatenated and used as input to a transformer-crf system. @xcite on the other hand use a gazetteer-augmented-bilstm model together with a transformer model to classify target sentences. the bilstm is pre-trained to produce token embeddings similar to the accompanying transformer when given sequence labels based on gazetteer matches. ## experimental setup implementation details we implement our models in pytorch and use pretrained models provided through the huggingface transformers library. we demonstrate our approach using xlm-roberta(xlm-r) foot_1 @xcite . training & hyperparameters we ran a limited grid search similar to @xcite to obtain the best hyperparameters for each model. the settings explored were learning @xmath0, 2 × 10 -5 }, batch @xmath1, 16}. the loss function used for both stage models was a softmax cross-entropy. ## results and analysis we present the results of our approach on the development sets in tables 1 & 2 . in order to get an accurate view of how well each stage is performing, we calculate f1 scores both with and without tags. we call this labelled f1 and unlabelled f1 respectively. in both cases we calculate span f1, where an entity is correct if the start and end of the span are correct, and for labelled f1 the assigned tag must also be correct. following the organisers we present macro-f1 scores for our labelled f1 results. the results of the span extraction stage is shown in additionally, while matched title and liked page title contexts don't clearly outperform each other across languages, the combined context beats both on 11 of the 13 languages, with a negligible drop in performance on the remaining two. the results of our entity classification stage are shown in our models outperform the weaker crf baseline across the board for all languages, including all individual models, with the exception of our matched titles model for hindi. additionally, our final pooled models outperformed the stronger raner baseline on 7 of 12 languages (multilingual baseline scores were not available). while the matched title and linked page title models beat the raner baseline on 6 of 12 languages, they are generally outperformed by the infobox and paragraphs models. the improved performance of the infobox model is a result of the class specific properties present in the infobox of the matched entity. for example albert einstein's infobox has keys such as born, citizenship, and education which are all clearly properties of a person and thus make labelling albert einstein as person a simpler task. similarly, the improved performance of the paragraph model is a consequence of the more complete context provided by the first paragraph of the matched entity. for example, the wikipedia page of j. robert oppenheimer says that they are a professor of physics at the university of california, berkeley where physics is a hyperlink and will be present in the additional contexts for matched title and linked page title models but what is actually important in that sentence is that they are a professor of physics, indicating that they are a scientist. paragraph-based models were generally the best performing models, beating the other contexts on 9 our final results are shown in the entity classification step also generalised fairly well from the dev. set to the clean test set, with macro f1 scores being 1-6% lower than on the dev. set, with the exception of bengali and hindi which fell by 10%. our approach is strongly impacted by the introduced noise for all languages. the contrast between our results on noisy and clean data points to the importance of accurate span extraction, which acts as an upper bound to the performance of the entity classification step. additionally, the results of span extraction were less impacted overall by the introduction of noise than entity classification. this is possibly due to the reliance of our entity classification on entity matching, while span extraction relies primarily on linguistic patterns, with additional context being supplementary. ## conclusion in this work we demonstrated a staged approach to ner, where we learn to extract spans before classifying entities. our approach achieves strong results on span extraction and is resilient to noise, while the entity classification stage is far more sensitive and dependant on the retrieved context. we achieve 3 rd place on the multilingual track and an average of 6 th place overall.