L3i at S em E val-2022 Task 11: Straightforward Additional Context for Multilingual Named Entity Recognition (2022)

Abstract: This paper summarizes the participation of the L3i laboratory of the University of La Rochelle in the SemEval-2022 Task 11, Multilingual Complex Named Entity Recognition (MultiCoNER). The task focuses on detecting semantically ambiguous and complex entities in short and low-context monolingual and multilingual settings. We argue that using a language-specific and a multilingual language model could improve the performance of multilingual and mixed NER. Also, we consider that using additional contexts from the training set could improve the performance of a NER on short texts. Thus, we propose a straightforward technique for generating additional contexts with and without the presence of entities. Our findings suggest that, in our internal experimental setup, this approach is promising. However, we ranked above average for the high-resource languages and lower than average for low-resource and multilingual models.

## introduction named entity recognition (ner) is the task of detecting entities and recognizing their type (e.g., person, location, organization) @xcite . standard benchmarks (e.g., @xcite ) for ner are either focused on a high-resource language or on a single domain (e.g., news). however, ner is required in different fields, bringing thus several new challenges regarding: the size of the textual content (e.g., short textual snippets like web queries @xcite or social media posts @xcite ), the characteristics of the entities to be extracted (i.e., capitalization/punctuation features @xcite ), unseen and emerging entities (e.g., entities that have a fast growth rate such as new songs and movies are released weekly (bernier-colborne and langlais, 2020)), complex entities (i.e., complex noun phrases such as particularly long nested person names and dates in historical documents @xcite ). moreover, multilingual documents in which entities of different languages than the rest of the text are present (i.e., code-mixed documents @xcite ) add another level of complexity to the task. transformer-based @xcite architectures for ner became popular since the release of the bert model and they hold the state of the art in ner @xcite @xcite @xcite . however, while most ner systems have been developed to generally address contemporary news data in high resource languages (e.g. english) @xcite @xcite , many challenges remain in ner from short texts with complex entities in low-resource scenarios @xcite . the semeval-2022 task 11, multilingual complex named entity recognition (multiconer) @xcite aims at developing complex ner systems for 11 languages. the task focuses on detecting semantically ambiguous and complex entities in short, lowercased, low-context monolingual, and multilingual settings. the languages were: english, spanish, dutch, russian, turkish, korean, farsi, german, chinese, hindi, and bangla. for some languages, an additional track with code-mixed data was offered. the organizers aim at testing the domain and language adaption capability of a ner system. for this reason, ner systems for monolingual settings are also constrained to avoid multilingual prediction models and vice versa. multilingual and code-mixed documents have motivated the development of systems based on multilingual transformer-based architectures @xcite . also, re-cently, @xcite proposed an approach to ner from short texts by finding external contexts of a sentence. the authors retrieved and selected a set of semantically relevant texts through a search engine, with the original sentence as the query, which proved to be efficient in detecting entities. in this paper, we propose a robust approach for ner and we tackle the following challenges for the language-specific sub-task, including the multilingual and mixed sub-tasks: (1) short texts: by adding multilingual contexts with and without entities, in order to improve and focus more on the context and less on the other surrounding entities, (2) lowercased texts: we prioritize the usage of uncased language models, and (3) code-mixed and low-resource languages: we add a multilingual language model along with a language-specific language model. ## data for each language, organizers provide fixed-sized training and development sets of 15, 300 and 800 sentences respectively, whereas test sets are composed of an average of 181, 418 (σ = 35, 181) sentences. a multilingual dataset consisting of the concatenation of all monolingual sets is also provided (with a selection of 471, 911 sentences for the test set) 1 . in addition, a smaller code-mixed dataset with entities in a foreign language of the rest of the sentence was given. in this case, 1, 500 sentences constitute the training set, 500 the development set, and 100, 000 the test set. we present a sample annotated example from the english training set in figure 1 . to highlight the important difference between the number of phrases in the train and the test sets, we present these numbers in ## methodology our proposed framework consists in augmenting each input sentence with similar contexts taken from the multilingual train collection and a ner model based on a bert-based pre-trained and finetuned language model as an encoder with several transformer @xcite layers stacked on top. for each language, we consider a languagespecific bert model, as presented in named entity recognition model our base model was recently proposed for coarse-grained and fine-grained named entity recognition @xcite . the method consists of a hierarchical approach, with a pre-trained and fine-tuned bert-based encoder @xcite . this model consists of a stack of transformer blocks language model english bigbird-roberta-large spanish bert-base-spanish-wwm-uncased dutch bert-base-dutch-cased russian rubert-base-cased turkish bert-base-turkish-uncased korean bert-kor-base farsi bert-fa-base-uncased-persiannews german bert-base-german-dbmdz-uncased chinese bert-base-chinese hindi bert-base-multilingual-uncased bangla bert-base-multilingual-uncased multilingual bert-base-multilingual-uncased mixed bert-base-multilingual-uncased on top of the bert encoder, and a conditional random field (crf) layer to decode the best tag path in all possible tag paths. first, the encoder is already based on a stack of transformer layers. a transformer block (encoder) is a deep learning architecture based on multi-head attention mechanisms with sinusoidal position embeddings. it is composed of a stack of identical layers. each layer has two sub-layers. the first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network. a residual connection is around each of the two sub-layers, followed by layer normalization. these language models generally expect the input data in a specific format: a special token, [sep], to mark the end of a sentence or the separation between two sentences, and [cls], at the beginning of the text. a transformer encodes the information contained in a given text in the [cls] token that flows afterward through the layers. this architecture with additional transformer layers proved to be efficient when the input is noisy and of varying lengths. adding context first, we encode the collection of training multilingual sentences using a pre-trained multilingual sentence bert model @xcite . we consider that using only sentences from the training set is enough for enriching the contexts, even if they are seen during the training process. the underlying idea behind this is that, since the data is multi-domain, we would expect a very small overlap between the train and the test set, thus the additional contexts could benefit from information from other domains that those existing. then, for each input sentence (for each dataset split), we rank them using the cosine similarity. from these retrieved texts, we select the first @xmath0, 10] sentences and we consider them as semantically relevant. next, as shown in the example in figure 2 with "street food" as a product (prod) entity, we concatenate the initial input sentence with the retrieved texts, separated by the special token @xcite tage of this additional context. first, we consider that simply concatenating the semantically relevant contexts as they are with the sentence in question are sufficient for bringing an improvement to the detection of entities. these models are referred to as context w/ entities. we also examine the scenario where these additional contexts are added without any entity present, as shown in figure 2 where the entities from the additional contexts are replaced with a special to-@xmath1. the idea behind this choice is that we are trying to improve the contextual information without confusing the detection of entities with the presence of others. we refer to these models as context w/o entities. for finding semantically relevant contexts, we used a multilingual sentence-bert (sbert) (reimers and gurevych, 2020b) model foot_1 , a modified pre-trained bert @xcite ) that uses a siamese and triplet network structure to derive semantically meaningful sentence embeddings that can be compared using cosine similarity. we keep the top-10 semantically relevant contexts. hyperparameters for each language, including the multilingual and the code-mixed cases, we choose a language-specific bert pre-trained model foot_2 , as presented in ## experiments next, we perform a detailed error analysis of our approaches foot_3 . the evaluation is performed in terms of macro precision (p), recall (r), and f1. our results are presented in ## conclusions in this paper, we presented a straightforward approach for adding semantically relevant contexts for ner in the monolingual, multilingual, and code-mixed datasets provided by semeval-2022 task 11 multilingual complex named entity recog-nition (multiconer). our findings show that, while adding contexts from the train set, with and without entities, is promising, the topics or domains overlap could influence the performance in both directions. future work will include the automatic generation of semantically relevant contexts without the presence of entities.