Exploring the Use of Foundation Models for Named Entity Recognition and Lemmatization Tasks in S lavic Languages (2023)

Abstract: This paper describes Adam Mickiewicz University’s (AMU) solution for the 4th Shared Task on SlavNER. The task involves the identification, categorization, and lemmatization of named entities in Slavic languages. Our approach involved exploring the use of foundation models for these tasks. In particular, we used models based on the popular BERT and T5 model architectures. Additionally, we used external datasets to further improve the quality of our models. Our solution obtained promising results, achieving high metrics scores in both tasks. We describe our approach and the results of our experiments in detail, showing that the method is effective for NER and lemmatization in Slavic languages. Additionally, our models for lemmatization will be available at: https://huggingface.co/amu-cai .

## introduction named entity recognition and lemmatization are important tasks in natural language processing. fine-tuning pre-trained neural language models has become a popular approach to achieve the best results in these tasks. however, the performance of this method can vary across languages and language families. in this paper, we investigate the performance of fine-tuned, language-specific neural language models in named entity recognition and lemmatization in a set of slavic languages and compare them with multilingual solutions. we describe adam mickiewicz university's (amu) solution for the 4th shared task on slavner, which is a part of the 9th workshop on slavic natural language processing (slavic nlp 2023). our solution is based on foundation models @xcite . in particular, we used models based on the popular bert and t5 model architectures. to increase the effectiveness of our approach, we conducted experiments with different versions of monolingual and multilingual models, investigating the potential benefits of each model variant for specific tasks. the data provided by the organizers and external resources used for named entity recognition and lemmatization were processed and prepared as described in section 2. specific details regarding the approach are further discussed in section 3. in order to evaluate the effectiveness of our method, we performed several experiments on the previous shared task edition test set. this particular set was chosen because it is a well-known benchmark for named entity recognition and lemmatization in slavic languages. the results of our experiments are described in section 4. ## data this section provides a brief description of the datasets used in our solution. in addition to the data released by the organizers, we also used external datasets for named entity recognition and lemmatization. all training and validation samples containing named entities were converted to a conll-2003 dataset format @xcite . ## approach we participated in the two subtasks of the multilingual named entity recognition task -named entity mention detection and classification and named entity lemmatization. the solution involved fine-tuning the foundation models using task-specific modifications and additional training data. all models used in the experiments can be found on the hugging face hub foot_0 . ## conclusions we described the adam mickiewicz university's (amu) participation in the 4th shared task on slavner for named entity recognition and lemmatization tasks. our experiments encompassed various foundation models, including monolingual and multilingual bert and t5 models. we found that incorporating a crf layer enhanced the quality of our named entity recognition models. additionally, our results indicate that the use of t5 models for lemmatization yields high-quality lemmatization of named entities. we will release the lemmatization models to the community and make them available at: https://huggingface.co/amu-cai .