Silp_nlp at S em E val-2023 Task 2: Cross-lingual Knowledge Transfer for Mono-lingual Learning (2023)

Abstract: Our team silp_nlp participated in SemEval2023 Task 2: MultiCoNER II. Our work made systems for 11 mono-lingual tracks. For leveraging the advantage of all track knowledge we chose transformer-based pretrained models, which have strong cross-lingual transferability. Hence our model trained in two stages, the first stage for multi-lingual learning from all tracks and the second for fine-tuning individual tracks. Our work highlights that the knowledge of all tracks can be transferred to an individual track if the baseline language model has crosslingual features. Our system positioned itself in the top 10 for 4 tracks by scoring 0.7432 macro F1 score for the Hindi track ( 7th rank ) and 0.7322 macro F1 score for the Bangla track ( 9th rank ).

## introduction multilingual complex named entity recognition) @xcite targets the recognition of complex named entities in multilingual and multidomain texts, encouraging researchers to develop new approaches to extract diversified entity types. the semantic structure of data @xcite , which is also provided by organizers for training and validation and testing purposes, has six types of coarse-grained entities: person, location, group, product, medical and creative work with more than 30 fine-grained entities of these 6 coarse-grained entities. the dataset was provided for 12 mono-lingual tracks, and, in addition, this task also provided multi-language data encouraging the development of more generic and adaptive systems. ner task is used in many other nlp tasks. our team participated in 11 mono-lingual tracks ( hindi, bangla, english, german, spanish, french, italian, portuguese, swedish, ukrainian, and chinese ). our system scores 7th rank in hindi and is secure in the top 10 in 4 languages @xcite . our system leveraged the advantage of crosslingual learning through multi-stage training similar to @xcite and @xcite . for this in the first stage, we have selected crosslingual pretrained models which cover a group of languages of our task and we fine-tune these pretrained language models for a combined dataset of all tracks in which languages are covered by these language models for 10 epoch. it creates a model checkpoint which has knowledge of multiple languages for our task. in the second stage, we fine-tuned the model checkpoints which are generated in the first stage for each track individually. for the hindi and bangla tracks our best model was achieved by muril large model @xcite which is combinedly fine-tuned for three tracks (english, bangla, hindi) in the first stage and thereafter the model checkpoints fine-tuned for hindi and bangla. for the best model for all other languages, we fine-tuned xlm-r @xcite for all the tracks combinedly in the first stage and in the second stage fine tuned all mono-lingual tracks. in addition to multilingual, there were two key challenges in this year's task, fine-grained entity taxonomy with over 30 different classes, and noisy testing data which is generated by adding simulated errors to the test set. our best prediction model is available here foot_0 . ## related work transformer-based language models like bert @xcite shows the competitive result for several nlp tasks like ner but the problem of low-context situations, long-tail entities, emerging entities and complex entities outlined by the @xcite hard to handle by these models since these models generate contextual word embeddings and pretrained on existing data only. the problem of emerging entities for web queries in multi-lingual code-mixed context addressed by @xcite . these challenges also occur in @xcite and @xcite . @xcite and @xcite integrated a gazetteer with language models as a solution for these challenges. there are some challenges with the use of a gazetteer, one of them is gazetteer should cover the entities in a sufficient ratio and also for new domain and entity types a specific gazetteer is required, which is subject to the availability of the gazetteer. another challenge described in @xcite is weak learning of the model due to the model knowing the entity type in advance for the entities, which are covered by the gazetteer. our work is the continuation of the multilingual ner task started in 2022 @xcite , which has only 6 entities present for 11 monolingual tracks with the multi-language and codemixed track @xcite . the challenge of @xcite was to predict complex entities like creative works with low context for the multi-lingual track. one of the leading approaches to generate a system for @xcite the problem is based on a knowledge-based system @xcite for lower context examples by augmentation of text from wikipedia. @xcite also performed well for all tracks by integrating language models like bert @xcite with a gazetteer network by minimizing the kl divergence. @xcite and @xcite apply multi-stage training for leveraging the advantage of multi-lingual tracks. @xcite also used ensemble architecture and data augmentation for making a more robust model for noisy environments. @xcite fine tuned muril-large and achieve 0.69 and 0.59 f1-scores for hindi and bangla tracks. this shared task provide 11 mono-lingual track and 1 multi-lingual track @xcite to encourage nlp researchers to develop ner systems. all tracks have split training, development and testing data however testing data is provided after the final ranking is done. the size of training, development and testing data of all tracks are tabulated in along with complex entities like creative works, there are 6 coarse-grained entities and more than 30 fine-grained entities are there for all tracks, which are challenging as they are harder to recognize. organizers @xcite provided around 9700 sentences only for training for hindi, bangla, chinese and german tracks, this made these tracks more challenging than other tracks as for other mono-lingual tracks around 16500 training data were available. ## methodology all track data are divided into training and validation data by the task organizer. similar to @xcite and @xcite our system selected strong cross-lingual transfer models ( xlm-r @xcite , pre-trained on more than 100 languages and muril @xcite , pre-trained on all indic and english languages ) for two-stage training as the base model. in the first stage, we performed multi-lingual training in which training data of all tracks were utilized in selected language models. figure 1 showed that in the first stage, our model utilized the annotations of all the tracks. this experiment was done with different hyperparameters for 10 epochs and the best multi-lingual checkpoint was selected based on average loss on validation data. thereafter in the second stage, we use the fine-tuned multilingual checkpoint of the first stage as the initial model for fine-tuning each mono-lingual track. in the second stage, training of each mono-lingual track is performed with different hyperparameters and we select the final model based on the loss of validation data of the corresponding track. as shown in figure 1 , first of all, the model tokenizer tokenizes the input sentence further preprocessing is done like @xcite . model_max_length of tokenizers set to 92 for better gpu utilization. thereafter language model generates word embeddings for each token. a linear layer is applied on each token embedding that projects this embedding into logits, a vector of 67 dimensions since the total unique annotations in the dataset are 67. further, for classification, we have used the cross-entropy loss function. ## experimental setup our best score was achieved with muril setup for the hindi and bangla track and the other track's best score was achieved with the xlm-r setup. detail about both the setup defined in the next subsection. implementation of our task with the cross entropy classification layer is done by xxxtokenclassificaion class defined in @xcite , where xxx refers to the selected model. ## results and analysis our best macro f1 score with official rank for 11 tracks is tabulated in achieved by xlm-r setup. ## conclusion in this paper, we performed multi-lingual track knowledge utilization for mono-lingual track for the multiconer ii shared task. our team achieved the top 10 on 4 tracks. we show that the performance of monolingual models can be enhanced by two-stage fine-tuning by allowing them to learn from the training data of all the languages. also, we have compared two cross-lingual language models, muril and xlm-r for multi-lingual knowledge transferability for the hindi, bangla and english track of this task. muril performed better for hindi and bangla tracks although xlm-r performed better for the english track. for future work, we plan to explore more language models for utilizing their knowledge for ner and other nlp tasks.