Cross-Lingual Named Entity Recognition for Low-Resource Languages: A H indi- N epali Case Study Using Multilingual BERT Models (2024)

Abstract: This study investigates the potential of cross-lingual transfer learning for Named Entity Recognition (NER) between Hindi and Nepali, two languages that, despite their linguistic similarities, face significant disparities in available resources. By leveraging multilingual BERT models, including RemBERT, BERT Multilingual, MuRIL, and DistilBERT Multilingual, the research examines whether pre-training them on a resource-rich language like Hindi can enhance NER performance in a resource-constrained language like Nepali and vice versa. The study conducts experiments in both monolingual and cross-lingual settings to evaluate the models’ effectiveness in transferring linguistic knowledge between the two languages. The findings reveal that while RemBERT and MuRIL perform well in monolingual contexts—RemBERT excelling in Hindi and MuRIL in Nepali—BERT Multilingual performs comparatively best in cross-lingual scenarios, in generalizing features across the languages. Although DistilBERT Multilingual demonstrates slightly lower performance in cross-lingual tasks, it balances efficiency with competitive results. The study underscores the importance of model selection based on linguistic and resource-specific contexts, highlighting that general-purpose models like BERT Multilingual are particularly well-suited for cross-lingual applications.

## introduction cross-lingual transfer learning has emerged as a crucial area in natural language processing (nlp), especially for languages with limited resources @xcite . this approach leverages the strengths of resource-rich languages to enhance model performance in underresourced languages, making it a valuable tool in the global effort to improve nlp applications across diverse linguistic contexts @xcite @xcite . in this context, hindi and nepali present an interesting case study due to their linguistic similarities coupled with significant disparities in nlp resources @xcite @xcite . hindi, with over 600 million speakers, benefits from comparatively extensive datasets and welldeveloped nlp tools @xcite @xcite . in contrast, nepali, spoken by around 30 million people, faces significant challenges due to the limited availability of resources and tools @xcite . given the shared linguistic heritage between hindi and nepali, cross-lingual transfer learning between these two languages could offer a promising avenue for improving ner performance in nepali by leveraging pre-trained hindi models and vice versa. this research evaluates the effectiveness of pretrained multilingual bert models-rembert @xcite , bert multilingual @xcite , muril @xcite , and distilbert multilingual @xcite -for cross-lingual transfer learning in ner tasks between hindi and nepali. by fine-tuning these models on individual language datasets and evaluating their performance in monolingual and cross-lingual settings, this research provides insights into the feasibility and potential of transfer learning in lowresource language contexts. furthermore, the study compares the models' performance in ner tasks for hindi and nepali without cross-lingual transfer learning. ## related work named entity recognition is a foundational task in nlp, focusing on identifying and classifying named entities within text @xcite . ner methodologies have evolved from traditional rule-based approaches to more sophis-ticated machine learning techniques and, recently, to large language models (llms) @xcite . among these, models like bert have significantly advanced the state of the art in ner by leveraging contextual embeddings and transformer-based architectures @xcite . in the context of hindi ner, research has spanned both traditional and llm-based methods, with resources like hiner contributing to notable advancements @xcite . although nepali ner has been less extensively studied, recent efforts have focused on applying llms to address the language's lowresource status, with specialized datasets and algorithms playing a critical role in these developments @xcite @xcite . cross-lingual transfer learning has shown significant promise in enhancing ner performance, particularly for low-resource languages @xcite . multilingual bert models, such as mbert @xcite and xlm-roberta @xcite , have demonstrated success across various nlp tasks by enabling the transfer of semantic properties across languages @xcite . to the best of our knowledge, this study is the first to investigate cross-lingual transfer learning between hindi and nepali, leveraging their linguistic similarities-a relationship that has not been explored in previous research. ## methodology this section is structured into three primary subsections, each providing a comprehensive understanding of the research approach. first, an overview of the linguistic characteristics of hindi and nepali is provided, emphasizing the similarities and distinctions between the two languages. second, the datasets utilized in this study are discussed, detailing their sources, statistical attributes, and the preprocessing techniques employed to ensure consistency across languages. lastly, the experimental setup is described, focusing on finetuning pre-trained multilingual bert models for the monolingual and cross-lingual ner task. ## experiments this study first pre-trains and evaluates the models on a single language dataset for the ner task. it is followed by fine-tuning and evaluating them on the second language dataset, as shown in figure 5 . initially, a multilingual base model from hugging face @xcite ) is pre-trained on the hindi language training dataset for ner, and the model's performance is evaluated on the test dataset in the same language using the f1 score @xcite as the evaluation metric, which is the harmonic mean of precision and recall scores. the pre-trained model is then fine-tuned on the nepali language training dataset, and its performance is evaluated on the nepali test dataset using the f1 score. the same experiment is repeated by pre-training and evaluating base models first on the nepali training and test dataset, then finetuning and evaluating on the hindi training and test dataset, and finally evaluating on the hindi test dataset for cross-lingual ner. these experiments are conducted for all four mentioned bert-based models. the hyperparameters used in the experiments are detailed in ## results and discussion the results of the experiments are presented in in monolingual tasks, rembert achieved the highest f1 score for hindi at 0.937, emphasizing its strong capability in managing languages with relatively rich resources. its deep architecture, coupled with rebalanced training data across 110 languages, enables it to capture subtle patterns within hindi, resulting in better performance. on the other hand, muril slightly outperformed other models in the nepali ner task, achieving an f1 score of 0.979, thereby demonstrating its effectiveness in contexts where resources are limited. the design of muril, which specifically focuses on indian languages and incorporates transliterated text to address code-switching, renders it particularly suitable for nepali. both bert multilingual and dis-tilbert multilingual displayed competitive performance, with distilbert multilingual achieving a marginally higher score in hindi (0.928) compared to nepali (0.972) despite its smaller and more efficient architecture. bert multilingual, with f1 scores of 0.922 for hindi and 0.974 for nepali, highlighted its versatility and balanced performance across both languages. in cross-lingual scenarios, bert multilingual showed the most notable improvements, with f1 hyperparameter value learning rate optimized using optuna @xcite , range: 1e ## conclusion distilbert multilingual, though slightly less effective in cross-lingual transfer, offered a commendable balance between performance and computational efficiency. these findings emphasize the critical role of model selection based on the task's specific linguistic and resource conditions, suggesting that general-purpose models like bert multilingual are particularly well-suited for cross-lingual applications.