Bridging the Gap: Efficient Cross-Lingual NER in Low-Resource Financial Domain (2025)

Abstract: We present an innovative and efficient modeling framework for cross-lingual named entity recognition (NER), leveraging the strengths of knowledge distillation and consistency training. Our approach distills knowledge from an XLM-RoBERTa model pre-trained on a high-resource source language (English) to a student model, which then undergoes semi-supervised consistency training with KL divergence loss on a low-resource target language (Arabic). We focus our application on the financial domain, using a small, sourced dataset of financial transactions as seen in SMS messages Using datasets comprising SMS messages in English and Arabic containing financial transaction information, we aim to transfer NER capabilities from English to Arabic with minimal labeled Arabic samples. The framework generalizes named entity recognition from English to Arabic, achieving F1 scores of 0.74 on the Arabic financial transaction dataset and 0.61 on the WikiANN dataset, surpassing or closely competing with models that have 1.7 and 5.3 more parameters, respectively, while efficiently training it on a single T4 GPU. Our experiments show that using a small number of labeled data for low-resource cross-lingual NER applications is a wiser choice than utilizing zero-shot techniques while also using up fewer resources. This framework holds significant potential for developing multilingual applications, particularly in regions where digital interactions span English and low-resource languages.

## introduction named entity recognition (ner) has become pivotal in natural language processing (nlp) within finance, driven by the vast amount of digital content and the need to extract meaningful insights from financial texts. ner involves identifying and classifying named entities such as organizations, currencies, financial instruments, and monetary values, critical for applications like sentiment analysis, risk assessment, and investment recommendation systems. however, developing accurate and efficient cross-lingual ner models remains challenging due to sparse labeled data in low-resource languages and the complexities of capturing cross-lingual variations. cross-lingual ner is particularly crucial for industries operating globally, where analyzing customer feedback, social media posts, and other unstructured data in multiple languages can provide valuable insights for strategic decision-making. achieving accurate entity recognition in diverse languages supports trend identification, enhances intelligence extraction, and improves operational efficiency. however, creating robust cross-lingual ner models requires substantial resources, including annotated data, multilingual expertise, and computational infrastructure @xcite @xcite . despite the major improvements in sota performance across a range of nlp tasks achieved by large language models (llms), several challenges remain for adapting them to cross-lingual ner and ner in general. supervised ner baselines have been shown to outperform llms on ner tasks up until very recently @xcite . despite that, many of these llms adapted for ner require specific prompting techniques with no guarantees that these prompts will work across different domains. additionally, these llms are extremely costly to train, fine-tune, and efficiently deploy due to their substantial parameter sizes, especially in low-resource cross-lingual settings. thus, there is a growing need for efficient, scalable cross-lingual ner models capable of transferring knowledge from resource-rich to resource-scarce languages. in this paper, we propose a novel framework to enhance cross-lingual named entity recognition. we designate english as a resource-rich language and arabic as a low-resource language @xcite . our framework leverages knowledge distillation to transfer insights from a teacher model pre-trained on a resource-rich language (english) to a compact student model. subsequently, we employ consistency training to fine-tune the student model specifically for arabic. our experiments focus on identifying entities in semi-structured sms texts containing financial transaction information in both english and arabic, with access restricted to a small number of labeled examples in the target language. additionally, we evaluate our model's performance using the wikiann dataset @xcite , comparing it against state-of-the-art models in cross-lingual knowledge transfer. our findings demonstrate that our model achieves superior or comparable results while using significantly less training data than existing benchmarks. the proposed approach showcases robust crosslingual learning capabilities at a fraction of the data labeling costs typically associated with such tasks. our contributions include: this paper contributes to advancing the field of cross-lingual ner by proposing a pragmatic and scalable solution that addresses the challenges of language resource disparity in real-world applications. ## related works cross-lingual named entity recognition (ner) in low-resource languages poses significant challenges and has been a focal point of recent research efforts. various approaches have emerged, leveraging transfer learning techniques to enhance ner performance across different languages. transfer learning, particularly using pre-trained models like bert and xlm-roberta, has proven effective in adapting ner models to cross-lingual scenarios @xcite . these models capitalize on large-scale pre-training on diverse datasets, enabling them to learn robust representations that generalize well across languages. knowledge distillation has emerged as a powerful technique in cross-lingual ner @xcite . by transferring knowledge from a pre-trained teacher model to a smaller student model, knowledge distillation facilitates efficient knowledge transfer while maintaining performance. this approach is particularly advantageous in scenarios with limited labeled data, such as low-resource languages, where it enhances the student model's ability to capture complex linguistic patterns. in addition to knowledge distillation, incorporating consistency training further enhances crosslingual ner performance @xcite . consistency training uses unsupervised learning principles to enforce consistent predictions under small perturbations of the input data. this regularization technique improves model robustness and generalization, crucial for adapting ner models to diverse linguistic contexts. our proposed approach integrates both knowledge distillation and consistency training to address cross-lingual ner challenges in semi-structured financial text data in low-resource languages. by combining these techniques, we aim to leverage the strengths of pre-trained models while enhancing model adaptability to specific target languages, such as arabic. ## methods in this section, we present the methodology employed for the problem of cross-lingual ner for semi-structured financial text data in low-resource languages. this section is structured into three sub-sections: problem formulation, model architecture, and training. ## limitations while our approach demonstrates promising results in cross-lingual ner, it has several limitations. one key limitation is the inconsistency in results for minority classes. this inconsistency arises from the scarcity of samples for certain classes in the data, which is already limited in the low-resource setting. this could be overcome by choosing wellbalanced data and skipping samples with high "o" class entities in the target language. it will lead to better generalization finally, our approach requires a small amount of labeled data in the target language for consistency training. while this requirement is minimal compared to fully supervised methods, it may still pose challenges in scenarios where even a small amount of labeled data is not available. ## conclusion in this paper, we introduce a novel framework that uses knowledge distillation and consistency training to enhance cross-lingual named entity recognition. knowledge is transferred from a teacher model pre-trained in english to a smaller student model, which is then fine-tuned for arabic. our model, kd+ct, is validated on banking transaction data (semi-structured) in both english and arabic, showcasing competitive performance compared to state-of-the-art benchmarks on several datasets. our modeling approach successfully combines knowledge distillation with consistency training, addressing the challenges of developing accurate cross-lingual ner models for low-resource languages. importantly, our model significantly outperforms the naive benchmark, the student, and the teacher models in entity recognition on the target language dataset (arabic) and achieves performance comparable to the larger teacher model while being approximately 3 times smaller in terms of parameters (66 million parameters compared to the teacher model's 270 million parameters) on the source language dataset (english). this demonstrates the remarkable cross-lingual generalization capabilities of our kd+ct model, effectively leveraging the knowledge distilled from the high-resource language and enhancing performance on the low-resource language through consistency training. additionally, we evaluate our model on the wikiann dataset, achieving competitive results against state-of-the-art methods, even with minimal labeled data in the target language. notably, our model showcases an improvement in the precision metric, achieving a precision of 0.87 compared to the teacher model's 0.64. this improvement is particularly significant in the financial sector, where label accuracy is vital. our proposed cross-lingual ner model offers valuable contributions to the development of multilingual applications, enabling the extraction of insights, identification of trends, and making wellinformed decisions across multiple languages. we hope that our work will inspire further research in this field and facilitate the development of efficient and effective cross-lingual ner models, benefiting low-resource languages and beyond.