Low-Resource Transliteration for R oman- U rdu and U rdu Using Transformer-Based Models (2025)

Abstract: As the Information Retrieval (IR) field increasingly recognizes the importance of inclusivity, addressing the needs of low-resource languages remains a significant challenge. Transliteration between Urdu and its Romanized form, Roman Urdu, remains underexplored despite the widespread use of both scripts in South Asia. Prior work using RNNs on the Roman-Urdu-Parl dataset showed promising results but suffered from poor domain adaptability and limited evaluation. We propose a transformer-based approach using the m2m100 multilingual translation model, enhanced with masked language modeling (MLM) pretraining and fine-tuning on both Roman-Urdu-Parl and the domain diverse Dakshina dataset. To address previous evaluation flaws, we introduce rigorous dataset splits and assess performance using BLEU, character-level BLEU, and CHRF. Our model achieves strong transliteration performance, with Char-BLEU scores of 96.37 for Urdu→Roman-Urdu and 97.44 for Roman-Urdu→Urdu. These results outperform both RNN baselines and GPT-4o Mini and demonstrate the effectiveness of multilingual transfer learning for low-resource transliteration tasks.

## introduction advancements in natural language processing (nlp), particularly transformer-based architectures, have significantly improved various linguistic tasks, including machine translation, transliteration, and cross-lingual transfer learning. however, low-resource languages continue to face challenges due to data scarcity, limited model training resources, and domain adaptation issues. one such underrepresented language is urdu, spoken widely in south asia, which features a perso-arabic script, right-to-left writing direction, and rich morphology @xcite . in parallel to its standard script, a romanized variant, known as roman-urdu, has become increasingly popular in digital communication, social media, and messaging platforms. this form of writing represents urdu using the latin alphabet, but lacks standardized spelling conventions, leading to high variability in the spelling of romanized words. as a result, transliterating between roman-urdu and standard urdu presents unique challenges, particularly in nlp applications. despite its widespread use online, research on roman-urdu to urdu or the opposite's transliteration remains scarce, limiting the development of language technologies that can effectively bridge the gap between these two writing systems. existing transliteration models for roman-urdu and urdu primarily rely on recurrent neural networks (rnns) and long short-term memory (lstm) architectures, as demonstrated in prior work using the roman-urdu-parl (rup) dataset @xcite . while these models achieve reasonable performance, they exhibit limitations in handling complex linguistic structures, proper names, and cross-domain generalization. furthermore, evaluation inconsistencies in prior research suggest that previously reported results may not fully reflect actual model performance. additionally, models trained solely on roman-urdu-parl struggle significantly when tested on other datasets such as google's dakshina dataset @xcite , revealing domain-specific weaknesses that impact real-world applicability. to address these challenges, we leverage multilingual transformer-based architectures and evaluate our model across diverse datasets to ensure robust generalization. we improve upon the current state-of-the-art by introducing a transliteration model using the m2m100 multilingual translation framework @xcite . our approach incorporates pretraining techniques, fine-tuning on multiple datasets, and masked language modelling (mlm; introduced in @xcite ) to enhance transliteration quality. mlm pretraining improves the model's ability to handle inconsistencies in roman-urdu spelling variations, enhancing transliteration robustness across datasets. to evaluate the effectiveness of our method, we assess performance on two datasets: we establish new benchmarks by evaluating bleu, character bleu, and chrf scores for both roman-@xmath0. our results demonstrate substantial improvements, particularly in generalization across datasets, addressing a key limitation in previous research. our key contributions are: one significant approach to addressing data scarcity has been multilingual transfer learning. @xcite introduced the xlm-r model, a multilingual language representation system trained on 100 languages, demonstrating that highresource languages can improve performance for low-resource counterparts. our work extends this approach by applying xlm-r to the task of transliteration between roman urdu and urdu, highlighting the potential of multilingual models to bridge data limitations in low-resource settings. transliteration for languages with arabicderived scripts-such as urdu, persian, and arabic has received limited attention compared to standard translation tasks. notable work on arabizi, the romanized form of arabic, highlights the relevance of transliteration research in other low-resource, informally written languages @xcite . however, roman-urdu transliteration remains underexplored. the roman-urdu-parl corpus @xcite , a large-scale parallel dataset of 6.37 million sentence pairs, has served as a benchmark in this domain. most existing systems based on this dataset rely on recurrent neural networks (rnns) or statistical models. these methods face limitations in generalization and domain robustness, particularly due to the high variability and informality of roman-urdu text. one challenge in roman-urdu-parl is its test set construction. the dataset was augmented with roman-urdu variants for 5,000 frequent words (adding over 4 million pairs), but the test set was randomly sampled afterwards. as a result, models may encounter near-duplicate examples in training and testing, potentially inflating bleu scores. although randomization was assumed to mitigate this issue, separating test samples before augmentation would have ensured cleaner evaluation. additionally, while a bleu score @xcite of 84 was reported, character-level metrics such as char bleu @xcite and chrf @xcite are more suitable for transliteration due to the nature of the task. additionally, the dakshina dataset @xcite has been utilized for training and evaluation of transliteration systems for south asian languages, including urdu. transformer-based architectures have shown improved character-level accuracy on this dataset compared to traditional models @xcite . these findings underscore the benefit of using more recent architectures and character-focused metrics in transliteration evaluation. furthermore, studies on cross-lingual transfer learning have shown promising results for lowresource nlp tasks @xcite . by leveraging multilingual models pre-trained on high-resource languages, researchers have demonstrated improved performance in text generation and translation tasks for underrepresented languages. another transliteration challenge arises in codemixed settings common in urdu and roman urdu, where english words appear within otherwise monolingual text. simple character-level mapping is often insufficient for handling such cases. prior studies @xcite emphasize the importance of contextual modeling to correctly interpret whether a word should be transliterated or left intact. context-aware multilingual models, especially those with sentence-level embeddings, have the potential to address these nuances effectively. ## background on romanization & roman urdu many multilingual communities, particularly in south asia and africa, use the latin script (english alphabet) to write their native languages-a practice known as romanization. this phenomenon emerged due to early computing and mobile technology limitations, where english-based keyboards and software lacked support for native scripts. as a result, speakers of languages such as urdu, hindi, bengali, and arabic adapted by using roman characters to represent their native words. this trend was further reinforced by the rise of the internet, where english remains the dominant digital language, making romanized text a convenient alternative for communication. among these, roman urdu, a romanized form of urdu, has gained widespread adoption in social media, e-commerce, online news, and digital communication platforms. unlike standardized latinbased writing systems, however, roman urdu lacks a formal spelling convention, leading to highly inconsistent and phonetic spellings. the same word may be written in multiple ways, depending on how a speaker perceives and pronounces it. for example, as also shown in figure 1 the urdu word for "what" can appear as "kia," "kya," or "keeya" in roman urdu, depending on the writer's preference. this lack of standardization poses significant challenges for nlp and transliteration models, as they must handle spelling variations, informal grammar, and code-switching with english. consequently, building an accurate and robust roman-urdu to urdu transliteration system requires not only data-driven learning approaches but also an understanding of phonetic spelling inconsistencies and context-aware modeling. ## methodology this section outlines the approach used to develop a roman-urdu and urdu transliteration model based on the m2m100 multilingual translation model. initially, we fine-tuned m2m100 directly on parallel transliteration data from the roman-urdu-parl (rup) dataset. while this yielded reasonable results, we observed that the model struggled with generalization when evaluated on a dataset it was not trained on, specifically the dakshina dataset, which contains roman-urdu text from a different domain. when fine-tuned solely on rup, the model performed well on rup but struggled on other transliteration datasets such as the google dakshina dataset due to the variations in writing style in roman-urdu. to mitigate this issue, we experimented with further fine-tuning on dakshina, but this led to a considerable drop in performance on rup, highlighting a trade-off between domain adaptation and dataset retention. this suggested that the model was overfitting to the dataset it was last trained on, rather than developing a generalizable transliteration capability. we also experimented with combining both datasets for training, but this approach was ineffective due to the large size difference between them. as discussed in section 4.3 and shown in section 5, we addressed this issue using mlm pretraining. overall, our methodology consists of two key phases: ## results and discussion in this section, we present the results of our roman-urdu and urdu transliteration experiments, comparing models trained with and without mlm pretraining. character-level bleu (char-bleu) is used as our primary evaluation metric in the main tables. while additional bleu and chrf scores are reported in the appendix a we begin by comparing our roman-@xmath1. as shown in gpt-4o mini, evaluated in a zero-shot setting, scored 80.96. in contrast, our fine-tuned m2m100 model trained solely on roman-urdu-parl data achieves a bleu score of 94.58, setting a new benchmark for this task and highlighting the effectiveness of transformer-based models over traditional sequence-to-sequence architectures. we selected bleu score for this comparison because the baseline results reported used this metric, and the model was not publically available. tables 2 and 3 present detailed char-bleu results for roman-@xmath3, respectively. without mlm pretraining, models trained on rup perform well on in-domain test data but struggle to generalize to the dakshina dataset. fine-tuning on dakshina improves performance on out-of-domain data but leads to some degradation on rup, reflecting a trade-off between domain adaptation and retention of previously learned mappings. we observe that mlm pretraining helps mitigate this trade-off. while the overall performance gain is modest, mlm-pretrained models demonstrate more balanced performance across datasets. in particular, fine-tuning on dakshina after mlm pretraining results in improved generalization without significantly harming performance on rup, suggesting that mlm helps stabilize learning by exposing the model to diverse subword and script-level patterns in advance. to provide additional context, we also evaluated openai's gpt-4o mini in a zero-shot setup using prompt-based decoding. gpt-4o serves as a strong multilingual baseline, having been trained on diverse transliteration patterns across multiple languages. while it performs reasonably well on dakshina, it lags behind our fine-tuned model on rup. notably, in the @xmath4, gpt-4o is more competitive, likely due to its broader exposure to romanized scripts and informal user-generated text. overall, we find that roman-@xmath5, likely due to the standardized nature of urdu script. in contrast, @xmath6, where multiple phonetic spellings may exist for the same word. while char-bleu offers a more forgiving measure than n-gram bleu, it still penalizes plausible variants not present in the reference. addressing this limitation-by incorporating multireference evaluation or designing more flexible scoring metrics-remains a promising direction for future work. in summary, our results demonstrate that transformer-based models fine-tuned on domainspecific data outperform both traditional rnns and large-scale general-purpose llms like gpt-4o. mlm pretraining contributes to better stability and cross-domain robustness, particularly in lowresource, non-standardized transliteration settings. ## conclusion and future work this paper presents a fine-tuned transliteration model for roman-urdu and urdu, based on the m2m100 multilingual translation model with masked language modeling (mlm) pretraining.