Challenges in U rdu Machine Translation (2024)

Abstract: Recent advancements in Neural Machine Translation (NMT) systems have significantly improved model performance on various translation benchmarks. However, these systems still face numerous challenges when translating low-resource languages such as Urdu. In this work, we highlight the specific issues faced by machine translation systems when translating Urdu language. We first conduct a comprehensive evaluation of English to Urdu Machine Translation with four diverse models: GPT-3.5 (a large language model), opus-mt-en-ur (a bilingual translation model), NLLB (a model trained for translating 200 languages), and IndicTrans2 (a specialized model for translating low-resource Indic languages). The results demonstrate that IndicTrans2 significantly outperforms other models in Urdu Machine Translation. To understand the differences in the performance of these models, we analyze the Urdu word distribution in different training datasets and compare the training methodologies. Finally, we uncover the specific translation issues and provide suggestions for improvements in Urdu machine translation systems.

## introduction neural machine translation (nmt) has shown remarkable performance on benchmark datasets, particularly following the introduction of transformer architectures @xcite . among these advancements, large language models like gpt-3.5 and 4 have demonstrated promising potential for machine translation, particularly for resource-rich languages including english, french, and german. however, these models face numerous challenges in translating low-resource languages (e.g., urdu) due to limited training compared to their high-resource counterparts @xcite . urdu is spoken by over 100 million people worldwide @xcite . it is predominantly spoken in pakistan, serving as the national language @xcite and holds significant cultural importance. urdu is also spoken in various regions of india, particularly in states like uttar pradesh, bihar, and telangana, with a sizable population of speakers. however, due to the scarcity of available linguistic resources for urdu, it is considered a lowresource language @xcite . in this work, we empirically evaluate four language models for urdu machine translation: gpt-3.5 -a large language model, opus-mten-ur -a bilingual model specifically trained for urdu translation, nllb -a multilingual translation model designed to cover 200 languages, incorporating a mix of both high-resource and largely low-resource languages and indictrans2 -a multilingual translation model designed for low-resource indian languages. indictrans2 demonstrates the highest sacrebleu and chrf on five diverse machine translation datasets, followed by nllb , gpt-3.5 and opus-mt-en-ur. to identify the challenges in urdu machine translation, we examine the translation capability of the four different models qualitatively and highlight the key areas where the bilingual, multilingual, and large language models struggle to perform. ## background machine translation is a crucial aspect of nlp, automating text translation between languages. it has evolved from rule-based to data-driven and neural approaches. traditional rule-based systems faced challenges with language complexities, while statistical methods improved but still struggled with syntax and semantics @xcite . neural machine translation (nmt) has significantly improved the performance, employing deep learning models like sequence-to-sequence architectures @xcite for more fluent and context-aware translations. the transformer architecture has improved the overall quality of machine translation. therefore, large language models, such as gpt-3.5, have emerged as potent candidates for machine translation tasks. numerous studies have been conducted to assess the effectiveness of these modals for neural machine translation. @xcite demonstrate that gpt-3.5 can generate remarkably fluent and competitive translation outputs, particularly in the zero-shot setting, especially for high-resource language translations. prior research has demonstrated the remarkable performance of large language models (llms) in high-resource bilingual translation tasks, such as english-german translation @xcite . @xcite observed that gpt-4 performs competitively with commercial translation products for high-resource european languages but demonstrates a notable drop in performance for low-resource and distant languages. @xcite show that gpt-4 is unsuitable for extremely low-resource languages. however, there is currently a lack of cross-evaluation of different types of language models for urdu machine translation. ## methodology and experiments we conduct empirical evaluation for urdu machine translation on four types of language models: large language model (llm), bilingual model, and two multilingual models using five diverse datasets. through this investigation, we aim to gain insights into the translation capabilities of these language models for the urdu language. ## limitations and conclusion in this work, we investigate the urdu translation capabilities of four diverse models and uncover the specific challenges. we find that indictrans2 outperforms other models for english to urdu translation, demonstrating superior performance on sacrebleu and chrf++ scores, primarily due to its specialized training process and superior urdu word distribution in its dataset. we uncover specific urdu translation issues including named entity recognition, mistranslation, word repetition, literal translations, word omissions, and transliteration errors. addressing these challenges requires focused efforts on constructing high-quality urdu training datasets, refining model training methods, and incorporating more robust evaluation metrics. for future work, our evaluation of urdu machine translation can be extended to additional domainspecific datasets and other low-resource indic languages to uncover additional issues.