Crossing Language Boundaries: Evaluation of Large Language Models on U rdu- E nglish Question Answering (2025)

Abstract: This study evaluates the question-answering capabilities of Large Language Models (LLMs) in Urdu, addressing a critical gap in low-resource language processing. Four models GPT-4, mBERT, XLM-R, and mT5 are assessed across monolingual, cross-lingual, and mixed-language settings using the UQuAD1.0 and SQuAD2.0 datasets. Results reveal significant performance gaps between English and Urdu processing, with GPT-4 achieving the highest F1 scores (89.1% in English, 76.4% in Urdu) while demonstrating relative robustness in cross-lingual scenarios. Boundary detection and translation mismatches emerge as primary challenges, particularly in cross-lingual settings. The study further demonstrates that question complexity and length significantly impact performance, with factoid questions yielding 14.2% higher F1 scores compared to complex questions. These findings establish important benchmarks for enhancing LLM performance in low-resource languages and identify key areas for improvement in multilingual question-answering systems.

## introduction the rapid advancement of llms has revolutionized natural language processing, demonstrating remarkable capabilities in various tasks, particularly in english and other high-resource languages. however, their effectiveness in low-resource languages, such as urdu, remains a critical area requiring systematic evaluation. as @xcite demonstrated that question answering (qa), as a fundamental test of language understanding, serves as an excellent probe for assessing these models' cross-lingual and multilingual capabilities. @xcite highlighted significant disparities in the performance of large language models (llms) between high-resource and lowresource languages. similarly, @xcite showed that while models like gpt-4 and mt5 achieve impressive results in english, their performance often degrades substantially when handling languages with limited training data or complex morphological structures. furthermore, @xcite , @xcite , and @xcite emphasized that urdu, spoken by approximately 170 million people worldwide, serves as a particularly intriguing case study due to its rich morphological structure, distinct script, and limited computational resources. the challenge of cross-lingual question answering has gained increasing attention in recent years. @xcite focused primarily on transfer learning and fine-tuning approaches. however, the emergence of large-scale multilingual models has opened new possibilities for zero-shot and cross-lingual applications. @xcite demonstrated the potential of cross-lingual representation learning, while @xcite explored adapter-based approaches for cross-lingual transfer. the development of urdu-specific resources has also seen notable progress. @xcite created uquad1.0, providing crucial benchmarks for evaluating model performance. these resources, combined with advances in multilingual model architectures, create an opportunity to systematically assess how well current llms handle cross-lingual and multilingual qa tasks involving urdu. @xcite proposed a context-aware qa framework tailored to urdu, utilizing sliding window score specifically designed for comprehension of long-context dependencies. their methodology sets a benchmark that aligns with this study's focus on evaluating cross-lingual model performance for low-resource languages. @xcite have shown that models with fewer parameters but more language-specific training often outperform larger, general-purpose models in urdu nlp tasks. this finding raises important questions about the trade-offs between model size and language-specific optimization, as discussed by @xcite . furthermore, @xcite suggest that carefully designed prompting strategies can significantly impact cross-lingual performance. the relationship between script systems and model performance presents another crucial consideration. unlike languages that use latin script, @xcite note that urdu's nastaliq script introduces additional complexity in text processing and token alignment. wang et al. (2019) demonstrated that script differences can significantly impact model performance in cross-lingual tasks, making this an important factor in our evaluation. our work makes several key contributions to this developing field: our evaluation framework includes five experimental settings: (e1) full urdu prompts, (e2) urdu questions with english context, (e3) english questions with urdu context, (e4) full english prompts, and (e5) mixed-language prompts. this setup allows us to examine various cross-lingual comprehension and generation challenges. the findings reveal significant performance gaps, with models experiencing noticeable degradation in urdu and cross-lingual settings. gpt-4, for instance, achieves an f 1 score of 89.1% in english but drops to 76.4% in urdu, with further declines in cross-lingual tasks. these results under-score the complexities of multilingual model development and the need for progress in low-resource languages like urdu. this study contributes valuable insights into llms' cross-lingual limitations, emphasizing the ongoing need for robust multilingual modeling, especially for morphologically complex languages. the remainder of this paper is organized as follows: section 2 provides a review of related work, highlighting key advancements and challenges in multilingual nlp and cross-lingual question answering. section 3 gives details of the methodology, including models selected and prompting techniques. section 4 describes the datasets used and experiments done. section 5 presents the results and discussion, focusing on performance gaps, question type analysis, and error patterns. section 5 outlines the limitations of the current study. ## related work the exploration of large language models (llms) in multilingual contexts, particularly for lowresource languages like urdu, has garnered significant attention in recent years. this literature review examines key studies that have contributed to understanding and advancing llms' capabilities in cross-lingual question answering (qa) and related tasks.cross-lingual qa involves answering questions in one language based on context provided in another, posing unique challenges for llms. @xcite investigated zero-shot cross-lingual transfer for multilingual qa over knowledge graphs, highlighting the difficulties llms face when transferring knowledge across languages without fine-tuning . similarly, @xcite proposed synthetic data augmentation to enhance zero-shot cross-lingual qa performance, demonstrating that generating synthetic data in target languages can improve model accuracy without additional annotated data . the scarcity of high-quality datasets in urdu has been a significant barrier to developing effective nlp models. to address this, @xcite introduced uqa, a corpus for urdu qa generated by translating the stanford question answering dataset (squad2.0) using the eats technique, which preserves answer spans in translated contexts . additionally, @xcite developed uquad1.0, an urdu qa dataset combining machine-translated squad data with human-generated samples, providing a substantial resource for training urdu qa models . evaluating llms on low-resource languages like urdu has revealed performance disparities compared to highresource languages. a study by @xcite assessed general-purpose models such as gpt-4-turbo and llama-3-8b against specialized models fine-tuned on specific tasks, focusing on classification and generation tasks in urdu. the findings indicated that models with fewer parameters but more language-specific data performed better than larger models with less language-specific data, underscoring the importance of tailored training for low-resource languages prompting techniques play a crucial role in zero-shot learning scenarios, where models are expected to perform tasks without task-specific training. @xcite explored zero-shot cross-lingual open-domain qa, emphasizing the impact of prompt design on model performance across languages. their work suggests that carefully crafted prompts can enhance llms' ability to generalize across languages, even in the absence of fine-tuning. despite advancements, challenges persist in developing llms for low-resource languages. the limited availability of high-quality training data, coupled with inherent linguistic complexities, hampers model performance. future research should focus on creating comprehensive multilingual datasets, developing effective cross-lingual transfer learning techniques, and designing models that can adapt to the nuances of low-resource languages like urdu. in summary, while significant progress has been made in cross-lingual qa and the development of resources for low-resource languages, ongoing efforts are essential to bridge the performance gap between high-resource and low-resource languages in nlp applications. ## methodology this study investigates the performance of large language models (llms) on urdu question answering (qa) using zero-shot and cross-lingual prompts. we evaluate multiple models, explore various prompt settings, and assess model responses to identify the strengths and limitations of llms in a low-resource language context. ## discussion this section discusses the findings from results, focusing on performance gaps, question type analysis, error patterns, prompt setting impacts, and model-specific observations. language performance gap: an analysis of the language performance gap shows a marked decrease in model accuracy when transitioning from english to urdu prompts. on average, em scores dropped by 18.5%, f 1 scores by 12.7%, and rouge-l scores by 13.3% when shifting from english to urdu. this significant drop highlights the models' limitations in handling lowresource languages, as well as the need for more language-specific training data to mitigate these gaps. the language performance gap is most apparent in mbert and xlm-r, which are pretrained on a wide variety of languages but still struggle with urdu-specific constructs and contextual understanding. question type analysis: uquad1.0, being an extractive machine reading comprehension dataset, exclusively contains questions with answers that are direct spans from the context. however, the models displayed varying levels of effectiveness across different question types.