Alif: Advancing U rdu Large Language Models via Multilingual Synthetic Data Distillation (2025)

Abstract: Developing a high-performing large language models (LLMs) for low-resource languages such as Urdu, present several challenges. These challenges include the scarcity of high-quality datasets, multilingual inconsistencies, and safety concerns. Existing multilingual LLMs often address these issues by translating large volumes of available data. However, such translations often lack quality and cultural nuance while also incurring significant costs for data curation and training. To address these issues, we propose Alif-1.0-8B-Instruct, a multilingual Urdu-English model, that tackles these challenges with a unique approach. We train the model on a high-quality, multilingual synthetic dataset (Urdu-Instruct), developed using a modified self-instruct technique. By using unique prompts and seed values for each task along with a global task pool, this dataset incorporates Urdu-native chain-of-thought based reasoning, bilingual translation, cultural relevance, and ethical safety alignments. This technique significantly enhances the comprehension of Alif-1.0-8B-Instruct model for Urdu-specific tasks. As a result, Alif-1.0-8B-Instruct, built upon the pretrained Llama-3.1-8B, demonstrates superior performance compared to Llama-3.1-8B-Instruct for Urdu specific-tasks. It also outperformed leading multilingual LLMs, including Mistral-7B-Instruct-v0.3, Qwen-2.5-7B-Instruct, and Cohere-Aya-Expanse-8B, all within a training budget of under $100. Our results demonstrate that high-performance and low-resource language LLMs can be developed efficiently and culturally aligned using our modified self-instruct approach.

## introduction the rapid advancement of llms @xcite has revolutionized natural language process-1 github: github.com/traversaal-ai/alif-urdu-llm ing (nlp) across multiple languages and applications. however, a significant disparity persists between high-resource languages, such as english, and low-resource languages, such as urdu. these disparities create technological barriers for billions of speakers of underrepresented languages, limiting their access to ai-driven tools and advancements. the inclusion of low-resource languages in llm development is not merely a technical challenge but a crucial step toward fostering inclusive, globally accessible ai systems that cater to diverse linguistic communities. developing high-performing llms for lowresource languages presents several challenges, including the scarcity of high-quality datasets, multilingual inconsistencies, translation inaccuracies, reasoning limitations, and ethical concerns. a common approach to addressing these challenges relies on leveraging translated data from high-resource languages. however, translations often fail to capture regional knowledge and cultural nuances, leading to compromised language representation and ineffective communication in low-resource settings @xcite . in the case of urdu llms, additional factors contribute to their underperformance. urdu's linguistic complexity, including its unique alphabet, intricate grammar, syntax, and morphology, poses significant challenges in adapting nlp techniques developed for english. furthermore, urdu has borrowed extensively from regional languages such as hindi, punjabi, and persian and is written in both the perso-arabic and devanagari scripts, adding additional layers of complexity. while multilingual models exhibit some degree of understanding, their generation capabilities remain inadequate, particularly for languages with syntactic structures and writing systems distinct from english. among these challenges, the lack of high-quality datasets stands out as a fundamental limitation. current urdu datasets are sparse, manually labeled, and contain only a few thousand instances-insufficient for training robust llms. this scarcity results from multiple factors, including limited digitization of urdu literature, funding and infrastructure constraints, and the complexities of annotating urdu text, which require linguistic expertise and standardized guidelines. furthermore, translated data often fails to retain cultural nuances @xcite , such as idiomatic expressions and contextual meanings, thereby reducing a model's ability to generate culturally relevant responses. additionally, multilingual llms suffer from catastrophic forgetting, where training across multiple languages or modalities can degrade performance on certain language subsets unless carefully managed. the challenge of evaluation further complicates this issue @xcite , as creating frameworks that fairly and accurately assess performance across diverse languages and cultures demands significant expertise and resources. these issues are particularly pronounced for south asian low-resource languages like urdu, which, despite its online presence, lacks the research-driven resources necessary to develop competitive models @xcite . the homogeneity of existing datasets and evaluation standards exacerbates the underrepresentation of diverse linguistic and cultural contexts in modern llms, highlighting the urgent need for targeted efforts to bridge these gaps and promote inclusivity in multilingual ai development. to address all these challenges, alif-1.0-8b-instruct model offers a promising solution to the limitations of conventional multilingual training approaches. by leveraging a modified self-instruct technique, this model incorporates a carefully curated urdu dataset, specifically designed to enhance urdu generation quality, bilingual translation, culturally aware understanding, and urdunative chain-of-thought based reasoning capabilities. this unique multilingual synthetic data distillation approach not only improves the model's performance on urdu and english tasks but also upholds ethical commitments to safety and cultural sensitivity @xcite . prior research has demonstrated that tailored datasets significantly enhance the effectiveness of language models, enabling deeper linguistic and cultural understanding @xcite . by using a care-fully curated urdu dataset, alif-1.0-8b-instruct addresses persistent challenges in multilingual language modeling within constrained computational budgets @xcite . alif-1.0-8b-instruct demonstrates a significant leap in urdu-specific task comprehension, outperforming leading multilingual llms. its training pipeline follows a structured process: continued pretraining to reinforce foundational understanding, fine-tuning on the synthetic urdu-instruct dataset to enhance comprehension, incorporation of translated urdu data for broader knowledge, and replayed english data to mitigate catastrophic forgetting. as a result, alif-1.0-8b-instruct, built upon the pretrained meta llama-3.1-8b base, demonstrates superior performance compared to llama-3.1-8b-instruct. @xcite in urdu-specific benchmarks while maintaining strong english fluency. it also outperforms prominent multilingual models such as mistral-7b-instruct-v0.3 @xcite , qwen-2.5-7b-instruct @xcite , and cohere-aya-expanse-8b @xcite , all within an optimized training budget of less than $100. ## urdu-instruct dataset the urdu-instruct dataset, consisting of 51,686 examples generated using gpt-4o, api-version '2024-08-01-preview', @xcite , is a crucial component in fine-tuning alif-1.0-8b-instruct. it contains instructions and responses for seven key urdu tasks: generation (5,907), ethics (9,002), qa (8,177), reasoning (9,590), translation (10,001), classification (4,662), and sentiment analysis (4,347). the dataset was created using a self-instruct @xcite technique improved for cultural and linguistic nuance as shown in figure 1 foot_0 and explained below. ## multilingual urdu-english model: alif-1.0-8b-instruct the development of alif involves the integration of multiple datasets, each selected to serve a distinct role in the continued pre-training and fine-tuning process. this carefully structured approach is essential to enhancing the model's proficiency across a diverse range of tasks, ensuring robust linguistic capabilities. ## results on instruction-following tasks evaluating large language models (llms) for lowresource languages like urdu presents unique challenges due to the limited availability of high-quality benchmarks. additionally, while instruction-tuned models such as llama-3.1-8b-instruct have demonstrated strong multilingual capabilities, their performance in urdu nlp tasks remains underexplored. in this section, we benchmark alif-1.0-8b-instruct (alif) against llama-3.1-8b-instruct (llama) and other llms using the alpaca chat template across various benchmarks. these evaluations were conducted on runpod, using an a40 gpu with 48gb vram. ## effect of different quantization methods the deployment of large language models (llms) on various hardware architectures has traditionally been constrained by high computational and memory demands. however, the development of open source frameworks, such as llama.cpp @xcite , has facilitated the quantization of llms, significantly reducing their resource requirements and maintaining comparable accuracy for some quantized formats. this advancement also enables efficient local development, minimizing reliance on cloud services and enhancing data privacy. ## conclusion building a high-performing urdu llm presents distinct challenges, including data scarcity, translation quality issues, and reasoning complexity. existing methods often depend on large-scale trans- to further strengthen the study, future work will incorporate objective, task-specific metrics such as exact match, f1, bleu, comet, and bertscore to more rigorously quantify alignment, factuality, and stylistic correctness in bilingual settings. comparing multiple judge models and prompts will help evaluate robustness across cultural and linguistic variations. moving forward, we aim to broaden high-quality datasets, enhance reasoning through model merging and reinforcement learning, and benchmark alif against evolving multilingual and reasoning standards. alif marks a key step toward culturally aligned, reproducible, and cost-effective urdu nlp, driving inclusive and trustworthy ai forward.