s P hin X : Sample Efficient Multilingual Instruction Fine-Tuning Through N-shot Guided Prompting (2025)

Abstract: Despite the remarkable success of large language models (LLMs) in English, a significant performance gap remains in non-English languages. To address this, we introduce a novel approach for strategically constructing a multilingual synthetic instruction tuning dataset, sPhinX. Unlike prior methods that directly translate fixed instruction-response pairs, sPhinX enhances diversity by selectively augmenting English instruction-response pairs with multilingual translations. Additionally, we propose LANGIT, a novel N-shot guided fine-tuning strategy, which further enhances model performance by incorporating contextually relevant examples in each training sample. Our ablation study shows that our approach enhances the multilingual capabilities of Mistral-7B and Phi-3-Small improving performance by an average of 39.8% and 11.2%, respectively, across multilingual benchmarks in reasoning, question answering, reading comprehension, and machine translation. Moreover, sPhinX maintains strong performance on English LLM benchmarks while exhibiting minimal to no catastrophic forgetting, even when trained on 51 languages.

## introduction large language models (llms) have demonstrated exceptional performance across various tasks in english. however, their performance in some non-english languages remains comparatively limited @xcite . further, the gap between the performance of large language models (llms) and small language models (slms) is more pronounced @xcite in non-english languages. cui et al. (2023) * denotes equal contribution, ∆ denotes equal advising, ♢ work done when the authors were at microsoft and @xcite utilize the method of finetuning models on datasets focused on particular languages. however, this can lead to catastrophic forgetting, which may negatively impact performance in english @xcite . few techniques have been proposed to bridge this gap, such as incorporating better pre-training data in various languages and improving base tokenizers @xcite . however, most of these changes need to be implemented in the pre-training stage, which demands extensive data and computational resources, making it practically unfeasible in many scenarios @xcite . consequently, the most well-studied technique involves fine-tuning models for specific languages and tasks. instruction fine-tuning (ift) has become a popular technique to enhance the performance of language models in specific languages. this method combines the benefits of both the pre-training, fine-tuning, and prompting paradigms @xcite . sample diversity is essential for effective instruction tuning in multilingual datasets. many recent datasets have been generated by translating english content into other languages or by employing selfinstruct techniques based on seed prompts @xcite . however, both methods can limit diversity. machine translation may result in the loss of semantic nuance @xcite , while self-instruct approaches often yield repetitive and homogeneous samples @xcite . this highlights the critical need for datasets that encompass a wide range of diverse samples. in this paper, we present a novel recipe for creating a multilingual synthetic instruction tuning dataset, sphinx. it comprises 1.8m instructionresponse pairs in 51 languages, derived by augmenting the orca instruction tuning dataset samples @xcite through selective translated augmentation using @xcite . we assess the effectiveness of sphinx by fine-tuning two models -phi-3-small and mistral-7b -across a range of evaluation benchmarks that test various language model capabilities across discriminative and generative tasks. we compare models fine-tuned on sphinx with those using other synthetic multilingual instruction tuning datasets like aya @xcite , multilingual alpaca @xcite , and bactrian @xcite and observe significant performance gains across languages. we also compare our proposed translation strategy with translating the entire instruction using azure translator api, as is done with the popular multilingual synthetic ift datasets to demonstrate the efficacy of our approach. the contributions of this paper are as follows: ## sphinx dataset in this section, we describe our dataset construction methodology ( §3.1), dataset filtering, and cleaning pipelines ( §3.2). ## langit following instruction tuning strategies of @xcite and also taking inspiration from @xcite , we devise language-specific nshot guided instruction fine-tuning (langit) figure : 1. this method aims to improve the model's ability to follow instructions by augmenting training examples with additional context from a set of few-shot examples in the same language. this added context helps guide the model, enabling it to learn more effectively from the provided examples. for each training example, we begin by sampling a number of few-shot examples, which are instruction-response pairs in the same language. the number of few-shot examples n is determined probabilistically, with a 30% chance of selecting no few-shot examples, a 20% chance of selecting one, and gradually lower probabilities for higher numbers of few-shots. the maximum number of fewshot examples we sample is six, due to constraints imposed by the model's context length (8192 tokens) and the typically higher tokenization length in languages other than english. once the number of few-shot examples is determined, they are prepended to the main training example, forming an augmented input. this augmented input is then fed into the model for instruction tuning. the purpose of this approach is to expose the model to additional examples of different tasks, helping it generalize better to new tasks in the same language. we performed experiments to analyze how the model performs on each dataset when fine-tuned using the langit strategy detailed in the next section ( §6). additionally, we fine-tuned the models on the sphinx dataset without using langit to provide a baseline for comparison. to assess the effectiveness of each instruction-tuning dataset on an equal scale, we conducted a comparative analysis of model performance on different benchmarks, fine-tuning each model on approximately 8 billion tokens per dataset using the lang strategy. this fine-tuning strategy is consistently applied across datasets for both the phi-3-small and mistral-7b base models. a comparison of token lengths across different datasets is provided in evaluation: we evaluate foot_4 our fine-tuned models along with the available base and instruction fine-tuned model variants of mistral-7b and phi-3-small (ift foot_5 ) on 4 discriminative tasks; xcopa (ponti et al., 2020)(4-shot), xs-torycloze @xcite (4-shot), xwinograd @xcite (0-shot), @xcite , belebele(0-shot) @xcite , and 2 generative tasks; xquad (3-shot) @xcite and translation (4shot) @xcite @xcite using the language model evaluation harness @xcite . the number of few-shot selection are inspired from these works @xcite @xcite apart from generative tasks such as xquad, and machine translation, we also evaluate our instruction-tuned models on open-ended generation prompts. for this, we use an llm-based evaluation approach to simulate win rates. we use the opensource test set from the aya dataset @xcite , which includes 250 prompts per language across six languages. we use gpt-4o as the llm evaluator to pick the preferred model generation on this test set, and we subsequently compute win rates (%) based on these preferences. to avoid a potential bias, we randomize the order of the models during the evaluation. the prompt for the evaluator is described in the appendix: §a.1. ## conclusion in this paper, we demonstrated how instruction tuning mistral-7b and phi-3-small on sphinx effectively improve their multilingual capabilities. we observed that instruction tuning the models using the sphinx dataset leads to performance improvement by an average of 39.8% and 11.2% for mistral-7b and phi-3-small respectively when compared to their corresponding base models across multilingual benchmarks. moreover, sphinx exhibits greater sample efficiency and diversity compared to other multilingual instruction tuning datasets. we also proposed langit, a strategy that enhances model performance by incorporating n few-shot examples, boosting results by 15% and 3.2% for mistral-7b and phi-3-small, respectively, over vanilla fine-tuning with sphinx. compared to the sphinx-t translation baseline, langit yielded gains of 11.7% and 6.3%. models fine-tuned on sphinx also showed improved performance in unseen languages without degrading english performance. we also observed that the gpt-4o win-rate evaluations fa-vored mistral-7b with langit, while phi-3-small showed more ties due to its stronger baseline. finally, we plan on releasing a subset of our augmented dataset built on openorca (open-sphinx).