XL -Suite: Cross-Lingual Synthetic Training and Evaluation Data for Open-Ended Generation (2025)

Abstract: Cross-lingual open-ended generation – responding in a language different from that of the query – is an important yet understudied problem. This work proposes XL-Instruct, a novel technique for generating high-quality synthetic data, and introduces XL-AlpacaEval, a new benchmark for evaluating cross-lingual generation capabilities of large language models (LLMs). Our experiments show that fine-tuning with just 8K instructions generated using XL-Instruct significantly improves model performance, increasing the win rate against GPT-4o-mini from 7.4% to 21.5% and improving on several fine-grained quality metrics. Moreover, base LLMs fine-tuned on XL-Instruct exhibit strong zero-shot improvements to same-language question answering, as shown on our machine-translated m-AlpacaEval. These consistent gains highlight the promising role of XL-Instruct in the post-training of multilingual LLMs. Finally, we publicly release XL-Suite, a collection of training and evaluation data to facilitate research in cross-lingual open-ended generation.

## introduction cross-lingual generation is the task of understanding a query in a given source language and generating a response in a different target language. this task has assumed greater relevance in the recent era of large language models (llms) with multilingual capabilities. @xcite noted its usefulness for a) companies that serve such llms across dozens of languages, but optimizing a prompt for each input language is inefficient in practice, and b) when a user needs a generation in a language they do not speak. the conventional cascaded approaches to cross-lingual generation @xcite @xcite could be problematic due to the noisy * work done while at unbabel. nature of machine translation, which leads to information loss or an unnatural-sounding response. it is also wasteful of inference time and cost, since the intermediary english response is thrown away once the desired cross-lingual output is obtained. the adaptation of llms to cross-lingual openended generation is relevant, given their versatile capabilities in both language conversion and question answering, but remains understudied. a primary reason is the absence of high-quality datasets and evaluation benchmarks. this work addresses the data deficiency for the cross-lingual generation task from both the modelling and evaluation perspectives. we first introduce xl-alpacaeval, a cross-lingual evaluation benchmark built on al-pacaeval @xcite , and we observe poor off-the-shelf performance for most open-source multilingual llms. as a solution, we propose xl-instruct, a synthetic data generation technique to create high-quality cross-lingual data at scale (illustrated in figure 1 ) and show that finetuning with xl-instruct significantly and consistently boosts cross-lingual performance across a range of base and instruction-tuned llms. beyond cross-lingual capabilities, we also created a machine-translated benchmark for same-language generation, m-alpacaeval, to demonstrate that our proposed data synthesis method achieves strong zero-shot transfer performance. in this work, we seek to answer the following research questions through a comprehensive set of experiments: ## xl-alpacaeval: a cross-lingual evaluation benchmark dataset to evaluate cross-lingual open-ended generation, we create the xl-alpacaeval benchmark, which is adapted from alpacaeval v1 @xcite . alpacaeval contains 805 multidomain prompts sampled from various test sets @xcite , including openassistant @xcite , koala @xcite , vicuna @xcite , self-instruct @xcite ) and anthropic's helpfulness test set @xcite . evaluation is carried out through the llm-as-a-judge approach @xcite , where an evaluator llm is used to estimate how often a model output would be preferred by humans over a baseline reference. to create xl-alpacaeval, we first manually examine the alpacaeval dataset and filter out prompts that are tailored towards eliciting responses in english. for example, questions about correcting grammar in an english sentence cannot be answered cross-lingually (refer to appendix a.1.1 for a detailed justification and a list of excluded prompts). the filtered test set consists of 797 prompts. next, we add cross-lingual generation instructions (such as "answer in {language}") to prompts randomly sampled from a list of templates (in appendix a.1.2) and create an evaluation set for eight languages, spanning resource availability, writing script, and geographical location: german (deu), portuguese (por), hungarian (hun), lithuanian (lit), irish (gle), maltese (mlt), simplified chinese (zho), and hindi (hin). we focus on the en-x direction in this work, as generating in non-english is usually more challenging for llms that are usually english-centric. it should be straightforward to extend our benchmark to other languages and pairs-by appending the cross-lingual templated instructions to our filtered test set. evaluation while the original implementation used gpt-4-turbo as both reference and evaluator models, we use gpt-4o-mini for reference and gpt-4o as the judge, given gpt-4o's strong multilingual capabilities. our choice of using gpt-4omini as the reference model is motivated by two reasons: 1) we experiment with ~7-9b llms in this work, making the gpt-4o-mini model a suitable baseline; and 2) using different reference and judge models, with the more capable one as the judge, should mitigate self-preference bias of models @xcite . finally, gpt-4o has also been shown to obtain state-of-the-art pairwise correlations with human ratings in multilingual chat scenarios @xcite . models to evaluate off-the-shelf cross-lingual capabilities of existing multilingual llms, we benchmark several strong open-weight models in the ~7-9b parameter range: aya expanse 8b @xcite , llama 3.1 8b instruct @xcite , gemma 2 9b instruct @xcite , qwen 2.5 7b instruct @xcite , eu-rollm 9b instruct @xcite , aya 23 8b @xcite , and salamandra 7b instruct @xcite . inference is performed using the alpacaeval repository @xcite , with the default decoding settings: temperature 0.7, maximum tokens 2048, and models loaded in bfloat16. ## experiments on xl-alpacaeval: boosting cross-lingual generation models we conduct sft of two base (eurollm 9b, qwen2.5 7b) and three instruction-tuned (eu-3 https://huggingface.co/datasets/viyer98/ xl-instruct/resolve/main/data_prompts.py rollm 9b instruct, qwen2.5 7b instruct, and aya expanse 8b) models. we choose eurollm and qwen since they relatively underperform on the xl-alpacaeval benchmark ( ## experiments on m-alpacaeval: exploring zero-shot transfer having seen task-specific improvements, we now seek to evaluate the zero-shot performance of models fine-tuned with xl-instruct on multilingual and english open-ended generation, since these are arguably the more common use cases of llms. for this purpose, we first construct the m-alpacaeval benchmark by machine translating the alpacaeval test set into our 8 languages of interest, following similar efforts to create m-arenahard @xcite . we use gpt-4o for translation of the prompts. the evaluation setup is similar to xl-alpacaeval, wherein gpt-4o-mini is used as the reference model and gpt-4o is used as the judge. we present our results in ## conclusion in this work, we propose data resources for advancing cross-lingual open-ended generation-loosely defined as a task in which the query and the desired (open-ended) response are in different languages. this can be viewed as a distinct yet crucial subtask of multilingual generation. while cross-lingual generation may also include more complex scenarios, such as providing context in one language while the query and response are in another (or even multiple) languages, we focus here on the simpler scenario: queries posed in english with responses required in one of eight target languages -which includes high, medium, and low-resource eu and non-eu languages. with this goal in mind, we make three key contributions. first, we introduce the xl-alpacaeval benchmark to evaluate the current state of open llms, and report poor performances and significant gaps against gpt-4o-mini. second, we propose the xl-instruct technique, and show that this synthetic data can substantially boost crosslingual performance, both in terms of win rates and fine-grained quality metrics. third, we show that it exhibits strong zero-shot transfer to monolingual generation, both in english and beyond. based on these results, we strongly encourage researchers to post-train and evaluate their multilingual llms on our publicly released xl-suite.