MIRAGE -Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems (2025)

Abstract: Traditional retrieval-augmented generation (RAG) benchmarks evaluate systems using heuristic-based metrics, but these require human preferences as the ground truth for reference. In contrast, arena-based benchmarks, where systems compete against each other, require an expensive large language model (LLM) as a judge for a reliable evaluation. We present a simple efficient technique to combine the best of both worlds. The idea is to train a surrogate judge using heuristic metrics as input, to output the LLM as a judge prediction.In our work, we develop MIRAGE-Bench, a synthetic arena-based RAG benchmark for 18 diverse languages on Wikipedia focused on multilingual answer generation evaluation. It extensively couples both heuristic features and LLM as a judge for evaluation. We benchmark 19 multilingual LLMs, and observe a high correlation (Kendall Tau ( \tau ) = 0.909) using our surrogate judge and between GPT-4o as a teacher using the Bradley-Terry framework. Our results show proprietary and large open-source LLMs currently dominate on MIRAGE-Bench. Our code and datasets are made publicly available here: https://github.com/vectara/mirage-bench.

## introduction large language models (llms) have recently gained popularity for information-seeking queries leading to the widespread adoption of retrievalaugmented generation (rag) @xcite @xcite . the naive rag setup traditionally includes a retrieval and a generation stage, conducted sequentially. rag systems such as bing search @xcite provide grounded responses, i.e., statements that include citations to one or more retrieved passages. the citations 1 mirage-bench has been coined as (miracl + rag + evaluation + benchmark). reduce factual hallucinations with easy verifiability and improve faithfulness to passages provided within context @xcite @xcite . however, existing rag benchmarks are englishcentric, due to uneven and scarce data available across multiple languages @xcite . so far, it is unclear how existing llms perform in multilingual rag, i.e., where queries and passages are non-english and the llm generates the answer in the same language. an example of a naive rag pipeline in hindi (hn) is shown in figure 1 . rag benchmarks can be broadly classified as either (i) heuristic-based, where benchmarks handcraft evaluation metrics (e.g., faithfulness or fluency) to evaluate systems on multiple dimensions @xcite @xcite or (ii) arena-based: where systems compete each other in a tournament and an llmbased teacher is used as the judge for evaluation बाइनरी भाषा म िकतने अ र होते ह ? ( @xcite . heuristic-based benchmarks are computationally cheaper to evaluate but require human preferences as gold truth for reference. they also face challenges in aggregating different features into a ranking order for models. on the other hand, arenabased benchmarks require a high-performance or reasoning-intensive llm as a teacher for accuracy @xcite , which makes exhaustive pairwise comparisons expensive for a large set of models. for instance, evaluating a single query on 19 models involves 19 2 = 171 comparisons and costs between $5-10 usd with @xcite . in our work, we get the best of both worlds with a surrogate judge, a learning to rank model, e.g. random forest @xcite , using heuristic features to estimate an arena-based leaderboard obtained with a bradley-terry model @xcite ) from pairwise evaluations using an llm as a judge @xcite . we use bootstrapping to obtain confidence bounds for better statistical estimates. after training, the learning to rank model can be used to estimate the performance of newer released models reliably in the future without the expensive llm as a judge. it also provides better interpretability and is easily retrainable with a different or newer set of heuristic features. we develop mirage-bench, a rag benchmark across 18 languages for multilingual generation evaluation on wikipedia. mirage-bench data is sourced from miracl @xcite , a multilingual retrieval dataset containing humangenerated queries and human-labeled relevance judgments for wikipedia passages. we benchmark 19 frontier multilingual llms in our experiments. our evaluation flowchart adopted in mirage-bench is shown in figure 2 . we evaluate seven heuristic features: (i) language detection, (ii) citation quality, (iii) support, (iv) reranker score, (v) answer overlap (traditional), (vi) answer overlap (llm-measured), and (vii) fluency (llmmeasured). we use gpt-4o as the judge to evaluate our pairwise rag comparisons on a smaller subset consisting of 100 queries. next, we train a random forest model as a surrogate judge, using the heuristic features as input, and learn to output the bradley-terry model logits as output. finally, during inference, we use our surrogate judge to produce a "synthetic" ranking leaderboard for all baselines across every language. more specifically, we ask the following research questions in our work: prior work on rag evaluation has been conducted exclusively in english. for example, benchmarks such as alce @xcite , freshllm @xcite , clapnq @xcite , hagrid @xcite and crag @xcite , all include long-form answers for english-only queries and are based on collections containing documents from either the english wikipedia, ms @xcite or nq @xcite . similarly, trec 2024 rag, a trec competition for rag evaluation is focused on evaluating queries in english. 2 multilingual rag. on the multilingual side, rag has not been well studied in prior literature. the rgb benchmark @xcite is limited in language scope covering only one additional language: chinese (zh). neuclir 2024 track @xcite evaluates long-form report generation from participants, but is limited to three languages. a concurrent work is bergen @xcite , which evaluates the multilingual opendomain qa setting on 13 languages. in contrast, in mirage-bench, we (i) evaluate the generation task in the multilingual rag pipeline on 18 languages, (ii) provide multilingual instruction-tuned data for rag fine-tuning, and (iii) utilize highquality native-speaker multilingual queries generated in miracl @xcite . ## an overview of mirage-bench we select 18 languages in mirage-bench as the starting point, representing an appropriate crosssection of the diversity of the languages spoken worldwide at this point. mirage-bench is a comprehensive multilingual rag benchmark focusing on the generation task evaluation. as shown in ## conclusion we present mirage-bench, a multilingual rag benchmark for 18 languages aimed at evaluating the multilingual generation part within rag and aggregate traditional heuristic-based features to train a lightweight learning to rank model as a surrogate judge to learn a bradley terry model with gpt-4o pairwise judgments. our results indicate a strong correlation between our surrogate judge and gpt-4o as a pairwise judge. this demonstrated the effectiveness of our efficient, cheap, and easy-toretrain random forest model as a surrogate judge trained using only computationally cheap heuristic features for arena-based leaderboard ranking by achieving a 0.909 kendall ω . on mirage-bench, we observe that most proprietary and opensource larger llms currently dominate, whereas smaller open-source llms continue to struggle. instruction tuning on mirage-bench training data helps improve smaller open-source llms, e.g. instruction-tuned mistral-v0.2 (7b) on gpt-4o distilled training data can outperform llama 3 (70b) on mirage-bench. mirage-bench is one of the first large-scale multilingual rag benchmarks. although not perfect, we below discuss a set of limitations in our work: • in mirage-bench, we focused on benchmarking the generation task in rag with oracle passages, we did not consider the retrieval task and its error propagating on the generation task.