I ndic G en B ench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLM s on I ndic Languages (2024)

Abstract: As large language models (LLMs) see increasing adoption across the globe, it is imperative for LLMs to be representative of the linguistic diversity of the world. India is a linguistically diverse country of 1.4 Billion people. To facilitate research on multilingual LLM evaluation, we release IndicGenBench — the largest benchmark for evaluating LLMs on user-facing generation tasks across a diverse set 29 of Indic languages covering 13 scripts and 4 language families. IndicGenBench is composed of diverse generation tasks like cross-lingual summarization, machine translation, and cross-lingual question answering. IndicGenBench extends existing benchmarks to many Indic languages through human curation providing multi-way parallel evaluation data for many under-represented Indic languages for the first time. We evaluate stateof-the-art LLMs like GPT-3.5, GPT-4, PaLM2, and LLaMA on IndicGenBench in a variety of settings. The largest PaLM-2 models performs the best on most tasks, however, there is a significant performance gap in all languages compared to English showing that further research is needed for the development of more inclusive multilingual language models. IndicGenBench isavailable at www.github.com/google-researchdatasets/indic-gen-bench

## introduction with the advances in generative language technologies powered by large language models (llms; @xcite @xcite @xcite , there has been a surge of interest in evaluating the multilingual capabilities of these models. recent work @xcite all authors are now part of google deepmind 2023a,b) shows a consistent performance gap between high resource languages and languages with lower amounts of web resources available. to develop highly multilingual generative llms which should work equally well for 100s of languages spoken by billions of people in the world, it is crucial to evaluate their capabilities across a variety of languages to uncover performance gaps and guide future research. in this work we focus on india, a country with 1369 rationalized mother tongues spoken by more than a billion people. 1 making progress on language technologies for indic languages will not only improve the state of affairs in this region, but will also provide valuable learning to the nlp community which will be applicable to other geographical regions and language families. there are has been much work from the community in building natural language understanding (nlu) models for indic languages @xcite , as well as evaluation datasets (dodda- we release indicgenbench, a multilingual, multi-way parallel benchmark for measuring language generation capabilities across diverse userfacing tasks in 29 indic languages across 4 language families ( as evident from the lists above, our benchmark provides a broad-coverage over languages with respect to their resourcedness, allowing users to evaluate language models on relatively highresource languages such as hindi and extremely low-resource languages such as manipuri in meitei script on a single benchmark. to curate the evaluation datasets for our benchmark, we use the following existing datasets as the source: crosssum (bhattacharjee et al., 2023) for cross-lingual summarization, flo-res @xcite for machine translation, xquad (artetxe et al., 2020) for multilingual qa, and xorqa (asai et al., 2021) for cross-lingual qa. from each of these datasets we select a subset of english examples to be a part of our benchmark, and then collect professional human translations for these examples in all target indic languages. some target languages are already covered by the source datasets in which case we re-purpose this existing data and only collect translations for the remaining languages. we also collect and release a small amount of training and validation data making possible evaluation of training techniques like fine-tuning, parameter-efficient training, in-context learning, and others. why extend existing benchmarks? we chose to collect human translations of existing benchmarks as opposed to creating evaluation data from scratch due to various reasons: annotators were professional data labelers working as contractors at our organization and with a vendor. annotators were paid competitive rates in compliance with applicable labor laws and prevailing market rates. our pay rate to annotators varied across languages, ranging from usd 2.80 per hour for pashto to usd 15.90 per hour for tibetan. ## experiments and analysis we use indicgenbench to benchmark multilingual and cross-lingual language generation capabilities of various llms on indic languages. we perform experiments with a variety of open-source llms -mt5 @xcite , llama (touvron et al., 2023), foot_5 , bloomz @xcite , gemma @xcite ; and proprietary llms -gpt-3.5, @xcite , and palm-2 @xcite . we compare and analyze the performance of different model size variants of these llms under various learning paradigm settings. we first evaluate model performance on one-shot prompting ( §3.1) and also measure performance across language categories based on resourcedness ( §3.2). we then evaluate the effect of number of in-context examples shown to the model as supervised data ( §3.3) and the effect of prompting in a higher-resource language such an english or hindi ( §3.4). using the training data contained in indicgenbench, we measure how the performance of llms after finetuning compares with few-shot prompting ( §3.5). finally, we perform qualitative analysis of models on indicgenbench and highlight some areas of improvement for future model development ( §3.7). evaluation metrics for the cross-lingual summarization and translation tasks, crosssum-in and flores-in, we report character-f1 (chrf) metric (popović, 2015) since token-level metrics like rouge and bleu are not reliable for lowresource languages @xcite . to stay consistent with existing literature on qa tasks, we report squad-style token-f1 on our xquad-in and xorqa-in qa tasks. on flores-in, we report translation performance in both directions-translating from english to the target language (enxx) and vice-versa (xxen). ## related work in the last few years, many multilingual llms have been developed-starting from mbart (liu et al., 2020) trained on 25 languages to llms that are pre-trained on hundreds of languages, such as mt5 @xcite , palm-2 (anil et al., 2023), gpt-4 @xcite , gemini (google, 2023), and others. these llms are typically evaluated on individual multilingual tasks for translation: wmt @xcite , flores @xcite ; question-answering: xquad (artetxe et al., 2020), ty-diqa @xcite , xorqa (asai et al., 2021); summarization: xlsum @xcite ; reasoning: mgsm (shi et al., 2022), xcopa @xcite to name a few, or on multilingual benchmarks such as, xtreme @xcite and xtreme-up @xcite . however, most of these evaluation resources contain only a handful of languages or do not contain data for low resource languages, especially indics. besides, cross-lingual evaluation data is even more sparse. this work is an effort to bridge these gaps by releasing indicgenbench, a suite of datasets covering diverse cross-lingual and multilingual generation tasks in indic languages. most work on creating evaluation data on indic languages have focused on natural language understanding (nlu) tasks. @xcite and @xcite have released nlu test sets in indic languages for a wide variety of tasks such as qa and nli. naamapadam (mhaske et al., 2023) is a named entity recognition dataset specifically for indic languages, massive @xcite is a slot-filling and intent classification dataset available in 7 indic languages, indicglue @xcite ) is an nlu benchmark for 11 indic languages, whereas gluecos (khanuja et al., 2020) is a hindi-english code-mixed benchmark, containing various nlu tasks. the belebele benchmark @xcite is a multiplechoice machine reading comprehension dataset for 122 languages of which 17 are indic. on the other hand, indicgenbench is a natural language generation (nlg) benchmark. recently, there has been work in creating evaluation benchmarks for natural language generation (nlg) on indic languages. indicnlg suite @xcite , consisting of 5 nlg taks in 11 indic languages, is a leap in this direction. these datasets in this suite are automatically created, ei-ther using data from the web (e.g., wikipedia) or using translation systems. there are few works which create evaluation data for individual tasks in indic languages. for example, indictrans2 @xcite creates an n-way parallel dataset for machine translation in 22 scheduled indian languages, mukhyansh @xcite and pmindiasum @xcite are headline generation datasets for 8 and 14 indic languages respectively, and tesum @xcite is an abstractive summarization dataset in the telugu language. ramesh et al. (2022) introduced samanantar, a large translation dataset covering 11 indic languages. our work complements indicnlgsuite and the other datasets in multiple ways. indic-genbench is manually annotated ensuring highquality, noise-free text which is not typically found on the web. our benchmark contains evaluation data for a much larger set of languages spanning low, medium and high resource. our datasets are multi-language parallel enabling better comparison among different languages. lastly, we focus on a complementary and challenging set of tasks, including cross-lingual summarization, cross-lingual and multilingual question answering, and translation. ## conclusion we release indicgenbench, the largest benchmark for evaluating llms on 5 user-facing generation tasks across 29 indic languages, providing evaluation data for many under-represented indic languages for the first time. indicgenbench is broad coverage along many dimensions -it covers 13 writing scripts, 4 language families, and spans languages across the available web resource spectrum. we carry out extensive comparison of current sota llms on indicgenbench and highlight areas for future improvement. we are hopeful indicgenbench will play an important role in further development of llms in indic languages ultimately benefiting a billion-plus population.