MILU : A Multi-task I ndic Language Understanding Benchmark (2025)

Abstract: Evaluating Large Language Models (LLMs) in low-resource and linguistically diverse languages remains a significant challenge in NLP, particularly for languages using non-Latin scripts like those spoken in India. Existing benchmarks predominantly focus on English, leaving substantial gaps in assessing LLM capabilities in these languages. We introduce MILU, a Multi-task Indic Language Understanding Benchmark, a comprehensive evaluation benchmark designed to address this gap. MILU spans 8 domains and 41 subjects across 11 Indic languages, reflecting general and culturally specific knowledge. With an India-centric design, incorporates material from regional and state-level examinations, covering topics such as local history, arts, festivals, and laws, alongside standard subjects like science and mathematics. We evaluate over 42 LLMs, and find that current LLMs struggle with MILU, with GPT-4o achieving the highest average accuracy at 74 percent. Open multilingual models outperform language-specific fine-tuned models, which perform only slightly better than random baselines. Models also perform better in high resource languages as compared to low resource ones. Domain-wise analysis indicates that models perform poorly in culturally relevant areas like Arts and Humanities, Law and Governance compared to general fields like STEM. To the best of our knowledge, MILU is the first of its kind benchmark focused on Indic languages, serving as a crucial step towards comprehensive cultural evaluation. All code, benchmarks, and artifacts are publicly available to foster open research.

## introduction recent advancements in large language models (llms) have reshaped the field of nlp, by enabling these models to perform a variety of tasks across diverse domains @xcite @xcite . while many llms now claim to support multiple languages, there is still a huge discrepancy in their performance in english and other languages @xcite . particularly languages using non-latin scripts, such as those in india, are affected the most by this discrepancy @xcite . one key reason for this is the absence of high-quality benchmarks for these languages. well-designed benchmarks are crucial in driving model development by revealing limitations and guiding improvements @xcite . however, most existing benchmarks focus primarily on english, leaving significant gaps in evaluating llms capability in low-resource and linguistically diverse languages. india's diversity presents a unique challenge for these models. with over 1.4 billion people speaking more than 120 languages and around 19,500 dialects across 28 states @xcite , many of which are underrepresented in nlp research, the need for not just linguistic but culturally appropriate benchmarks becomes urgent @xcite . standard benchmarks like mmlu @xcite and agieval @xcite , while useful for evaluating general world knowledge, falls short in capturing these intricacies of india. each state in india has its own history, traditions, festivals, and art forms, forming a rich cultural mosaic. translating existing english benchmarks into indian languages fails to capture this knowledge, which is required for real-world applications. given the increasing deployment of llms in tasks that directly impact local populations, the need for a benchmark that evaluates both linguistic competence and cultural understanding has become more pressing than ever. in this work, we introduce milu-multi-task indic language understanding benchmark, a comprehensive evaluation dataset designed to address these gaps. milu spans 8 domains and 41 subjects across 11 indic languages, reflecting both general and culturally specific knowledge. we designed milu with an india-first perspective by collecting questions from various national, state, and regional exams. these questions include culturally relevant subjects such as local history, arts, festivals, and laws, alongside traditional academic subjects like science. following previous efforts @xcite , we create this benchmark by collecting questions from over 1500 competitive exams from india. we focus on regionspecific exams to authentically capture local knowledge in the respective language. we evaluate 45 different llms -a mix of closed proprietary, open-source, and language-specific models-on milu. our findings suggest that models struggle with milu, with gpt-4o achieving the highest average accuracy at 74%. interestingly, open multilingual models outperform languagespecific models, which only achieve slightly better than random scores. our analysis of in-context learning reveals that adding more examples improves performance in base models, but the effect on instruct models remains inconclusive. we also explore how performance scales with the number of parameters, finding significant improvements as model size increases. our domain-wise analysis reveals that models perform poorly in culturally relevant areas, such as arts & humanities and social sciences, compared to more general fields like stem. all the artifacts will be released publicly. ## milu: the indicmmlu benchmark in this section, we describe the collection process ( §3.1), the cleaning and filtering process ( §3.2), and present the analysis ( §3.3). ## experimental setup we evaluate 42 different models on milu, including large proprietary models, open-source multilingual models, and popular fine-tuned models specific to indic languages. both the base versions and instruction fine-tuned variants of these models, wherever applicable, are evaluated to measure the improvements gained from fine-tuning. all models, except for proprietary models and llama-3.1-405b, are tested under 0-shot, 1-shot, and 5-shot setups. we maintain a separate validation set of approximately 9,000 questions to serve as examples for few-shot evaluations. for non-api-based models, we use the lm-evaluation-harness @xcite to ensure clean and reproducible evaluations. we use the log-likelihood method, where the probability of a given output string is computed by conditioning it on some provided input @xcite . specifically, the loglikelihood of an answer (a) given the question (x), i.e., logp (a|x), is calculated by concatenating the answer (a) with question (x), and then summing up the log probabilities, of each target token. for multiple choice questions, given k possible answer strings, we select the answer string (a i ) with the highest conditional log probability, i.e., argmax(logp (a 1 |x), ..., logp (a k |x)). the api-based models are evaluated using the generative approach due to the lack of support for prompt log probabilities. we explicitly prompt these models to generate the correct response in a structured json format to simplify response parsing. due to the high costs involved, these models are evaluated only in the zero-shot setup. ## results and discussions in this section, we discuss the results and our findings across all the experiments conducted. ## conclusion in this paper, we introduced milu-multilingual indic language understanding benchmark-a comprehensive benchmark specifically designed to evaluate llms across 11 indic languages, spanning to foster open research, we release milu along-side all code, and other artifacts. as llms continue to become pivotal in modern applications, we hope that milu offers a foundational benchmark for developing more inclusive, culturally aware models that perform well across both general and culturally relevant domains.