M i L i C -Eval: Benchmarking Multilingual LLM s for C hina’s Minority Languages (2025)

Abstract: Large language models (LLMs) excel in high-resource languages but struggle with low-resource languages (LRLs), particularly those spoken by minority communities in China, such as Tibetan, Uyghur, Kazakh, and Mongolian. To systematically track the progress in these languages, we introduce MiLiC-Eval, a benchmark designed for minority languages in China, featuring 24K instances across 9 tasks. MiLiC-Eval focuses on underrepresented writing systems. Its parallelism between tasks and languages can provide a faithful and fine-grained assessment of linguistic and problem-solving skills. Our evaluation reveals that open-source LLMs perform poorly on syntax-intensive tasks and multi-script languages. We further demonstrate how MiLiC-Eval can help advance LRL research in handling diverse writing systems and understanding the process of language adaptation.

## introduction large language models (llms) have achieved remarkable success in high-resource languages such as english and chinese, demonstrating the ability to perform sophisticated tasks, including creative writing @xcite , complex planning @xcite , and scientific reasoning @xcite @xcite . however, thousands of low-resource languages (lrls) remain underexplored by llms, which exhibit significant limitations in their basic linguistic capabilities for these languages @xcite . this challenge is particularly pronounced for minority languages spoken in china, such as tibetan (bo), uyghur (ug), kazakh (kk), and mongolian (mn). although spoken by tens of millions, these languages are largely marginalized in nlp research due to their limited digital representation and the scarcity of training data @xcite . furthermore, their use of non-latin scripts, including non-mainstream ones like those in mongolian and kazakh, poses additional challenges for tokenization and language modeling. currently, there are no standardized benchmarks for evaluating llms on these languages. to fill this gap, we introduce milic-eval, a comprehensive framework with 24k instances across 9 tasks, focusing on 4 minority languages in china. it overcomes limitations in previous lrl benchmarks such as inadequate attention to low-resource writing systems and unsystematic task lineups @xcite @xcite . milic-eval offers the following key features: (1) focus on underrepresented writing systems such as traditional mongolian and tibetan scripts; (2) faithful evaluation with parallel tasks across languages and formats; (3) fine-grained skill evaluation, assessing both linguistic and problem-solving abilities, from vocabulary to symbolic reasoning, as shown in figure 1 . our evaluation of open-source multilingual llms reveals that they especially struggle with languages using less common writing systems, such as mongolian. skill-wise evaluation shows that llms have basic lexical knowledge in these languages and can perform simple tasks such as topic modeling. however, they still lack the abilities for generation and translation, which require intensive syntactic knowledge. finally, we discuss how milic-eval can serve as a valuable resource for advancing lrl research. first, milic-eval can facilitate the development of multilingual llms that are inclusive and robust across diverse writing systems. our findings highlight inefficiencies in current llms, particularly in tokenization and the handling of multiple scripts, which pose significant challenges for lrls with unique or complex writing systems. second, milic-eval provides a faithful and reliable way to evaluate models' abilities in lrls. its taskparallelism design prevents biased conclusions resulting from over-reliance on a single task format. the human-translated data avoids the misinterpretation of model performance caused by the noise in machine-translated data. third, milic-eval enables a deeper investigation into how llms acquire and exhibit various abilities during language adaptation. using milic-eval, we reveal limitations in prevailing practices, calling for more effective techniques for lrl adaptation. our main contributions are as follows: (1) we introduce milic-eval, a standardized benchmark for lrls in china, emphasizing underrepresented writing systems. (2) we provide a skill-wise evaluation of state-of-the-art multilingual llms, uncovering their limitations in handling syntax-intensive tasks and multiple writing systems. (3) we demonstrate the utility of milic-eval in advancing lrl research, offering transferable insights for the study of lrls in other underrepresented regions. ## milic-eval we introduce milic-eval, the first standardized benchmark for evaluating llms' performance on four minority languages in china. the benchmark includes 9 distinct tasks with 24k instances. basic information about each task is provided in ## evaluating multilingual llms we conduct a systematic evaluation of existing multilingual llms with milic-eval and analyze the current progress in china's minority languages. ## conclusion we present milic-eval, a multilingual benchmark comprising 24k instances across 9 tasks and 4 minority languages in china. milic-eval is distinguished by three key features: (1) a focus on underrepresented writing systems, (2) cross-language and cross-task parallelism, and (3) fine-grained skill-wise evaluation. we hope that milic-eval will not only advance llm support for lrls in china but also inspire research on underrepresented languages in other regions, such as africa, india, and southeast asia.