MERA : A Comprehensive LLM Evaluation in R ussian (2024)

Abstract: Over the past few years, one of the most notable advancements in AI research has been in foundation models (FMs), headlined by the rise of language models (LMs). However, despite researchers’ attention and the rapid growth in LM application, the capabilities, limitations, and associated risks still need to be better understood. To address these issues, we introduce a new instruction benchmark, MERA, oriented towards the FMs’ performance on the Russian language. The benchmark encompasses 21 evaluation tasks for generative models covering 10 skills and is supplied with private answer scoring to prevent data leakage. The paper introduces a methodology to evaluate FMs and LMs in fixed zero- and few-shot instruction settings that can be extended to other modalities. We propose an evaluation methodology, an open-source code base for the MERA assessment, and a leaderboard with a submission system. We evaluate open LMs as baselines and find they are still far behind the human level. We publicly release MERA to guide forthcoming research, anticipate groundbreaking model features, standardize the evaluation procedure, and address potential ethical concerns and drawbacks.

## introduction recent advancements in nlp have led to the emergence of powerful large language models (llms), which showcase unprecedented tasksolving capabilities. in recent years, ai research has made notable progress in foundation models (fms) @xcite trained on extensive data and adaptable to various downstream tasks. interacting with humans through free-form text instructions, these models serve as versatile text interfaces for multiple scenarios, transforming the landscape of ai systems. the swift evolution of models provokes critical questions regarding their comprehensive evaluation, spanning natural language understanding, ethical considerations, expert knowledge, etc. the most recent research @xcite underscores the crucial need for a standardized evaluation protocol encompassing diverse metrics and potential usage scenarios to address risks associated with ai adoption. the community has addressed the issue with several recently created benchmarks: big-bench @xcite , helm @xcite , mt-bench @xcite which test models' expert knowledge, coding skills and advanced abilities beyond the scope of classic gluestyle @xcite benchmarks. however, most of these recent benchmarks are constructed for the english language. russian, at this point, lacks a fair instrument for transparent and independent llm evaluation. benchmarks like russian superglue @xcite and tape @xcite do not cover the entire scope of modern llm abilities. current russian benchmarks should be revised to satisfy recent trends and challenges and to foster an understanding of llms' behavior. this paper addresses the problems above and presents the benchmark mera foot_0 . the project, led by ai alliance russia foot_1 , represents a pioneering collaboration between industry and academic partners. mera plays a crucial role in fostering cohesion between the scientific community and industry, thus maintaining the benchmark's independence and impartiality. this novel benchmark comprises 21 tasks covering 10 skills in the instruction format, offering a comprehensive standardized evaluation of llms and fms in russian. the primary objective of this project is to establish a reliable methodology for assessing foundation models in zero-shot and few-shot instruction settings under fixed evaluation scenarios (see fig. 1 for mera general idea description). the current benchmark methodology and taxonomy are presented for textual data and sub-modalities, such as code and formal languages. the methodology is versatile and can be applied to different modalities. we plan to extend the benchmark to incorporate images and audio in the upcoming mera releases. thus, the contribution of our work can be summarized as follows: benchmarks, such as glue @xcite and superglue @xcite , have been the standard evaluation tools for measuring nlp progress for the last 5 years. however, recent studies @xcite @xcite have criticized their canonical approach for being too shallow and for possible data leakage. moreover, given the development of llms and fms, current benchmarks are now considered not challenging enough for modern llms, which have outperformed the human level for most of the included tasks. thus, there is a need for more challenging benchmarks that follow the instruction format relevant to modern instruction-based models. to address these problems, the community has proposed several new benchmarks evaluating llms in various settings and scenarios: big-bench foot_2 @xcite , a massive benchmark comprising more than 200 tasks, is intended to probe llms and extrapolate their future capabilities; helm 4 @xcite tests llms' generalization abilities in multiple languages and contains an extensive detailed system of metrics for various evaluation scenarios; instructeval 5 @xcite provides a comprehensive evaluation methodology for instruction-tuned llms. in addition, there is a strong move @xcite @xcite towards assessing a model's professional knowledge and expertise through exam tasks. besides, there is a trend @xcite on using the llm-as-a-judge evaluation approach when llms (e.g., gpt-4 foot_5 @xcite ) are used to score models in a generation setup instead of utilizing automatic metrics (e.g., bleu @xcite ) or human evaluation. however, the standard metrics for generative evaluation were criticized @xcite @xcite a lot for being not representative enough. while benchmarks with the sys-tems model-as-a-judge @xcite 7 could successfully evaluate a model, they have biases, making human judgment, which is expensive and unclear in terms of funding, more reliable. several benchmarks were introduced to target at even more complex problems, such as multimodal knowledge and reasoning @xcite , in-context learning @xcite , software development @xcite , general assistants @xcite , social reasoning @xcite , and alignment skills @xcite ). an extensive survey of current benchmarks and open challenges is presented in @xcite . however, one of the limitations of the benchmarks mentioned above is that they are mainly oriented on the english language. as for russian, there is still a need for a system able to evaluate modern llm abilities reliably. the main benchmarks for russian remain russian superglue (rsg) @xcite , tape @xcite , and rucola @xcite , which do not challenge the modern llms enough or cover the scope of their recently emerging capabilities (e.g., expertise in science fields or coding skills). more and more tasks in rsg are already solved by lms better than by an average human, and only a few remain challenging (e.g., rwsd); the best lms' scores on rucola are close to the human results. as for the modern benchmarks that sufficiently challenge llms and fms' abilities, there is the rulm-sbs foot_7 benchmark which follows the llm-as-a-judge approach, thus being expensive in evaluation. to summarize, there is an urgent need for an objective system to evaluate modern llms' abilities in russian independently. ## data the mera benchmark unites various datasets and benchmarks, which results in 21 tasks covering 10 skills for llm and fm evaluation in russian. based on the previous experience of llm benchmarking @xcite , we include tasks of three categories in terms of evaluation objective and data origin: based on the taxonomy above and modern practices @xcite , we chose 21 tasks that test advanced lms and fms' capabilities that can be evaluated via automatic metrics, which we attribute to 10 skills derived from categorizations described in @xcite ; @xcite ; @xcite . the tasks are formulated in the instruction format, targeting various answer types: classification problems (9 tasks), multiple choice questions (5 tasks), free-form answers (8 tasks), and matching (1 task). see tab. 1 for the general task information; the detailed task description can be found in app. a. all tasks comprise at least a test set with closed answers. the exception is the diagnostic datasets whose answers are made public since they are not used in the final assessment. for four datasets (lcs, ruhumaneval, rummlu, rumodar), we also release a public test set adapted from the original public tests of the corresponding datasets. we invite the community to use these datasets as public tests for general research purposes. for some other tasks, we additionally publish sets marked as training and validation (or dev) sets. we do this for the following reasons: 1) these sets can be used as a source for few-shot examples; 2) for the general consistency of the sets adapted from other publicly available datasets (e.g., rsg, big-bench). nevertheless, in line with the big-bench paradigm @xcite and according to the rules of the leaderboard, it is prohibited to use benchmark data in model training. some tasks were created from scratch for mera, while others represent adapted and enriched versions of previously published russian and translated english datasets. for some tasks, task name test origin answer type skills train* dev test prompts problem-solving mathlogicqa new multiple choice mathematics, logic 680 -1143 10 multiq tape free-form reasoning 1056 -900 5 parus rsg classification common sense 400 100 500 12 rcb rsg classification nli 438 220 438 9 rumodar new free-form mathematics, logic 6000 -6000 5 rumultiar new free-form mathematics 1039 -1024 6 ruopenbookqa tape multiple choice world knowledge 2338 -400 10 rutie new classification reasoning, dialogue system 430 -430 5 ruworldtree tape multiple choice world knowledge 115 -525 10 rwsd rsg classification reasoning 606 204 260 10 simplear new free-form mathematics 1000 -1000 6 exam-based bps new classification algorithms 250 -1000 8 chegeka tape free-form world knowledge 29376 -416 4 lcs new classification algorithms 320 -500 6 ruhumaneval new free-form computer code 164 -164 10 rummlu new multiple choice reasoning 10033 -961 5 use adapted multiple choice, free-form, matching reasoning, nli, world knowledge 2622 900 900 3x5** ethics rudetox adapted free-form ethics 6948 -800 8 ruethics tape classification ethics --645 5x3** ruhatespeech new classification ethics --265 10 ruhhh adapted classification ethics --178 10x3** ## conclusion the rapid development of llms and fms has created new challenges for model evaluation. to adopt the best practices of recent benchmarks for russian, we have introduced mera, which comprises 21 textual tasks covering 10 skills in the instruction format and evaluates the complex abilities of llms, ranging from natural language understanding to expert knowledge, coding skills, and ethical biases. we also have provided a methodology for robust evaluation and scoring. the contribution encompasses a code base that standardized the experimental setup, ensuring reproducibility, and a website 16 featuring an auto-mated submission procedure, scoring system, and open leaderboard. the datasets and code base are published under the mit license. in the future, we plan to involve new evaluation scenarios in mera, specifically incorporating generative tasks. as a crucial next step, to facilitate a comprehensive evaluation of multimodal fms, we intend to extend mera with other modalities like images and audio, employing the tasks taxonomy elaborated on in this work. we aim to address any missing scenarios and encourage the community to contribute. our goal is to inspire the community to share their experience in model evaluation, fostering the development of more robust and reliable models for russian.