T urkish MMLU : Measuring Massive Multitask Language Understanding in T urkish (2024)

Abstract: Multiple choice question answering tasks evaluate the reasoning, comprehension, and mathematical abilities of Large Language Models (LLMs). While existing benchmarks employ automatic translation for multilingual evaluation, this approach is error-prone and potentially introduces culturally biased questions, especially in social sciences. We introduce the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU, to evaluate LLMs’ understanding of the Turkish language. TurkishMMLU includes over 10,000 questions, covering 9 different subjects from Turkish high-school education curricula. These questions are written by curriculum experts, suitable for the high-school curricula in Turkey, covering subjects ranging from natural sciences and math questions to more culturally representative topics such as Turkish Literature and the history of the Turkish Republic. We evaluate over 20 LLMs, including multilingual open-source (e.g., Gemma, Llama, MT5), closed-source (GPT 4o, Claude, Gemini), and Turkish-adapted (e.g., Trendyol) models. We provide an extensive evaluation, including zero-shot and few-shot evaluation of LLMs, chain-of-thought reasoning, and question difficulty analysis along with model performance. We provide an in-depth analysis of the Turkish capabilities and limitations of current LLMs to provide insights for future LLMs for the Turkish language. We publicly release our code for the dataset and evaluation: https://github.com/ArdaYueksel/TurkishMMLU

## introduction benchmarking plays an important role in understanding and measuring the capabilities of language models. recent multitask multiple-choice question answering (qa) benchmarks like mmlu @xcite cover a wide range of use cases for language models, making them highly popular as one of the main evaluation benchmarks question: aşağıdakilerden hangisi 1923'ten sonra türk dış politikasında ele alınıp çözümlenmeye çalışılmış konulardan biri değildir? (which of the following is not one of the issues that were tried to be addressed and resolved in turkish foreign policy after 1923?) l a n g u a g e math natural sciences s o c ia l s c ie n c e s a n d h u m a n it ie s subject: history in recent llms such as gpt 4 @xcite and gemini @xcite . for the multilingual adaptation of the mmlu benchmark, recent works @xcite have focused on automatic translations. however, automatic translations are often prone to errors and may fail to capture the linguistic and cultural nuances of the target language. consequently, there have been manual efforts to create multitask multiple-choice bench-marks in various languages, including arabic (ara-bicmmlu, @xcite , korean @xcite , and chinese @xcite . in our work, we introduce turkishmmlu, the first multitask multiple-choice qa benchmark specifically designed for the turkish language. our dataset includes 10,032 multiple-choice questions, each with five options, spanning nine subjects categorized into four groups: natural sciences, mathematics, turkish language and literature, and social sciences and humanities. these questions are sourced from a high-quality online learning platform created by the turkish ministry of education, which aims to support high school students in preparing for the university entrance exam. a unique feature of turkishmmlu is the correctness ratio, which reflects the actual performance of students on these questions, offering a more accurate measure of question difficulty. we illustrate the distribution of subjects and an example from turk-ishmmlu in figure 1 . after introducing this dataset for benchmarking in turkish, we evaluate a wide range of current language models, more than 40, including multilingual autoregressive llms, both open models like gemma @xcite , llama-3 and aya-23 @xcite and closed-source models such as gpt 4o, claude and gemini. in addition, we also cover multilingual encoder-decoder models such as mt5, mt0, aya and turkish-adapted llms such as trendyol-llm, a lora adaptation of multilingual llms. we also cover many different setups including zero-shot, few-shot, and chainof-thought @xcite . we further provide analysis of llms based on subjects and difficulty. our additional analysis provides insights for the design of future llms for turkish and beyond. we publicly release our code for the dataset and evaluation: https://github.com/ardayueksel/ turkishmmlu . our contributions are as follows: to evaluate the emerging capabilities of the llms, new benchmarks are proposed that focus on more advanced capabilities such as common sense reasoning @xcite , multi-hop reasoning @xcite , programming (chen et al., 2021) and multi-turn conversations. additionally, some studies aimed at evaluating these capabilities through extensive datasets that cover a broad range of knowledge-based topics @xcite . one prominent example is mmlu (massive multitask language understanding) @xcite ; it covers 57 diverse fields from basic arithmetic to intricate areas like legal studies and computer science. although many of these benchmarks have focused on english, there have been significant efforts to adapt and develop similar benchmarks for other languages @xcite @xcite @xcite . turkish benchmarks: one of the initial efforts in turkish benchmarking was thquad (soygazi et al., 2021), a variant of the squad questionanswering benchmark (rajpurkar et al., 2016) that focuses on extracting information from historical passages and answering questions about ottoman and islamic history in an open-book format. mukayese (safaya et al., 2022), another turkish benchmark, was created by combining multiple existing datasets for various tasks. however, most of the tasks that are included in mukayese, such as ner (named entity recognition), sentence segmentation and spellchecking, do not effectively capture the knowledge and the language understand-question set / 12th grade / biology / multiple choice / test -1 12th grade biology question test -1 number of correct answers: 3 number of wrong answers: 7 number of empty answers: 0 l. bone marrow ll. eye cell lll. human egg cell lv. umbilical cord blood which of the above are used as stem cell sources? your answer: b show solution which of the following methods are used to obtain the insulin hormone that regulates blood sugar in humans? i. gene cloning ii. polyploidy iii. isolation from animals your answer: d show solution question 1: 3740 a) only i b) only ii c) only iii d) i and iv e) iii and iv question 6: 3745 a) only i b) only ii c) only iii d) i and iii e) i, ii and iii figure 2: sample biology test from the eba platform (translated to english, see figure 5 in appendix for the original turkish test). black boxes indicate the correctness ratio (difficulty level). green borders appear when the user's choice matches the ground-truth answer, while red borders indicate incorrect choices. ing capabilities of llms due to their low level nature. several other studies that created multilingual benchmarks for specific tasks such as xcopa (cross-lingual choice of plausible alternatives) (ponti et al., 2020) and xnli (cross-lingual natural language inference) (conneau et al., 2018) also include turkish among several other languages. a recent study that focuses on turkish llms (acikgoz et al., 2024) created the turkish versions of the truthfulqa multiple choice (mc) (lin et al., 2022) and arc (ai2 reasoning challenge) (clark et al., 2018) datasets to evaluate turkish llms. these benchmarks are constructed by machine translating the english versions of the corresponding datasets, which is usually followed by manual verification and editing to ensure good quality. overall, despite some efforts to evaluate the capabilities of llms for turkish, turkish still lacks a high quality and comprehensive evaluation resource that covers multiple domains. in this study, we address this by introducing turkish mmlu. ## dataset turkishmmlu is curated using resources from online learning materials for turkish high school education. in the turkish educational system, high school education spans four years, and students take the national university entrance exams af-ter completing their studies. this exam contains multiple-choice questions covering various subjects from the curricula. to assist students in preparing for these exams, official and commercial exam preparation booklets, video guides, and online practice tests in multiple-choice question-answering format are available. the turkish ministry of education (meb) has developed an online platform called the education information network (eba), which aims to provide electronic resources such as lecture notes, videos, tests and solutions, and interactive books to facilitate the learning process for students. this platform 1 contains multiple-choice questions and their solutions that form the basis of our study. the questions are driectly sourced from the platform which ensures that each question is crafted and verified by domain experts, guaranteeing correctness and high quality. we conducted repeated manual reviews, randomly selecting 30 questions from the complete dataset during each review session to assess their formatting and accuracy. whenever we detected formatting issues-typically due to irregular presentation on the source website-we revised our parsing code to effectively address these edge cases. this process was continued until we ensured con-sistent formatting accuracy across multiple consecutive rounds of the sampled questions. figure 2 illustrates the eba platform interface. users generate tests by specifying grade level and subject, upon which the platform provides multiple 10-question tests. after test completion, users can review ground-truth answers and video solutions. each question's difficulty is denoted by a correctness ratio (black boxes in figure 2 ), calculated as the percentage of correct user responses. for each test, we extract question text, multiple-choice options, correct answer, topic, subject, grade, and difficulty level. figure 3 displays the distribution of correctness ratios. questions are categorized as easy (top 30%), medium (middle 40%), or hard (bottom 30%), with percentile thresholds at 41 and 28, respectively. we manually selected 25 questions per subject for the development set, maintaining subject-grade distributions and mirroring the overall difficulty distribution. for few-shot examples, we focus on 5shot experiments with 5 questions per subject due to context window constraints and compute budget limitations, each with different correct answers to avoid selection bias. for chain-of-thought (cot) prompting, we manually provide step-by-step solutions for these 5 questions per subject. the large scale of our test dataset, including 9,807 questions, raises significant challenges. experiments with state-of-the-art proprietary models like gpt 4 and claude-opus face budget constraints, while using chain-of-thought (cot) prompting with open-source models generates excessively long responses, resulting in long inference times. to address these issues while maintaining comprehensive evaluations, we create a smaller version of turkishmmlu, called turkishmmlu sub with 100 randomly selected questions per subject, totaling 900. we uniformly sampled 25 questions per grade for each subject, except for philosophy, which has 50 questions evenly distributed between grades 10 and 11. this sample is representative of grades and subjects, enabling in-depth model evaluation, but can be easily used in resource-constrained scenarios. we measure the correlation between turkishmmlu sub and turkish-mmlu in §4.5, finding a strong correlation across 32 models. ## evaluation results after finalizing turkishmmlu, we now evaluate various multilingual and turkish-adapted openand closed-source llms. we cover a wide range of models, from 60m to 141b parameters, and various experimental setups. experimental setup our main evaluation setup is 5-shot in-context learning evaluation, following the prior evaluation setups in recent llms @xcite on english mmlu @xcite . from the development set proposed in §3, we select a fixed set of questions for each subject and include 5 of them in our few-shot prompt, with the question, multiple-choice options, and the answer. we carefully design these prompts to ensure that each question has a different option (in our dataset, the five options are always a, b, c, d, e) as the answer. for evaluation, we report accuracy by using the lm-evaluation-harness framework from eleutherai @xcite . for open-source models, we perform log-prob based evaluation; for closed-source models we perform greedy decoding and then parse the prediction. in order to ensure validity of parsing, we utilize regex patterns based on the fewshot example structure which can be formulated as question: ...; choices: a. ... e. ... ; answer: [a-e] . our second evaluation is a zero-shot evaluation to compare few-shot and zero-shot performance of the models. additionally, we evaluate llms with a 5-shot chain-of-thought (cot) evaluation. especially for questions requiring further reasoning and elaboration, such as mathematics, directly giving answers may be a limitation in our main evaluation. therefore, we evaluate a wide range of models, including closed-source models, with cot reasoning @xcite . in this setup, we provide cot solutions for each question in our few-shots for each subject and perform greedy decoding. we put the final answer option at the end of the solution in the prompts, and then parse the predicted answer in the generated solution. we formatted the cot fewshot examples as they would end with the turkish phrase of the correct choice is @xcite . this way we can parse the predicted answer easily from the generated solution. since turkishmmlu includes real-world data for difficulty, we also conduct a difficulty analysis to evaluate models. this expands our evaluation setup from comparing models on different subjects to varying difficulty levels. in all of our evaluations, we use a small subset of turkishmmlu, turkishmmlu sub , because the closed-source experiments are quite expensive. 2 with public mod-els, we calculate performance on both turkish-mmlu and turkishmmlu sub to test our assumption that they would yield similar results. language models: we evaluate a diverse range of models, including turkish-adapted, multilingual open-source and closed-source llms. for turkish-adapted models, we use trendyol-llm 7b, a llama-2 model further pretrained on turkish foot_2 , available in base, chat, and chat-dpo forms on huggingface. we also include kanarya @xcite , a pretrained autoregressive 2b turkish model. in the multilingual open-source category, we evaluate models with encoder-decoder architectures such as mt5 @xcite (from small to xxl), mt0 @xcite (with the same sizes as mt5), and cohere's aya-101 @xcite . for autoregressive models, we include meta's llama-2 @xcite for multilingual closed-source models, we evaluate openai's gpt models (3.5, 4-turbo, and 4o), anthropic's claude-3 models (haiku, sonnet, and opus versions), and google's gemini models (pro versions 1.0 and 1.5). ## conclusion in this study, we introduced turkishmmlu, the first turkish multitask question answering benchmark designed for evaluating llms. our dataset consists of 10,032 multiple-choice questions covering nine subjects from the turkish high school curriculum and university entrance exams, complete with correctness ratios to indicate question difficulty. we evaluated a wide range of llms, including turkish-adapted and multilingual models, in various setups such as zero-shot, few-shot, and chain-of-thought reasoning. our results highlighted the superior performance of closed-source models like gpt 4o and claude-3 opus and the notable improvements in newer open-source autoregressive models like llama-3 70b-it. the benchmark demonstrates significant performance variation by subject and question difficulty, emphasizing the strengths and limitations of current llms in understanding and reasoning in turkish. furthermore, as llms mature, it will become increasingly crucial to shift the focus of the field from english to broader coverage of the languages of the world. we see turkishmmlu as a promising contribution towards ensuring that all language communities will be equally served by nlp in the future.