Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on T urkish (2025)

Abstract: The reliance on translated or adapted datasets from English or multilingual resources introduces challenges regarding linguistic and cultural suitability. This study addresses the need for robust and culturally appropriate benchmarks by evaluating the quality of 17 commonly used Turkish benchmark datasets. Using a comprehensive framework that assesses six criteria, both human and LLM-judge annotators provide detailed evaluations to identify dataset strengths and shortcomings.Our results reveal that 70% of the benchmark datasets fail to meet our heuristic quality standards. The correctness of the usage of technical terms is the strongest criterion, but 85% of the criteria are not satisfied in the examined datasets. Although LLM judges demonstrate potential, they are less effective than human annotators, particularly in understanding cultural common sense knowledge and interpreting fluent, unambiguous text. GPT-4o has stronger labeling capabilities for grammatical and technical tasks, while Llama3.3-70B excels at correctness and cultural knowledge evaluation. Our findings emphasize the urgent need for more rigorous quality control in creating and adapting datasets for low-resource languages.

## introduction natural language processing has made significant advances in recent years, with large language models achieving impressive results in various tasks @xcite . however, the quality and reliability of these models mostly depend on the datasets used for training and evaluation @xcite . for languages with relatively low resources, such as turkish, the availability of high-quality datasets is crucial for developing robust and accurate systems. turkish natural language processing resources are significantly based on datasets translated from english or adapted from multilingual resources @xcite @xcite . although these datasets enable progress in low-resource natural language processing, their quality and suitability for specific tasks are not thoroughly examined. the use of translated or adapted datasets raises concerns about their adherence to grammar, cultural nuances, and overall coherence, potentially leading to biased or inaccurate model performance. motivated by the need for reliable and culturally appropriate benchmarks in natural language processing, this study aims to evaluate the quality of commonly used data resources in turkish as a case study. this evaluation is crucial to advance the field of low-resource natural language processing by identifying potential shortcomings. to address this gap, we present a comprehensive analysis of 17 widely used turkish datasets. our rationale for selecting these datasets is that they are widely used datasets in the literature or published within popular benchmarks (see our findings reveal that 70% of the benchmark datasets fail to meet our criteria, and 85% of the criteria are not satisfied by these datasets. llm judges are not as effective as human annotators, particularly in understanding cultural common sense knowledge, and interpreting fluent and unambiguous text. our results emphasize the importance of developing high-quality and novel benchmark datasets for more accurate and culturally sensitive settings. the observations are valuable not only for the turkish language but also for all languages that need high-quality data resources in terms of correctness, grammar, and cultural sensitivity foot_0 . the gsm1k dataset @xcite evaluates the performance of language models on reasoning tasks. the dataset is kept private to prevent contamination. they conducted a three-stage annotation process that includes an initial review by experienced annotators, a secondary validation by independent annotators, and a final audit by a dedicated quality assurance team. contamination data contamination in large language models has become an increasing concern. as models are trained on large-scale datasets scraped from the internet, the integrity of benchmark datasets is challenging to maintain. @xcite emphasize the critical need to assess whether a model's performance is due to its genuine reasoning capabilities or mere memorization. contamination is detected by matching test splits with training data. @xcite employ exact match detection methods, normalizing text for capitalization and punctuation to identify instances of overlap. @xcite , on the other hand, use n-gram overlap to measure contamination. the mega benchmark @xcite has a comprehensive case study on contamination by detecting potential training data leakage. they show that some of the benchmark datasets, which were translated into turkish and analyzed in this study, exhibit data contamination. annotation guideline several studies have established guidelines for human evaluation to ensure consistency and reliability. @xcite emphasized the importance of structured annotation guidelines to provide a clear and replicable evaluation criteria. liang et al. ( 2023 ) designed annotation guidelines to assess disinformation scenarios. to maintain annotation reliability, they implemented quality control measures including hidden "secret words" in instructions to verify comprehension and attention checks to detect careless responses. llm-as-a-judge @xcite propose the method of using powerful llms to label and score from a group of candidates. @xcite introduced the judge-bench, a benchmark that evaluates llm's abilities to replicate human judgments. this benchmark incorporates 20 diverse datasets, each focusing on different tasks and annotation methods. their findings reveal that while llms can effectively align with human judgments in specific tasks, their performance varies significantly across different tasks. @xcite proposed that using a smaller group of llms together, called llm jury, instead of relying on a single model would yield a higher correlation with human evaluation. this alternative approach reduces costs while improving reproducibility and applicability. @xcite introduced big-bench, a collaborative benchmark comprised of 204 diverse tasks designed to evaluate the capabilities of llms. their findings show that large models struggle with tasks that require complex reasoning and understanding, and llm performance is relatively worse than human annotators. our differences this study evaluates the quality of llm benchmarks in a comprehensive framework that includes multiple criteria. we conduct a use case study on popular turkish datasets for this framework. the approach described in this study can be generalized to other languages that suffer from having low resources and cultural sensitivity. ## datasets this study examines 17 datasets, listed in xquad xquad @xcite ) is a multilingual open-ended reading comprehension benchmark. the dataset includes 1.190 question-answer pairs from the squad v1.1 benchmark @xcite and human translations from the english text to 10 different languages including turkish. the model is evaluated based on its capability to extract correct answers from a given passage. xcopa xcopa (ponti et al., 2020) is a multilingual dataset of common sense causal reasoning. xcopa is the human translation of the verification and test sets of copa @xcite to 11 languages including turkish. this dataset evaluates the model based on its understanding of causal relations and inferential capability. belebele belebele @xcite is a multiple-choice and multilingual reading comprehension benchmark. the multilingual passages are obtained from flores-200 (nllb team, 2022), and the questions were written by humans. the benchmark was translated from english into other languages including turkish, resulting in a 122 language multilingual dataset. belebele evaluates the llm model's understanding of the information given in the text. xl-sum xl-sum @xcite is a multilingual summarization benchmark. the dataset spanning 44 languages was created with a similar process as xsum @xcite . in addition, the quality of the summaries in 10 languages were evaluated by human annotators. this benchmark aims at abstractive summarization in which the summary can have new phrases that are not present within the original text. xnli xnli @xcite is a multilingual natural language inference benchmark. this dataset is obtained from the human translations of @xcite into 15 languages. model is evaluated on the basis of their ability to recognize textual entailment. turkish plu turkish plu (uzunoglu and şahin, 2023) is a language understanding benchmark based on turkish wikihow, having six subsets as follows. goal inference evaluates the model's ability to identify the overarching goal dataset size cite dload bench. xquad 1.190 791 5k xtreme mega cetvel xcopa 600 250 6k mega cetvel belebele 900 79 14k cetvel xl-sum 34k 365 114k mega cetvel xnli 400.2k 1.4k 14k xtreme mega xglue cetvel turkish plu linking 1.759 4 48 cetvel turkish plu goal infer 260.8k 4 213 cetvel turkish plu step infer 129.6k 4 190 cetvel turkish plu step ordering 550k 4 128 cetvel turkish plu next event prediction 93k 4 130 cetvel turkish plu summarization 125k 4 -cetvel wikiann 40k 511 63k xtreme mega udpos v2.5 9.4k 142 -xtreme mega mkqa 10k 148 284 cetvel offenseval tr-2020 35.2k 177 391 cetvel sts-b-tr 8.6k -397 cetvel mmlu-pro-tr 11.9k -180 - mkqa mkqa @xcite is a multilingual question answering benchmark that includes human translates from the english natural questions (nq) @xcite , where the questions are obtained from google queries. the model is evaluated on the basis of their ability to respond correctly to knowledge-based questions. offenseval-tr 2020 çöltekin (2020) have sentences extracted from turkish tweets that are labeled as offensive or non-offensive. the dataset also breaks down the offensive label into two as targeted and not-targeted. targeted label is further split into group, individual, and other. stsb-tr stsb-tr @xcite is a semantic textual similarity benchmark in turkish, which is machine-translated from stsb english dataset @xcite . two sentences are given and a decimal score between 0.0 and 5.0 is the target prediction, where a score closer to 5.0 means that the sentences portray more similar meaning. @xcite , which is the updated version of mmlu @xcite . the translation is provided by gemini 1.5 pro with human oversight. mmlu-pro-tr also includes hand-picked stem problems, theoremqa, and scibench in addition to mmlu-pro. ## methods in this section, we present our criteria for assessing the quality of datasets. we then explain two types of evaluation; human annotations and llm-judge. ## conclusion this study evaluated the quality of 17 commonly used turkish benchmark datasets. our findings re-veal that 70% of the benchmark datasets fail to meet our criteria, and 85% of the criteria are not satisfied by these datasets. the successful datasets include mmlu-pro-tr, tr-plu next event prediction, tr-plu step inference, wikiann, and xcopa, while the successful criterion is the correctness of technical term usage. these results highlight the need for more rigorous quality control in curating datasets for low-resource languages. we also considered llm-judge annotations as an alternative to human annotations. our results show that llm judges are not as effective as human annotators, particularly in understanding cultural common sense knowledge, and interpreting fluent and unambiguous text. in addition, gpt-4o demonstrates stronger labeling capabilities for grammatical and technical tasks, whereas llama3.3-70b performs better in correctness and cultural knowledge evaluation. in future work, we aim to construct a reliable and high-quality benchmark dataset that addresses the shortcomings identified in this study.