Comparing Human and Machine Translations of Generative Language Model Evaluation Datasets (2025)

Abstract: The evaluation of Large Language Models (LLMs) is one of the crucial current challenges in the field of Natural Language Processing (NLP) and becomes even more challenging in the multilingual setting. Since the majority of the community’s benchmarks exist only in English, test sets are now being machine translated at scale into dozens of languages. This work explores the feasibility of that approach, comparing a Finnish machine translation (MT) of ARC-Challenge with a new human translated version. Our findings suggest that since absolute scores are fairly close and model size rankings are preserved, machine translation is adequate in this case. Surprisingly, however, the datasets reverse the order of base models compared to their chat-finetuned counterparts.

## introduction and background generative large language models (llms) have made significant progress in the past few years and their usefulness is being explored in many applications. this exploration is occurring world-wide, and as such there are many multilingual models available which have been trained with data in several languages simultaneously. however, a central challenge in building multilingual models is that access to quality data in languages except for the largest ones is limited, and this challenge crucially extends to evaluation datasets. our ability to train acceptably performing llms in new languages has far outpaced our abilities to create high-quality evaluations for those languages, in part because training can rely on transfer effects, where competence acquired in one language generalises to another language to some extent (e.g., @xcite . constructing new test sets in the language under consideration allows for controlling the quality as well as cultural validity of test items, but translating existing test sets (usually in english) to a target language involves less effort, less cost, and provides a basis to compare results across languages. translating entire test suites involves considerable human effort, so using automatic translation tools is an obviously attractive option. given the immediate need to evaluate multilingual models, these machine translations of evaluation datasets have started proliferating -for example, @xcite automatically translate the popular benchmarks hellaswag @xcite , mmlu @xcite and arc @xcite into 26 languages. however, these strategies carry a certain risk of systematic bias and introduced error into the test, and very little work has been done to verify that the resulting evaluations can be trusted. besides actual translation errors, sometimes the objectives of the test are rendered moot by linguistic differences: for instance, tests that exploit structural ambiguities translate poorly from an isolating language to an agglutinative one. this work investigates how automatically translated tests compare to manually translated tests. we study the case of arc-challenge, the challenging subset of arc, a four-way multiplechoice task that has become a popular english llm benchmark. we compare the performance of several finnish llms on an automatically translated version (arc-c-fi-mt) with a new manually translated version (arc-c-fi-ht), which we release publicly. we find that in this task setup, llms perform comparably on the machine-and human-translated versions, so that machine translation may actually suffice in this case. one surprising caveat is that when considering model orderings, base models outperform their chat-tuned counterparts on human data, while the chat-tuned models are stronger on machine translation data. ## llm families we evaluate three model families, poro, viking and ahma, chosen because they are effectively the only llm families trained especially for finnish. note that although @xcite is absent, it can be viewed as a predecessor of poro, since poro uses an extended version of the same training data, and the models use an identical architecture. ## methodology we use eleutherai's lm-evaluation-harness @xcite to run the evaluations. for each of the datasets, we use the default parameters of english arc-challenge. in particular, we evaluate using the multiple choice setting, with doc to text: "question: {{question}}\nanswer:" and num @xmath0. this means that answers are obtained using logprobs instead of running inference. for each possible choice, the logprob of the choice text given doc to text is computed, and the model's answer is the maximum logprob choice. we compute both acc (accuracy) and acc norm. the latter metric computes accuracy when logprobs are normalized by answer length, so that longer answers are not deemed less likely only due to their length. to guarantee that results are tokenizer-agnostic, normalization is performed using number of characters rather than, for example, number of tokens. ## experimental results and analysis one straightforward way of analyzing the translated datasets is through comparing absolute scores per model. however, the datasets can also be compared in terms of whether they preserve the ranking between two sets of evaluated models. the model rankings ( ht mt en poro 34b-c acc norm .397 .391 .485 acc .376 .374 .452 poro 34b acc norm .414 .369 .462 acc .361 .341 .424 viking 13b acc norm .387 .329 .402 acc .346 .312 .359 viking 7b acc norm .363 .326 .366 acc .308 .301 .340 ahma-7b acc norm .358 .327 .275 acc .302 .276 .248 ahma-3b-i acc norm .323 .310 .250 acc .290 .259 .220 ahma-3b acc norm .324 .307 .255 acc .278 .270 .195 viking 7b models show similar performance on finnish -it is unlikely their change in ranking between datasets is significant. secondly, however, for both poro 34b and ahma-3b, we see a change in ordering of the base model and chat model variants (note that for other base models the finetuned versions are not yet available). the base completion models rank higher when evaluated using human translation, while the chat models rank higher for the machine translated version. still, the size of these differences varies per model. one result worth noting is that the chat models perform similarly between the two datasets in absolute terms (.397 and .391 for poro 34b chat; .323 and .310 for ahma-3b-instruct). however, and particularly for poro, there is a larger difference for the base models (.414 and .369 for poro 34b; .324 and .307 for ahma 3b). thus the chat models seem more robust across the datasets, but at a cost to performance (.369 for poro 34b @xmath1.391 for poro 34b chat on the machine translation condition). we also find that the poro and viking models, trained on both english and finnish, perform better on the english dataset than on the finnish datasets. this is unsurprising given that in the case of poro, there were more than four times as many english tokens in the training distribution. the ahma models, lacking english training data, reach the expected performance of around 25% on english given 4-way multiple-choice. ## discussion we propose a technique to find particularly interesting examples by 1) filtering for cases where the model is correct on one dataset and incorrect on the other and 2) sorting by the difference in logprobs on the prediction for the correct answer. in this way, we find samples where the difference in translation has the greatest effect on model prediction and performance. this reveals some clear mistakes in machine translation. for instance, in mercury sc 414274 the correct choice is the moon is covered with many craters. here the human translation is kuun pinnalla on paljon kraattereita whereas machine translation outputs kuu on monien kraatterien peitossa, a more literal and less fluent translation. as a result it is only on the human translation data that poro-chat-34b manages to select the correct answer (with a logprob of -10.2 instead of -36.0). question mercury 7165218 about geology provides a more egregious error, where the choice rift is left untranslated as rift. there are also cases, however, where flaws in machine translation actually increase model scores. in question mercury sc 406710 about chameleons, the choice hunt for food is translated correctly by humans as saalistaa ruokansa, using the verb reserved for predators, but is machine translated as metsästää ruokaa, using the verb for human hunting foot_2 . however, perhaps since metsästää is more common, poro-34b-chat correctly chooses this as the answer, whereas it fails to do so for saalistaa. thus the human translation reveals the incomplete semantics of the model in this case, while the machine translation does not. a similar case occurs in mcas 2014 8 6, where poro-34b-chat makes the correct choice only for the machine translated answer con-taining the phrase tektonisten laattojen (tectonic plates). the human translation uses the phrase litosfäärilaattojen (lithospheric plates), which is heavily discounted by the model at a logprob of -73.0. both translations are correct, but litosfäärilaattojen is a slightly more scientific and technical term in finnish. tektonisten (of tectonics) is perhaps more common in layman's language, which would explain both its generation by the mt system and its higher logprob in the llm's answer. in such cases, machine translated evaluations assign an inflated accuracy to the models, which should be able to respond positively to both the common and rarer scientific terms. for an agglutinative case-based language such as finnish, similar cases would be possible when a human translator chooses a more accurate but less common grammatical case. there are many future research avenues here. one option is to further investigate the chat and completion model reordering. this is possibly explained by an alignment of the fine-tuning training data with machine translation data -in both cases, models are trained using curated sentence pairs (whereas base model pre-training data consists of large chunks of text from massive corpora that tend to be less curated). perhaps, then, finetuning a base model pulls it in the direction of the machine translation model distribution. future work that compares more pairs of base and chat models, along with extended logprob analyses of both models types, may elucidate the picture. future work will also investigate the complex set of benefits and drawbacks of human translation. human subjectivity and inconsistencies in judgment may introduce bias, and from a practical standpoint manual reviews can be time-consuming and expensive. one concrete direction is to compare the gold standard to arc-c-fi-htv1 and versions with alternative choices of normalization. it would also be worthwhile to explore alternate mt solutions, especially ones in which the models have access to the question as context when translating the answers. ## conclusion following the recent trend to machine translate english evaluation datasets at scale, this work compares a new human translation of arc-challenge into finnish with a machine translated version. our results indicate that for finnish arc-challenge, the machine translated dataset rivals the usefulness of the ht dataset for comparative evaluation of llms. this is observed through the small absolute differences between scores (with models performing slightly more favorably on human translations as expected), as well as through the preservation of ordering of model sizes. one interesting caveat is that while chat-finetuned models outperform base models on machine-translated evaluation data, base models actually outperform their chat-finetuned counterparts on the human translated data, warranting further investigation. thus although there are drawbacks to using machine translation, especially for literature or other longer-form data, this work reveals that for comparative evaluation of finnish language models on short multiple-choice questions, mt is sufficient. future work can continue to reveal distributions of evaluation data, language translation pairs and model classes where this holds. it is clear that the intersection of translation and llm evaluation provides unique challenges and opportunities that now deserve more attention than ever.