How Good Are LLM s for Literary Translation, Really? Literary Translation Evaluation with Humans and LLM s (2025)

Abstract: Recent research has focused on literary machine translation (MT) as a new challenge in MT. However, the evaluation of literary MT remains an open problem. We contribute to this ongoing discussion by introducing LITEVAL-CORPUS, a paragraph-level parallel corpus containing verified human translations and outputs from 9 MT systems, which totals over 2k translations and 13k evaluated sentences across four language pairs, costing 4.5k€. This corpus enables us to (i) examine the consistency and adequacy of human evaluation schemes with various degrees of complexity, (ii) compare evaluations by students and professionals, assess the effectiveness of (iii) LLM-based metrics and (iv) LLMs themselves. Our findings indicate that the adequacy of human evaluation is controlled by two factors: the complexity of the evaluation scheme (more complex is less adequate) and the expertise of evaluators (higher expertise yields more adequate evaluations). For instance, MQM (Multidimensional Quality Metrics), a complex scheme and the de facto standard for non-literary human MT evaluation, is largely inadequate for literary translation evaluation: with student evaluators, nearly 60% of human translations are misjudged as indistinguishable or inferior to machine translations. In contrast, BWS (BEST-WORST SCALING), a much simpler scheme, identifies human translations at a rate of 80-100%. Automatic metrics fare dramatically worse, with rates of at most 20%. Our overall evaluation indicates that published human translations consistently outperform LLM translations, where even the most recent LLMs tend to produce considerably more literal and less diverse translations compared to humans.

## introduction with the advent of large language models (llms), literary translation, once the exclusive domain of human translators @xcite ; error annotation 0 2 6 4 sqm worst bws best percentage of cases where @xmath0.4% 100% 62.8% 42.5% 82.1% student professional 42.7% source translation figure 1: adequacy comparison per scheme complexity and evaluator expertise. sqm represents scalar quality metrics. professional and student evaluators perform free and mqm error annotation, respectively. see figure 4 and section f.5 for more details. @xcite , now stands at the threshold of a technological transformation @xcite . llms have the potential to increase the volume of translated literary works, particularly for low-resource languages @xcite . however, they pose a dual threat: potentially decreasing the salaries of professional translators or even displacing them entirely and diminishing the aesthetic value of translations if llms' output quality is persistently overestimated. as a consequence, the field has received much increased attention recently, both in the academic domain @xcite @xcite @xcite and in the translation community @xcite @xcite . evaluating literary translation poses unique challenges that extend beyond mere linguistic accuracy. it demands a delicate balance of cultural insight, authorial voice, creative interpretation, and aesthetics @xcite @xcite . these nuances surpass the scope of traditional sentence-level mt evaluation methods typically used in non-literary domains, presenting substantial obstacles for existing assessment techniques @xcite . we notice the following gaps in existing research regarding literary translation: (1) for automatic evaluation, previous metrics like d-bleu, bleurt and blonde @xcite @xcite fall short in discrimination between professional human translations and machine outputs in literary mt @xcite . more powerful recent popular metrics such as xcomet @xcite , gemba-mqm @xcite , and prometheus @xcite remain untested for literary texts. (2) the adequacy of human evaluation frameworks such as multidimensional quality metrics (mqm) @xcite , scalar quality metric (sqm) @xcite , and best-worst scaling (bws) @xcite remains largely unexplored: to our knowledge, there is no comprehensive comparison of these schemes for evaluating literary translations. this critical gap is concerning, as current studies apply these approaches arbitrarily, resulting in findings that are not directly comparable and potentially leading to unreliable conclusions. (3) existing datasets are small, cover few llm models, and lack verification for human translations, including translator qualifications and experience @xcite . these issues can lead to substantial misjudgments when comparing llm outputs to human translations, potentially overestimating the abilities of llms to the detriment of human professional translators. in this paper, we address these gaps with the following key contributions: ## related work human evaluation for literary texts literary mt evaluation primarily relies on human judgments of various types. mqm, the widely used guided error annotation scheme in the renowned mt venue wmt @xcite , has been shown effective for non-literary texts @xcite but its suitability for literary translation is uncertain. however, @xcite indicate that the complex mqm error categorization system requires costly trained experts and may lead to low annotation agreement. bws is also employed for literary texts, including poetry @xcite , but has limitations in comparing multiple systems simultaneously @xcite . recently, @xcite claim that llm-based multi-agent literary translations achieve parity with human translations, based on preference comparison by crowdsourced and llm evaluator-though the expertise of human translators remains unclear. sqm, a lik-ert scale rating, is used in non-literary domains. to our knowledge, a comprehensive comparison of these schemes for literary mt is lacking, with different studies employing various methods ad libitum. our work provides the first systematic comparison of these evaluation methods for literary texts. human evaluation of language models has been the subject of recent debate @xcite . human evaluation is typically assessed via consistency, i.e., whether human annotations are similar. besides consistency, we focus on adequacy: whether the implications of evaluation are plausible (e.g., whether human reference texts are ranked the highest) @xcite . automatic metrics for mt have been extensively studied at sentence and document levels in non-literary domains. these include nlibased menli @xcite , trained metrics like the comet series @xcite , bert-based scores such as bleurt @xcite and bertscore @xcite , prompting-based metrics like gemba-mqm @xcite and mqm-ape @xcite , in addition to documentlevel blonde @xcite and dis-coscore @xcite . while some researchers have applied these metrics to literary texts @xcite , their effectiveness in literary domain remains understudied. previous work has examined bleu, bleurt, and blonde for identifying top-performing systems @xcite , with limited correlation analysis (e.g., brief discussions in appendices) of comet, comet-qe, bleurt, and bertscore @xcite . our study addresses this gap by evaluating the effectiveness and adequacy of recent llm-based metrics for literary translation. corpora for literary mt evaluation existing parallel corpora are insufficient for literary translation evaluation. ## liteval-corpus we detail our liteval-corpus in this section, covering data construction and statistics. ## human evaluation we use liteval-corpus to examine the consistency and adequacy of different schemes. ## automatic metrics we use liteval-corpus to assess recent automatic metrics. while human mqm and sqm may not fully capture the nuances of literary translation quality, as discussed in the previous section, it is still valuable to examine how consistently current metrics align with human mqm/sqm evaluations. this is particularly relevant given that the recent top-performing metrics are built on similar principles. we focus on reference-free metrics due to the scarcity of reference translations for the majority of literary works @xcite and measure the performance by segment-level correlation between automatic metrics and human evaluation. we examine four sota automatic metrics that rely on llms: prometheus 2 (7 billion parameters) @xcite , an open-source model trained to evaluate other language models, which is queried using a tailored prompt (shown in the best metric (gemba-mqm) correlates moderately with human mqm but it cannot distinguish human translation from llm outputs. figure 2 (a) illustrates the segmentlevel correlation between human mqm and automatic metrics. gemba-mqm, particularly its original version, consistently outperforms other metrics. gemba-mqm (literary) performs similarly to gemba-mqm (original), though slightly worse across the board. xcomet-xl and xcomet-xxl models rank second overall, demonstrating poor to moderate correlation with human evaluation. their performance notably declines for more distant language pairs, such as de-zh, possibly due to differences or limitations in their training data for this language direction. moreover, we notice that xcomet-xl and xcomet-xxl are negatively correlated with the length of the target translation at -0.420 and -0.393, respectively. in contrast, the correlations between the length of the target and other metrics are all below 0.05 in absolute value. additionally, prometheus performs the worst, assigning a score of 4 (equivalent to "good" translation) to 54.9% of segments. its correlation to human sqm ranks lowest among all automatic metrics, as shown in even the best automatic metric is not adequate for literary mt evaluation. gemba-mqm, despite moderate correlation with human sqm and mqm, faces adequacy issues in literary translation evaluation. meanwhile, fluency and style demonstrate poor to fair correlations. notably, terminology consistently shows near-zero correlations, which could be problematic for distant language pairs with more culture-specific terms, e.g., en-zh with terminology as the second most frequent error category. these findings suggest that future metric development should focus on improving the assessment of fluency, style, and terminology. human mqm vs. gemba-mqm: llms demonstrate a tendency towards more literal and less diverse translations. we further compare human mqm with gemba-mqm in system rankings considering characteristics of the translation systems. to assess the literalness of translation, we analyze the syntactic similarity between source-translation pairs. we utilize the recent document-level tree kernel-based syntactic similarity metric fastkassim @xcite . while high syntactic similarity preserves the source's form, it frequently sacrifices naturalness in the target language and hinders creativity in translations @xcite . diversity is another concern in literary domain @xcite . to measure lexical diversity of one system compared to all other systems, we calculate pairwise lexical overlap between pairs of translations from different systems and average them per system. 18 a lower value suggests a more diverse vocabulary selection. figure 3 shows results based on all publications, illustrating human mqm scores for 10 translation systems averaged over four language pairs (z-axis) against syntactic similarity (x-axis) and lexical overlap (y-axis) (see figure 10 in the appendix for results on contemporary works). the rank of systems is marked as [rank per human mqm, rank per gemba-mqm ]. we show detailed examples in deepl has the highest syntactic similarity of 0.27. (3) the third cluster demonstrates a performance gap to other clusters, with qwen 2 and towerinstruct performing slightly better. qwen 2 paraphrases more for de-zh and en-zh pairs and sometimes includes chinese characters in de-en and en-de outputs, which results in low syntactic similarity and lexical overlap but still higher than human. summary: gemba-mqm outperforms other current automatic metrics but struggles with non-accuracy aspects. all metrics lack adequacy in distinguishing human and llm translations. llms tend to produce more literal and less diverse translations showing higher syntactic similarity to the source and greater lexical overlap with other systems than human translations. further, llm as evaluator prefers more literal translations and is biased towards their outputs. ## conclusion this paper introduces liteval-corpus and a comprehensive study on literary translation evaluation with human and automatic metrics. our study reveals the limitations of error annotation, particularly mqm when comparing topperforming systems. this may be attributed to three factors: (1) the complexity of mqm com-pared to other schemes; (2) the current mqm framework failing to account for "errors" in literary translation intended for certain equivalence in the target language; (3) current leading systems producing fewer errors than previous mt systems, rendering mqm insufficient. while using bws or experienced professionals yields more plausible assessments, bws has limitations for further improvements like post-editing when compared to mqm. these results have substantial implications, particularly for determining human parity-a crucial issue in today's llm landscape. they are especially relevant for professional translators, as naive translation companies might eliminate human translators due to misjudgments based on mqm or sqm evaluations by inexperienced practitioners. while we highlight mqm's limitations in literary mt evaluation, our current study does not propose specific improvements to the existing mqm guidelines. future work can explore these aspects in depth. furthermore, our results indicate that automatic metrics development should focus on aspects such as style and terminology. llms still need to address the issue of overly literal outputs, and a substantial gap remains between llm and human quality in literary translation, despite the clear advancements of recent models.