A comparison of translation performance between D eep L and Supertext (2025)

Abstract: As strong machine translation (MT) systems are increasingly based on large language models (LLMs), reliable quality benchmarking requires methods that capture their ability to leverage extended context. This study compares two commercial MT systems – DeepL and Supertext – by assessing their performance on unsegmented texts. We evaluate translation quality across four language directions with professional translators assessing segments with full document-level context. While segment-level assessments indicate no strong preference between the systems in most cases, document-level analysis reveals a preference for Supertext in three out of four language directions, suggesting superior consistency across longer texts. We advocate for more context-sensitive evaluation methodologies to ensure that MT quality assessments reflect real-world usability. We release all evaluation data and scripts for further analysis and reproduction at https://github.com/supertext/evaluation_deepl_supertext.

## introduction after the transition from statistical to neural modelling roughly a decade ago @xcite @xcite , the field of mt is undergoing another paradigm shift towards leveraging llms @xcite @xcite . llm-based translation offers the potential for significantly improved translation quality, especially with respect to consistent translation of documents. unlike neural machine translation (nmt) systems, which typically process documents as isolated sentences or paragraphs @xcite , many llms operate with context windows that can span thousands of words, allowing them to maintain consistency throughout a document -for instance, by ensuring that a word's translation in the final sentence matches its previous forms @xcite . in the most recent shared task at the conference on machine translation (wmt24) that focuses on evaluating the state of the art in general-domain translation quality, the majority of the 28 system submissions were already based on llms @xcite . although without statistical significance and for the language direction english to german only, one system even outranked the human reference translations as evaluated by professional human annotators. despite this impressive achievement, findings of human-machine parity should be approached with caution. similar claims already emerged with pre-llm technology @xcite ), yet have subsequently been refuted due to limitations in the evaluation design focusing on single segments in isolation @xcite @xcite . the wmt24 shared task also highlights that evaluations based on automatic metrics (rather than human evaluation) can lead to wrong conclusions when comparing strong mt systems @xcite . however, these insights are often overlooked in evaluations of commercial mt systems. for example, intento's the state of machine translation 2024 report, foot_0 which assesses 52 providers across 11 language pairs, serves as a valuable resource for potential users in real-world settings, but its benchmarking methodology relies on automatic scoring of sentence-level data, and the authors acknowledge that 'you may need a human linguist' to ensure greater reliability. in this paper, we evaluate two commercial translation systems (section 2) under conditions that allow for leveraging the full-text capabilities of llms. the segmentation of the source text is handled by the translation systems alone without any prior splitting (section 3), and the resulting translations are rated by professional translators considering the full document context (section 4). we find that while both systems translate a similar number of segments better than the other, the difference is more pronounced on the document level (section 5), which we attribute to differences in how much context the systems consider during translation (section 6). our findings suggest that the adoption of llms creates opportunities for smaller players to challenge dominant industry leaders (section 7). ## systems we compare the free online offering of two commercial mt providers: deepl deepl foot_1 is a widely used mt provider boasting 'unrivalled translations that set the standard'. 4 in the latest intento report, it scores best among nine 'real-time engines' and, together with gpt-4, is found to 'consistently outperform other models'. 2 due to its closed source, the technology behind deepl's translation system is not publicly known. supertext supertext foot_3 builds on an open, generalpurpose llm that has been specialised for the task of translation with proprietary methods and data. while the system can be adapted to specific domains, we use the freely available generic version. for the purpose of the evaluation described in this paper, we use both systems with default settings. for example, we do not specify politeness (formal/informal) although supported by both systems in some language combinations. while both deepl and supertext provide target language variants for english (british and american), supertext also provides target language variants for german (austria, germany, switzerland), french (france, switzerland), and italian (italy, switzerland). as our use case is machine translation for people in switzerland, we use the swiss target language variant whenever available (section 3.2). 6 language direction texts segments words 3 data ## evaluation setup we conduct a blind a/b test in which professional translators rate deepl and supertext outputs with full document-level context. ## evaluation results segment-level and text-level preference ratings are shown in figures 1 and 2 , respectively. ## discussion our evaluation highlights that conclusions drawn from mt quality assessments may vary significantly depending on the unit of measurement. while raters in our study preferred a similar number of translated segments by deepl and supertext overall, the difference becomes more pronounced at the document level. this discrepancy suggests that segment-level assessments alone may not fully capture translation quality as perceived in real-world usage, where coherence and consistency across entire documents play a critical role. notably, while segment-level ratings indicate no strong preference between the two systems in most language directions, document-level aggregation reveals a more distinct pattern. raters favour supertext's translations at the document level in three out of four language directions, with the most pronounced difference observed in @xmath0. this suggests that supertext may provide better consistency or fluency across longer texts in these language directions. while we have yet to conduct a systematic qualitative comparison of system outputs, we find texts where the same word is translated differently by deepl and consistently by supertext across paragraphs. an example is shown in in contrast, for @xmath1, raters show a clear preference for deepl at both the segment and document levels, indicating a potential strength of deepl in handling this specific language combination. our preliminary analysis is inconclusive at this point, but the supertext outputs seem to contain a higher number of within-sentence errors such as wrong choices for individual words or omissions. another hypothesis is that supertext, which supports three different german target language variants, may introduce inconsistencies by mixing region-specific elements in translation outputs. ## conclusion our study highlights the growing significance of document-level evaluations in mt quality benchmarking, especially as llm-based systems leverage broader context windows to enhance translation consistency. while segment-level assessments suggest no clear preference between deepl and supertext in most of the language directions we examined, document-level aggregation reveals notable differences. supertext is preferred in three out of four language pairs, where its translations exhibit greater consistency. in contrast, @xmath2, possibly due to fewer within-sentence errors or differences in regional language handling. as llm-based mt systems continue to evolve, future studies should further investigate the impact of context length on commercial mt benchmarking campaigns. insights into how different systems leverage context and resolve ambiguities will be essential for advancing evaluation methodologies and ensuring that translation systems meet realworld user expectations.