Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist (2023)

Abstract: Large language models (LLMs) are competitive with the state of the art on a wide range of sentence-level translation datasets. However, their ability to translate paragraphs and documents remains unexplored because evaluation in these settings is costly and difficult. We show through a rigorous human evaluation that asking the GPT-3.5 (text-davinci-003) LLM to translate an entire literary paragraph (e.g., from a novel) at once results in higher-quality translations than standard sentence-by-sentence translation across 18 linguistically-diverse language pairs (e.g., translating into and out of Japanese, Polish, and English). Our evaluation, which took approximately 350 hours of effort for annotation and analysis, is conducted by hiring translators fluent in both the source and target language and asking them to provide both span-level error annotations as well as preference judgments of which system’s translations are better. We observe that discourse-level LLM translators commit fewer mistranslations, grammar errors, and stylistic inconsistencies than sentence-level approaches. With that said, critical errors still abound, including occasional content omissions, and a human translator’s intervention remains necessary to ensure that the author’s voice remains intact. We publicly release our dataset and error annotations to spur future research on the evaluation of document-level literary translation.

## introduction large language models (llms) such as chatgpt (openai, 2022) demonstrate remarkable performance as stand-alone translation systems, rivaling and sometimes surpassing commercial models on sentence-level benchmarks (vilar et al., 2022; hendy et al., 2023; jiao et al., 2023). furthermore, llms are increasingly being deployed for document-level translation (book maker, 2023; translations produced by gpt-3.5 across 18 different language pairs. in all cases, para produces fewer errors than sent, which demonstrates that gpt-3.5 takes advantage of discourse context during translation. pawlak, 2023), a scenario for which there are currently no reliable automatic evaluation methods. in this paper, we hire human translators to conduct a rigorous fine-grained evaluation of gpt-3.5's ability to translate paragraph-level texts from literary works across 18 different language pairs. our results (figure 1 ) demonstrate that gpt-3.5 foot_0 effectively leverages discourse-level context to produce higher-quality translations than when translating sentences in isolation. why literary texts? translating works of literature poses unique challenges due to the intricate nature of creative work and the importance of capturing the author's voice and contextual nuances. translators thus apply a wide range of translation techniques (chesterman, 1997; molina and hurtado albir, 2004), from simple shifts in grammatical categories to more complex stylistic or content-based rearrangements that often cross sentence boundaries. translators may also merge or split sentences and paragraphs, which renders the traditional sentence-level pipeline insufficient for capturing the full scope of the original text (toral and way, 2015; taivalkoski-shilov, 2019b; post and junczys-dowmunt, 2023; jiang et al., 2023). @xcite taken together, these properties make literary texts a good testbed for document-level machine translation (thai et al., 2022); in our work, we focus on the paragraph 4 as a minimal discourse-level unit. why human evaluation? the absence of rigorous document-level evaluations of llm translators is striking but also somewhat understandable given the unreliability of automatic metrics (thai et al., 2022) and the difficulty of properly conducting human evaluations (castilho, 2021). furthermore, evaluations of llm translators are especially difficult due to data contamination (aiyappa et al., 2023; chang et al., 2023), as it is unclear whether the models are pretrained on existing benchmarks (e.g., from wmt). we fill this gap by first collecting paragraphs from recently-published literary translations. then, we provide human translators with two candidate machine translations of a given source paragraph and ask them to (1) mark error spans and categorize them based on a predefined @xcite at least 55% of the reference target paragraphs used in our study split or merge sentences from the source text (measured with an automatic sentence tokenizer). @xcite we broadly define a paragraph as a distinct passage within the novel, focusing on a single theme. schema inspired by mqm (lommel et al., 2014b; freitag et al., 2021), (2) make preference judgments of which of the two translations is of higher quality, and (3) provide free-form justifications of their preference judgments. in total, we collect such annotations on 720 pairs of translated paragraphs across 18 different language pairs (using three diverse target languages of english, japanese, and polish), which we then leverage for a finegrained analysis of the behavior of different llm translation methods. llms produce better translations when provided with paragraph-level context: our evaluations reveal that using gpt-3.5 to translate complete paragraphs via few-shot prompting (para) yields translations of significantly higher quality than both the sentence-by-sentence gpt-3.5 methods (sent, para_sent) as well as google translate. our detailed analysis of annotated translation errors and free-form comments shows that para exhibit increased coherence, better preservation of literary style, and improved handling of contextdependent expressions (see figure 2 ). that said, para makes many critical mistranslations and other errors across different language pairs, which shows that llm-based translators still have significant room to improve, particularly when translating contextually-rich literary texts. ## background before describing our dataset and evaluation, we first contextualize our work within the recent body of research on translation via large language models. we also survey the broader body of document-level foot_1 mt research in §a. translation with large language models: llmbased translation is attractive because a single model, without training or fine-tuning on large parallel corpora, can produce high-quality translations across many language pairs. foot_2 recent work explores llms' capabilities in this space (wang et al., 2023) spanning paragraph-level post-editing with llms (thai et al., 2022), translating sentence-level inputs (vilar et al., 2022; jiao et al., 2023), analyzing hallucinations in llm-generated translations (guerreiro et al., 2023), and employing llms to evaluate machine translation (kocmi and federmann, 2023). simple sentence-level english prompt templates have been found effective for paragraph translations (zhang et al., 2023), and automaticallygenerated dictionaries can assist llm-based translation (ghazvininejad et al., 2023; lu et al., 2023) along with selecting high-quality demonstrations (vilar et al., 2022). to the best of our knowledge, the only prior work other than ours that evaluates llms for paragraph-level translation is hendy et al. (2023), who conduct automatic evaluation of context-aware sentence-by-sentence translation; in contrast, we perform a fine-grained human evaluation of paragraph-level translation. ## data & methods our work differs from existing research on translating with large language models in two key ways: we focus on translating literary text at the paragraph level. in this section, we describe and motivate the paragraph-level translation dataset used in our study, which covers 18 unique language pairs (three target languages) and is sourced from recently-published novels. then, we outline the different ways in which we leverage gpt-3.5 to translate these paragraphs at both the sentence and paragraph levels. ## evaluating document-level literary translation how do we compare the translation quality of the systems described above? automatic metrics such as bleurt and comet are untested on documentlevel inputs as well as literary texts, and as such we do not consider them reliable, although we do @xcite all paragraphs were translated in january 2023 using the googletranslate api. the system was provided entire paragraphs, which it likely partitioned and translated sentenceby-sentence. source text: une autre photo, signée du même photographe... grammar mistranslation untranslated inconsistency register format any omissions or additions? error annotation translation 1: inny zdjęcie, podpisane przez tego samego fotografa... which translation is better? why is it better? i prefer t1 to t2 as it doesn't... translation 2: inne zdjęcie, podpisane tym samym fotografem... read the source text. mark and categorize span-level errors binary choice (yes/no) binary choice (translation 1 vs 2) binary choice (confident vs unsure) justify preference was it significantly better? report them in §g. @xcite human evaluation is equally problematic, as direct assessments of translation quality (e.g., "rate the quality of this translation from 0-100") suffer from calibration issues that are exacerbated with longer texts (karpinska et al., 2021). thus, we opt for a human evaluation inspired by multidimensional quality metrics (lommel et al., 2014b, mqm), in which annotators mark and classify error spans within the translation. specifically, for each of the 18 language pairs studied in this work, we hire translators to identify all span-level errors in two competing translations. for each evaluated pair, the annotators were also asked to choose the better translation and provide a free-form rationale. for each source paragraph, the translators make three binary judgments of which translation is higher quality: sent vs para, para_sent vs para, and gtr vs para. recruiting annotators: as our task is complex and requires fluency in both the source and target language, we hire translators to provide the annotations. we recruit 13 translators via the upwork freelancing platform, @xcite each of whom is a native speaker of english, polish, or japanese. @xcite one translator, hired directly, was a bilingual speaker of english and polish with advanced knowledge of german; as such, she performed the pl-en, de-en, and de-pl evaluations. evaluation of ja-pl, pl-ja, and pl-en texts was done by the first author in a collaboration with native speakers of polish/japanese to avoid any potential bias. each translator was paid @xmath05 bonus to cover the time spent familiarizing themselves with the instructions. we asked them to compare three pairs of system translations (para vs. sent, para vs. para_sent, para vs. gtr) for 10 paragraphs per language pair 18 ; as such, 180 total source paragraphs were used in our evaluations. altogether, we paid approximately @xmath1955. annotation task: first, we tasked the hired translators @xcite with annotating a subset of mqm translation errors identified through a pilot analysis and annotation of the system's outputs. specifically, we ask them to highlight spans within the candidate translations that contain errors belonging to any of the following error categories: in addition to canonical mistranslations, we also include overly literal translation errors that occur when systems nonsensically translate word-by-word into the target language. after the span-level annotation is complete, we then ask the translators to further identify if any of the candidate translations contains significant content additions or omissions in relation to the source text. @xcite finally, they are asked to choose the better translation and provide a justification for their choice in two to five sentences. we instruct them to additionally mark whether their chosen translation is significantly superior, or if the decision was difficult because both translations are of roughly comparable quality (see figure 3 and §d for details). ## results in this section, we compare our different literary translation methodologies using both automatic metrics and aggregate statistics from the human evaluations. overall, we observe that the para configuration outperforms competing methods across all evaluations and language pairs. these results demonstrate that gpt-3.5 effectively leverages paragraph-level context to produce better translations than sentence-level methods, and also that the less efficient sentence-by-sentence translation with context is (para_sent) is unnecessary to achieve high translation quality. figure 4: the distribution of translator preference judgments between sentence-level translation (sent) and paragraph-level translation (para). para is preferred (i.e., more votes) in every language pair except de-ja, fr-en and de-en. ## analyzing translation errors the aggregate statistics from the previous section confirm that para-level translation via gpt-3.5 is the strongest literary translator of the methods that we study. translations produced by para are favored by both automatic metrics and human translators, and it makes fewer errors than competing methods. in this section, we dive deeper into specific types of errors that are made within each high-level category (e.g., grammar, mistranslation), and we present examples of errors associated with overall, models with access to paragraph-level context commit fewer translation errors. ## conclusion in this paper, we demonstrate that llms leverage paragraph-level context to produce translations that are more coherent and enjoyable than sentence-bysentence translation while containing fewer mistranslations and grammatical issues. our evalu-ations reveal that professional translators prefer paragraph-level translations over both sentencelevel translations produced by the same language model, and also to those generated by an offthe-shelf commercial system (gtr). we release our dataset and error annotations to help facilitate the development of new evaluation methodologies and automatic metrics for document-level machine translation. finally, a full-length novel extends far beyond the confines of paragraph-level translation. in future work, we will focus on integrating individual paragraphs into cohesive chapters, which can then be expanded to encompass the entire novel.