Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature (2022)

Abstract: Literary translation is a culturally significant task, but it is bottlenecked by the small number of qualified literary translators relative to the many untranslated works published around the world. Machine translation (MT) holds potential to complement the work of human translators by improving both training procedures and their overall efficiency. Literary translation is less constrained than more traditional MT settings since translators must balance meaning equivalence, readability, and critical interpretability in the target language. This property, along with the complex discourse-level context present in literary texts, also makes literary MT more challenging to computationally model and evaluate. To explore this task, we collect a dataset (Par3) of non-English language novels in the public domain, each aligned at the paragraph level to both human and automatic English translations. Using Par3, we discover that expert literary translators prefer reference human translations over machine-translated paragraphs at a rate of 84%, while state-of-the-art automatic MT metrics do not correlate with those preferences. The experts note that MT outputs contain not only mistranslations, but also discourse-disrupting errors and stylistic inconsistencies. To address these problems, we train a post-editing model whose output is preferred over normal MT output at a rate of 69% by experts. We publicly release Par3 to spur future research into literary MT.

## introduction while the quality of machine translation (mt) systems has greatly improved with recent advances in modeling and dataset collection, the application of these new technologies to the task of automatically 1 https://github.com/katherinethai/par3/ ⋆authors contributed equally. in this work, we seek to understand how both state-of-the-art mt systems and mt evaluation metrics fail in the literary domain, and we also leverage large pretrained language models to improve literary mt. to facilitate our experiments, we introduce par3, a large-scale dataset to study paragraph-level literary translation into english. par3 consists of 121k paragraphs taken from 118 novels originally written in a non-english language, where each paragraph is aligned to multiple humanwritten english translations of that paragraph as well as a machine-translated paragraph produced by google translate (see we show that mt evaluation metrics such as bleu and bleurt are not effective for literary mt. in fact, we discover that two of our tested metrics (bleu and the document-level blonde) show a preference for google translate outputs over reference translations in par3. in reality, mt outputs are much worse than reference translations: our human evaluation reveals that professional translators prefer reference translations at a rate of 85%. while the translators in our study identified overly literal translations and discourse-level errors (e.g., coreference, pronoun consistency) as the main faults of modern mt systems, a monolingual human evaluation comparing human reference translations and mt outputs reveals additional hurdles in readability and fluency. to tackle these issues, we fine-tune gpt-3 (brown et al., 2020) on an automatic post-editing task in which the model attempts to transform an mt output into a human reference translation. human translators prefer the post-edited translations at a rate of 69% and also observe a lower incidence of the above errors. overall, we identify critical roadblocks in evaluation towards meaningful progress in literary mt, and we also show through expert human evaluations that pretrained language models can improve the quality of existing mt systems on this domain. we release par3 to spur more meaningful future research in literary mt. ## the par3 dataset: parallel paragraph-level paraphrases to study literary mt, we collect a dataset of parallel paragraph-level paraphrases (par3) from public domain non-english-language (source) novels with their corresponding english translations generated by both humans and google translate. par3 is a step up in both scale and linguistic diversity compared to prior studies in literary mt, which generally focus on one novel (toral et al., 2018) or a small set of poems or short stories (jones and irvine, 2013). par3 contains at least two human translations for every source paragraph ( ## related work our work builds on previous work in literary machine translation. some early work focused on poetry translation (genzel et al., 2010; jones and irvine, 2013), which has recently been tackled with neural approaches (chakrabarty et al., 2021). other works have targeted novels, like those in par3, with focuses on manual post-editing (toral et al., 2018, 2020) and comparisons of neural mt to statistical mt systems (moorkens et al., 2018; toral and way, 2018, 2015). most of these works experiment with datasets far smaller than par3 (arenas and toral, 2022; fonteyne et al., 2020). more recent work has involved studying the linguistic characteristics of post-edited literary machine-translated text (castilho and resende, 2022; macken et al., 2022). work towards document-level nmt has built on sentence-level mt (tiedemann and scherrer, 2017; jean et al., 2017; bawden et al., 2018; miculicich et al., 2018; agrawal et al., 2018). the a critical lack of parallel document-level corpora has inspired the creative use of parallel sentencelevel data (zhang et al., 2018) and techniques for creating parallel document-level data (junczys-dowmunt, 2019). our work also builds on efforts to tackle discourse-level errors specific to document-level mt and is very similar to that of voita et al. (2019), but we specifically focus on the literary domain. ## conclusion we study document-level literary machine translation by collecting a dataset (par3) of 121k parallel paragraphs from 104 novels. our experiments show that existing automatic metrics of translation quality are not meaningful in the literary domain. a human evaluation experiment with professional literary translators reveals that commercial-grade mt systems are too literal in their translations and also suffer from discourse-level errors. we mitigate these problems to a certain extent by developing an automatic post-editing model using gpt-3. overall, our work uncovers new challenges to progress in literary mt, and we hope that the public release of par3 will encourage researchers to tackle them.