Machine Translation of literary texts: genres, times and systems (2023)

Abstract: Machine Translation (MT) has taken off dramatically in recent years due to the advent of Deep Learning methods and Neural Machine Translation (NMT) has enhanced the quality of automatic translation significantly. While most work has covered the automatic translation of technical, legal and medical texts, the application of MT to literary texts and the human role in this process have been underexplored. In an effort to bridge the gap of this under-researched area, this paper presents the results of a study which seeks to evaluate the performance of three MT systems applied to two different literary genres, two novels (1984 by George Orwell and Pride and Prejudice by Jane Austen) and two poems (I Felt a Funeral in my Brain by Emily Dickinson and Siren Song by Margaret Atwood) representing different literary periods and timelines. The evaluation was conducted by way of the automatic evaluation metric BLEU to objectively assess the performance that the MT system shows on each genre. The limitations of this study are also outlined.

## rationale recent advances in artificial intelligence and mt have brought a new perspective to the ongoing discussion on the automatic translation of literary texts among academics. more specifically, the significantly improved performance of neural machine translation has triggered a debate among translation professionals about the future role of the translators. it has been demonstrated that mt delivers better results when applied to scientific and technical texts which lack ambiguity and provide a precise message @xcite . on the other hand, literary texts are rich in rhetorical devices, ambiguity, and precise a certain level of creativity, becoming a great challenge for the mt to face when translating this type of texts @xcite , and as a result producing more literal translations which do not convey the essential meaning of the texts @xcite . nevertheless, the research on how feasible mt for literary texts is or on the development of new techniques to improve the quality of literary mt, has been insufficient. the related work has centred on determining the main factors of the use of mt as well as identifying the cognitive effects on the human translator when applying to its workflow. these studies have proven that mt is both useful and powerful tool used in the translation process and can enhance the productivity of the human translator @xcite . this study has been motivated by the recent advances of nmt and by the fact that the topic of the application of mt to literary texts has been underexplored. in particular, this study seeks for the first time to: the rest of the paper is structured as follows. section 2 discusses related work. section 3 details the methodology used by outlining the data used and the experiments conducted. section 4 reports the evaluation results and provides a brief discussion of the obtained results. section 5 lists the limitation of this study and finally section 6 presents the conclusions. ## related work most related work has focused on the feasibility of mt applied to scientific texts. with regard to literary mt, the studies have been focused on the narrative genre and the cognitive efforts of the human translator when mt is applied to the translation workflow. it has been proven that the use of mt in scientific and technical texts have enhanced the human translators' productivity @xcite . since literary texts exhibit more ambiguity and use literary devices to infer its meaning, the widespread view is that mt cannot cope with this type of text @xcite @xcite . nevertheless, there are researchers who consider possible the use of mt for translating literary texts successfully @xcite @xcite . @xcite studied the machine translation of a poem, considering its metric, length, and rhythm. the results showed that the format could be preserved, but it could not obtain the same quality nor preserve the meaning. @xcite focused on the referential cohesion in literary and non-literary texts and their outputs when processed by mt. they concluded that although literary texts had more cohesive references than non-literary texts and although mt was able to cope with them, the referential cohesion is a key factor for good mt performance. richardson (2012) employed microsoft translator hub in the translation process of a church to cater for the demand of translation into several languages, creating corpora and glossaries and resulting in a higher productivity. @xcite reported the results of a study on the applicability of mt to literary text taking into account how related are the languages involved (french, english and italian; spanish and catalan). they proposed to fine-tune the mt systems to the different types of literary texts regarding their characteristics such as cohesion, literary devices, dialogue, etc. they experimented @xcite with an nmt system customised to translate distant languages such as english and catalan and compared the results with the previous translations by an smt system. they drew their study on previous research @xcite ) where factors such as limitation and freedom of translation were taken into consideration. @xcite studied the perception of literary mt (based on both smt and nmt) by human translators. six professional translators were asked to translate from english to catalan in three different modes: translating from scratch, post-editing nmt output and post-editing smt output. it was shown that human translators preferred translating from scratch literary texts, but considered useful the suggestions offered by the mt systems. in line with this work, @xcite also proposed to incorporate automatic systems to the translation workflow to help the human translator improve their productivity. @xcite examined the challenges that nmt faced when applied to literary texts with english and russian as language pair and reported better performance after fine-tuning the mt systems to this particular language pair. in another study, @xcite applied nmt to slovenian literary texts with the results showing increase in the productivity. omar and gomaa (2020) identified the challenges mt systems face when translating literary texts; they concluded the most typical mistakes are pragmatic, structural and lexical. @xcite analysed the impact of post-editing and mt on creativity and literature, as well as the perceptions of professional translators on this issue. @xcite studied how mt used in the translation process affects the translator's voice. finally, @xcite evaluated the recent improvements of nmt systems when applied to literary texts aiming to produce coherent translation at a textual level. among the most recent work is that from ruffo (2022) who questions the lack of inclusion of literary translators in the discussion on technological advances of the automatic translation tools. a survey was conducted to identify the perception of technology applied to translation by professional translators. although professional translators are not reluctant to technology, the negative views mostly had to do with the use and development of translation tools applied to literature. ## methodology: data and experiment four texts from different literary genres were selected to establish their impact on the quality of mt; this study also sought to determine if the time period of the literary texts could influence the performance of mt. from the prose genre, two novels were chosen: pride and prejudice by jane austen, and 1984 by george orwell; the poetry was represented by the following two poems: i felt a funeral in my brain by emily dickinson, and siren song by margaret atwood. see in order to objectively assess the quality of mt performance, we implemented the bleu score metric @xcite . to this end, we chose two human translations from different time periods for each selected text. the bleu metric system was set up both at sentence level with sentence_bleu() function and at corpus level with corpus_bleu() function contained in the nltk (natural language toolkit) library, as well as a cumulative score is obtained by assigning a cumulative weighting of 4-grams. in addition to the bleu score, the most significant linguistic features of each text were described, and the approaches taken by both the machine translation and the human translation were compared and correlated with the bleu score. ## results and discussion the evaluation results calculated by bleu suggested that the lowest (0.3 out of 100) and highest (39.79 out of 100) quality scores were both achieved by systran. however, reviewing the questions put forward in the section 1 of this paper, it is safe to consider the following results. the mt system which achieved best performance out of the three was deepl, estimating the average bleu score for the four texts. overall, deepl system was capable of producing a translation similar to the human translation with a higher consistency and quality as compared with the other systems. although systran and yandex made the same type of mistakes, yandex was more consistent and closer to deepl (see therefore, according to bleu score, mt fared much better on poetry than prose when compared to a human translation. see finally, all three mt systems delivered a lowquality output on the older classical works (those by jane austen and emily dickinson) as see in in summary, the mt systems were able to perform better on modern literature which is expected to have a less complex style. we conjecture that another reason for that is because nmt systems are usually trained on more contemporary data. ## limitations it should be noted that the results and conclusions should not be taken fully representative due to the following limitations of this study: ## conclusion the aim of this study is to analyse the performance of the mt systems selected for to this end, three nmt systems (deepl, systran and yandex) were selected to assess the performance and quality when translating prose and poetry from different time periods. the results suggest that the best performing system on these texts according to our experiments was deepl. this nmt system produces more coherent and similar texts to those produced by humans. in addition, the obtained bleu scores show that: a) mt fares better on poetry and does not do so well on prose b) mt delivers better results on modern contemporary texts and does not do so well on older classic texts. however, it is essential to acknowledge the limitations of this study as outlined in the previous section. in future studies, it will be preferrable to use larger and more representative data in the experiments. the bleu evaluation metric could be compared and correlated with other metrics such as ter (translation error rate) and wer (word error rate) to.