The MAKE - NMTVIZ System Description for the WMT 23 Literary Task (2023)

Abstract: This paper describes the MAKE-NMTVIZ Systems trained for the WMT 2023 Literary task. As a primary submission, we used Train, Valid1, test1 as part of the GuoFeng corpus (Wang et al., 2023) to fine-tune the mBART50 model with Chinese-English data. We followed very similar training parameters to (Lee et al. 2022) when fine-tuning mBART50. We trained for 3 epochs, using gelu as an activation function, with a learning rate of 0.05, dropout of 0.1 and a batch size of 16. We decoded using a beam search of size 5. For our contrastive1 submission, we implemented a fine-tuned concatenation transformer (Lupo et al., 2023). The training was developed in two steps: (i) a sentence-level transformer was implemented for 10 epochs trained using general, test1, and valid1 data (more details in contrastive2 system); (ii) second, we fine-tuned at document-level using 3-sentence concatenation for 4 epochs using train, test2, and valid2 data. During the fine-tuning, we used ReLU as an activation function, with an inverse square root learning rate, dropout of 0.1, and a batch size of 64. We decoded using a beam search of size. Four our contrastive2 and last submission, we implemented a sentence-level transformer model (Vaswani et al., 2017). The model was trained with general data for 10 epochs using general-purpose, test1, and valid 1 data. The training parameters were an inverse square root scheduled learning rate, a dropout of 0.1, and a batch size of 64. We decoded using a beam search of size 4. We then compared the three translation outputs from an interdisciplinary perspective, investigating some of the effects of sentence- vs document-based training. Computer scientists, translators and corpus linguists discussed the linguistic remaining issues for this discourse-level literary translation.

## introduction in order to analyse literary translations, we have gathered an interdisciplinary team of translators, linguists and computational scientists. we used this opportunity to explore neural machine translation of literary texts as a test set for test suites and unsolved issues for mmt literary translations, especially for the chinese-english language pair. while the topic of literary machine translation has gained momentum in the last years, there have still been few attempts to customize systems to liter-ary data, although this idea is also drawing attention @xcite . indeed, research has been carried out on this subject, notably on catalan @xcite , but also on slovenian @xcite , german and russian @xcite , and on french @xcite , where research suggests that mt systems can be further fine-tuned on specific genres and individual translator styles @xcite . of course, these very attempts bring about many issues concerning textual ownership, copyright, translator status and livelihood, possibly lowered quality, cognitive friction, etc. @xcite . it is therefore important to include these ethical aspects into the research and clarify its objectives: for instance, whether mt should serve as a reading aid @xcite , or as a postediting tool that may decrease the effort needed to translate @xcite and constrain creativity @xcite . part research has also focused on evaluating the use of existing tools for literary texts. in the context of chinese to english, attention has been paid to some of the specific shortcomings of mt systems, such as the translation of adjectival possessive pronouns @xcite , or theme-rheme progressions @xcite . such limitations can indeed have a drastic impact on readers' acceptance, which @xcite explores in the context of online folktales, confirming that the text's function plays a large role in this respect. lastly, @xcite have also pointed the incompatibility of mt metrics, document-level or otherwise, for literary texts, concluding that "hu-man expert evaluation is currently the only way to judge the quality of literary mt". the rest of the paper is organised as follows: section 2 details our approaches to the task and the training data of our experiments, section 3 presents the results and section 4 discusses them. ## data and tools used this section details the toolkits we used and our training data for the three submissions authorised for the task. we first used part of the training data proposed by the organisers @xcite to observe the translations from mbart50 from chinese into english before fine-tuning mbart (primary submission). we then used a fine-tuned context-aware concatenation-based transformer trained at document level (contrastive1 submission) and a traditional sentence-level transformer (con-trastive2 submission). ## experiments and results we provide a human analysis of the primary model by discussing the improvements observed with the mbart fine-tuning with respect to the baseline. additionally, we report the bleu scores of our three systems. ## conclusion this paper presented the make-nmtviz system description for the wmt2023 literary shared task. we participated in the chinese-to-english task with a model trained at sentence level and at document level. we only used the data provided by the organisers but also analysed the translations produced with mbart50 before our submissions. as we did not receive scores from the organisers of the task, we mostly focused on the qualitative analysis of our translations. we resorted to a typology of translation errors and highlighted prominent error types that remained in our translations.