Incorporating Human Translator Style into E nglish- T urkish Literary Machine Translation (2023)

Abstract: Although machine translation systems are mostly designed to serve in the general domain, there is a growing tendency to adapt these systems to other domains like literary translation. In this paper, we focus on English-Turkish literary translation and develop machine translation models that take into account the stylistic features of translators. We fine-tune a pre-trained machine translation model by the manually-aligned works of a particular translator. We make a detailed analysis of the effects of manual and automatic alignments, data augmentation methods, and corpus size on the translations. We propose an approach based on stylistic features to evaluate the style of a translator in the output translations. We show that the human translator style can be highly recreated in the target machine translations by adapting the models to the style of the translator.

## corpus compilation in this study, we have compiled two corpora, the translator corpus and the reference corpus foot_1 . the translator corpus is an english-turkish bilingual corpus and the reference corpus consists of turkish monolingual texts. ## automatic alignment manual alignment is a time-consuming job that requires meticulousness. although it is absolutely necessary to manually align the english and turkish books at least for the purpose of evaluation to arrive at reliable assessments, automatic alignment is a preferable method regarding human resources and time during the training phases. in this research, we worked with the hunalign sentence aligner 5 @xcite to automatically align the texts. however, the automatic alignment resulted in a considerable amount of erroneously aligned sentences, which deteriorated the translation performance when used as a parallel corpus. the problem was mostly caused by the omissions performed by the translator at hand from the original english text, or the merges of multiple english sentences into a single turkish sentence. to eliminate the incorrectly aligned sentence pairs, we devised a method that makes use of machine translations of source sentences. the english sentence in each english-turkish sentence pair in the hunalign output is translated into turkish using the pre-trained mt model that we use in this work (opus-mt-tc-big-en-tr, see section 6). by taking this translation as reference and the turkish sentence in the hunalign output as prediction, we computed the bleu, meteor, google bleu (gleu, @xcite al. (2016) ), and bertscore f1 @xcite scores that evaluate the match between the two turkish sentences. taking these four scores as features, we trained an svm @xcite model that predicts whether it is a correct alignment or not with a training set of 20 manually aligned books through the scikitlearn library @xcite . in all of our automatically aligned datasets explained in section 9.1, we used this svm model to extract the correct alignments from the hunalign outputs and 5 https://github.com/danielvarga/hunalign ignored the rest. ## machine translation model the transformer architecture @xcite is dominant in the machine translation area, reaching state-of-the-art results in many language pairs. however, it is difficult to achieve high generalization in non-general domains, especially in the literary domain without a large training set. this is especially so if the research relies on capturing the style of a specific translator, in which case we face with the scarcity of the training data in addition to the cost and effort required in compiling and aligning the data. even though all of the books of a translator are retrieved and aligned, the number of sentences may be as low as 200k. this amount of data is not adequate to train a successful transformer model without augmentation. taking turkish-english machine translation at hand, the findings of wmt17 and wmt18 @xcite show that all of the participating systems make use of back-translation in some way or another, and the state-of-the-art results are achieved by the university of edinburgh, where the initial news corpus of 200k sentences has been oversampled five times and augmented with 2.5m back-translated and 1m copied sentences @xcite . keeping the importance of data in mind, we also observe a trend in nlp, where large pre-trained transformer language models receive high popularity due to their success in various downstream tasks just by fine-tuning with much smaller training sets. the newest advances include text-totext transformer models such as t5 @xcite , faster and more efficient ways of scaling and training text-to-text language models @xcite , and combinations of different denoising objectives @xcite . these recent trends brought to mind leveraging a large pre-trained machine translation model, and fine-tuning on the small training set that we obtain from the books of a specific translator. with this motivation, we selected helsinki-nlp's english-turkish pre-trained transformer models trained as part of the opus-mt project foot_4 (tiedemann and thottingal, 2020). the models have been trained on the english-turkish opus corpus foot_5 @xcite and the corpus gathered in the scope of the tatoeba challenge @xcite in the marian-nmt framework @xcite . we used the opus models in the huggingface platform @xcite , specifically the opus-mt-tc-big-en-tr foot_6 model for the english-turkish direction, which is the main translation direction in this research, since we aim to mimic the style of a turkish translator. the turkish-english translation direction has only been used for back-translation, where the opus-mttc-big-tr-en foot_7 model has been exploited. the english-turkish pre-trained transformer model has been fine-tuned on different training sets for 5 epochs which was seen as the optimal epoch number on the validation set, with a batch size of 64 fit into 4 tesla v100 gpus. the maximum source and target sentence lengths have been selected as 128, and the learning rate as 2e-5 using the adam optimizer with weight decay (0.1) @xcite . ## augmentation creating parallel data for training machine translation models is extremely challenging, whereas monolingual data in nearly all the languages are abundant. literary machine translation requires a large amount of literary parallel data, which is currently unavailable and very expensive to align. due to the low number of aligned literary data, two data augmentation methods have been carried out in this research to increase the quality of literary machine translation. ## stylistic evaluation we quantify the style of a translation text using the set of 29 numeric features listed in we use two metrics, cosine similarity and pearson's correlation coefficient, to measure the style match between the two translations of a text. the main motivation behind this choice is based on the assumption that texts with similar style have similar style vectors and these metrics adequately show the similarity between vectors. for the stylistic evaluation of a machine translation model on a test set, we take the translation output by the model and the original translation of the translator as the two translations and employ the similarity and correlation metrics on the style vectors. the expectation is to have high similarity and correlation scores if the model output is stylistically similar to the translation of the translator. ## conclusions in this paper, we proposed an approach for literary machine translation that can adapt itself to the style of a translator and produce translations close to that style. as a case study, we focused on the english-turkish language pair and a distinguished turkish literary translator. in this direction, we leveraged a large pre-trained machine translation model and fine-tuned it on the works of the translator. the experiments were conducted using both manually and automatically aligned data compiled from the books of the translator. we also tested the effect of two data augmentation methods, self-training and back-translation, on the performance. to measure how much the translations obtained by the fine-tuned model reflect the style of the translator, we made a detailed analysis of literary style and identified a set of stylistic features. the experiments showed that adapting a pre-trained model to the works of a translator increases the bleu score about 45-56% on the literary data and captures the translator's style 18-40% better in terms of cosine similarity compared to the pre-trained model. as future work, we plan to incorporate other evaluation metrics in addition to the bleu score that can capture the semantics of the translations better. we also aim at conducting a human evaluation for both translation quality and stylistic properties. another interesting direction will be including other literary translators, adapting the machine translation models to different styles, and experimenting with style transfer between works of the translators.