Towards Tailored Recovery of Lexical Diversity in Literary Machine Translation (2024)

Abstract: Machine translations are found to be lexically poorer than human translations. The loss of lexical diversity through MT poses an issue in the automatic translation of litrature, where it matters not only what is written, but also how it is written. Current methods for increasing lexical diversity in MT are rigid. Yet, as we demonstrate, the degree of lexical diversity can vary considerably across different novels. Thus, rather than aiming for the rigid increase of lexical diversity, we reframe the task as recovering what is lost in the machine translation process. We propose a novel approach that consists of reranking translation candidates with a classifier that distinguishes between original and translated text. We evaluate our approach on 31 English-to-Dutch book translations, and find that, for certain books, our approach retrieves lexical diversity scores that are close to human translation.

## introduction with the introduction of neural machine translation (nmt), the performance of high-resource automatic translation has improved substantially. especially since the introduction of the transformer architecture @xcite , state-of-theart nmt systems have outperformed previous approaches considerably @xcite , with some works even claiming human parity @xcite . however, these claims are based mostly © 2024 the authors. this article is licensed under a creative commons 4.0 licence, no derivative works, attribution, cc-by-nd. on accuracy and fluency measures, while style is often overlooked. in fact, according to expert evaluation, machine translation (mt) did actually not reach human parity @xcite . for instance, mt models have been found to exacerbate linguistic patterns that occur frequently, while underrepresenting patterns that are found less commonly @xcite . as a result, automatically translated texts are found to be lexically poorer than human translations (ht). this 'artificially impoverished language' has previously been referred to as machine translationese @xcite . in this paper, we focus on the translation of novels. contrary to technical domains, where meaning preservation is the main criterion for acceptable translations, literary translations have the additional criterion of style. this is because apart from meaning preservation (what is written), maintaining a certain reading experience (how it is writ-ten) is vital for novels @xcite . importantly however, writing style (and linguistic complexity) can vary considerably between books. some books contain repetitive language use, while others are written in embellished language (see section 3). current approaches that aim to mitigate the loss of lexical diversity do not accommodate this. state-of-the-art previous work @xcite increases lexical diversity in a rigid way, not allowing for flexibility at inference time. contributions (i) we show that lexical diversity varies considerably across books, and argue that this should be taken into account in mt; (ii) we introduce a novel flexible method for recovering lexical diversity in mt, informed by the diversity of the original. (ii) we evaluate our method on 31 english novels which are translated to dutch, and find that our approach is effective when it comes to book-tailored promotion of lexical diversity. machine translationese following recommendations from jiménez-crespo (2023), we will largely refrain from using the term translationese in the rest of this paper. however, it is important to note that previous work that aims to increase lexical diversity in mt has mostly been framed as part of 'machine translationese' reduction @xcite @xcite . translations have been found to differ from original texts in a number of ways. for one, @xcite argues that human translations into a language tend to be lexically simpler than text originally written in that language. automatic classification approaches have been effective in detecting this difference @xcite @xcite @xcite . more recently, work has investigated linguistic differences between @xcite . thus, it seems that modelling characteristics of original versus translated texts has a direct link to lexical diversity. previous work @xcite leveraged these detectable differences in their approach to increase the naturalness of output translations. we take inspiration from their lexical diversity evaluation methods, and implement their method as a baseline. ## why recover rather than increase lexical diversity? in this paper, we argue for tailored recovery of lexical diversity. in this section, we first discuss support for this idea from the field of literary studies. then, we provide empirical evidence by applying lexical diversity metrics to our test set. ## reranking method as illustrated in figure 1 , our approach consists of two parts: hypothesis generation and hypothesis reranking. firstly, we generate the n best translation candidates for each source sentence in the test set with a vanilla domain-specific mt system (section 5.1). note that we decode all books separately, instead of concatenating all test set books. then, for each book, we apply a classifier (section 5.2) to the translation hypotheses and, through a softmax layer, obtain the probability for each candidate that it is an original dutch sequence. based on these probabilities, we rerank the translation candidates. in order to obtain the (expected) most lexically rich candidate, we would then choose the rank with the highest original-text probability. however, note that this simple approach is flexible in the sense that, instead of choosing the most original-like option, we have the option to choose a lower originaltext rank. we leverage this flexibility for tailoring rank selection to the lexical diversity of the original english book. first, for each original book, we calculate a lexdiv score, which consists of the average of the normalized ttr, yule's i and mtld scores (see section 6). then, we bin the books according to their lexdiv score, relative to the total distribution. that is, given a list that is sorted based on lexdiv, we categorize these into groups, where the number of groups depends on the number of nbest candidates in decoding. for example, for @xmath0, we bin the books into 5 different groups of 6 books (adding any remainders into the last bin). the bin per book corresponds to the original-text rank that is selected. as such, the selected rank for each book depends on the lexical diversity of its source, relative to the other books. reranking translation candidates is a suitable solution to our task, because it accomodates flexibility, which is tunable at inference time. there is no need to train a separate model per diversity setting, saving computational expenses. additionally, our approach is model-agnostic: reranking can be applied to any mt model that can generate multiple translation candidates. we train a transformer-based translation model using the fairseq toolkit @xcite . more specifically, we use the transformer iwslt de en architecture. this is a transformer base model with 6 encoder and decoder layers and an embedding dimension of 512. during training, we use an adam optimiser, a learning rate of 5e-4, the loss function cross entropy with label smoothing 0.1 and the batch size is 64. each model is trained until convergence with a patience of 3 epochs, using the bleu score as a maximisation metric for finding the best checkpoint. decoding strategies by default, we use beam search for decoding. reranking approaches rely heavily on the diversity of the translation hypotheses: if the hypotheses are all very similar, reranking them is not likely to have a large effect. to ensure diverse hypotheses, we use a beam size of 20. additionally, we experiment with decoding through diverse beam search @xcite . we follow @xcite by using 3 groups, with a beam size of 21. beyond beam search, we investigate the effects of top-k and topp sampling, with the default parameters and sampling size 10. ## evaluation we introduce three classes of metrics. firstly, we look at general text metrics, which are commonly used for evaluating lexical diversity. secondly, we use translation-specific metrics. lastly, we evaluate the general translation quality. ## conclusion we have argued for flexible recovery of lexical diversity in literary mt. we showed that default diversity varies per book in our dataset, and that this lexical diversity is partially lost through mt. we presented the first approach towards tailored rescoring of translation candidates, which matches ht more closely than previous baselines for some books. future work could explore how our method can be combined with previous work, as it is in principle model-agnostic. investigations with document-level translation, instead of sentencelevel translation only, could provide additional insights. furthermore, it may be useful to address this task at an even finer-grained level, by exploring diversity reranking on a sequence-level, instead of a book-level.