Detecting Non-literal Translations by Fine-tuning Cross-lingual Pre-trained Language Models (2020)

Abstract: Human-generated non-literal translations reflect the richness of human languages and are sometimes indispensable to ensure adequacy and fluency. Non-literal translations are difficult to produce even for human translators, especially for foreign language learners, and machine translations are still on the way to simulate human ones on this aspect. In order to foster the study on appropriate and creative non-literal translations, automatically detecting them in parallel corpora is an important step, which can benefit downstream NLP tasks or help to construct materials to teach translation. This article demonstrates that generic sentence representations produced by a pre-trained cross-lingual language model could be fine-tuned to solve this task. We show that there exists a moderate positive correlation between the prediction probability of being human translation and the non-literal translations’ proportion in a sentence. The fine-tuning experiments show an accuracy of 80.16% when predicting the presence of non-literal translations in a sentence and an accuracy of 85.20% when distinguishing literal and non-literal translations at phrase level. We further conduct a linguistic error analysis and propose directions for future work.

## introduction translation is a cross-lingual and intercultural process helping to communicate across language barriers. human-generated parallel corpora have been largely exploited to develop machine translation (mt) systems, which rapidly evolve and are widely used in production nowadays @xcite @xcite @xcite . multiple human-generated references could be used to evaluate each mt hypothesis, since the same meaning could be expressed in different ways. when we take a closer look at human translations, apart from literal translations, fixed corresponding expressions or the most commonly used renderings, various translation techniques could be employed to produce non-literal translations. the resulting variations reflect the richness of human languages. furthermore, because of the existing differences between languages and cultures, it is sometimes inevitable to translate non-literally to ensure adequacy and fluency. different techniques of producing non-literal translations are systematically studied by linguists and translation scholars, such as generalization, particularization, modulation, transposition, etc. @xcite @xcite . based on these studies, @xcite investigated the possibility of annotating and automatically recognizing different translation techniques at sub-sentential level. for natural language processing (nlp) tasks, non-literal translations can bring difficulties for automatic word alignment @xcite or for paraphrase extraction via bilingual pivoting @xcite it's nothing if not ambitious. → c'est véritablement ambitieux. (lit. it's really ambitious.) that's why we've made it our primary goal to spend a large sum of money on an advertising effort to help bring out and complicate the truth about coal. → c'est pourquoi nous avons décidé de dépenser massivement pour une campagne de communication destinée à dire, et à brouiller, la vérité sur le charbon. (lit. that's why we decided to spend massively on a communication campaign intended to tell, and blur, the truth about coal.) in order to foster the study on non-literal translations, automatically detecting them in parallel corpora is an important step, which can help constructing materials to teach translation to human learners or serve as the first step to assembling as much representative data as we can to train mt systems to start producing more non-literal translations than their literal alternatives. our research questions are the following: do non-literal translations occur more often in human translations? could pre-trained language models be fine-tuned to detect the presence of non-literal translations in a sentence? and could the architecture be adapted to distinguish literal and non-literal translations at phrase level? to answer these questions, our approach is based on the advances in cross-lingual pre-trained language models. recently, @xcite introduced a new supervised learning objective that improves cross-lingual language model's pre-training when parallel data is available. 2 having a neural architecture based on transformer @xcite , their pre-trained cross-lingual lms (xlms) provide general-purpose text representations, which can encode any sentence into a shared embedding space. for our task of detecting non-literal translations, we use an english-french corpus of ted talks annotated with translation techniques at sub-sentential level @xcite , and we demonstrate that the generic sentence representations produced by xlm are transferable to our task after fine-tuning. ## related work research at the intersection of translation studies and nlp around non-literal translations has attracted the attention of many researchers. in translation theories, different typologies of translation techniques are proposed to formalize human translators' choices when they translate non-literally @xcite @xcite . with established annotation guidelines and an adapted typology, @xcite annotated en-fr and en-zh parallel corpora with translation techniques, and conducted multi-class classification of translation techniques. non-literal translations could bring difficulties for nlp and inappropriate ones could be undesirable for certain tasks. @xcite semi-automatically identified, categorized and quantified seven types of translation divergences between chinese and english, which arise either because of cross-lingual differences or because of non-literal translations. various models are proposed to automatically detect translation divergences in parallel corpora, with the goal of filtering out divergent sentence pairs to improve mt systems' performance @xcite @xcite . in the process of bilingual pivoting paraphrasing @xcite , non-literal translations could be a reason leading to diverse semantic relations in the resource ppdb 2.0 apart from true paraphrases @xcite . on the other hand, non-literal but appropriate translations are not negligible and are studied by different approaches. @xcite investigated the effects of cross-lingual syntactic and semantic distance on translation production times and found that non-literality makes from-scratch translation and post-editing difficult. in a case study of comparing human and machine translations, @xcite suggested that the human translator used several procedures that seem to be beyond the reach of the mt system (such as sentence splitting, shifts of phrase function and/or category, explicitation, modulation and paraphrasing). the project of @xcite aims to preserve cultural heritage and language diversity by analyzing the translation adaptations in multilingual corpora of translated literary texts, which is particularly important for low-resource languages. in order to provide insights for discourse-aware mt system's development, discourse-related language contrasts are analyzed for english-croatian and english-german @xcite . for inspiring mt's further improvement on fluency and for human translators' reference, @xcite proposed a method for detecting free translation examples from bilingual parallel corpora, which is based on an innovative use of attention scores. @xcite proposed a stacked neural network for finegrained human translation quality estimation, and they discussed that this model has limited validity for adequate scoring of free but still valid translations. in this paper, by using the english-french ted talks corpus annotated with translation techniques by @xcite , we investigate whether pre-trained cross-lingual language models could be finetuned to detect non-literal translations at sentence and phrase level. for the latter, we compare the results with those obtained by @xcite . ## detecting non-literal translations at sentence level our first goal is to detect whether there are non-literal translations in a sentence. by assuming that human translators employ more non-literal translations than machines @xcite , we first transfer the problem into training a model to distinguish human and machine translations @xcite , expecting that the classifier would learn the linguistic differences between them and further help predict the presence of non-literal translations in a sentence. after training this human-vs-machine translation classifier, we investigate whether there is a positive correlation between the prediction probability of human translation and the non-literal translations' proportion in a sentence. finally, we test the hypothesis that resuming the fine-tuning task on detecting the presence of non-literal translations in a sentence after loading this human-vs-machine translation classification model could get better performance. ## comparison with state of the art and discussion for our detection task at phrase level, we present the confusion matrix in in order to compare the performance of this xlm-based classifier with a non-neural classifier which leverages linguistic features, we reuse the 198 features designed by @xcite . after a ten-fold cross validation on the same training data, a tuned randomforest classifier foot_5 gives the best average accuracy of 82.91%±1.28%. we then apply the saved randomforest model trained on all available data on the same test set, and get an accuracy of 90.90%. the confusion matrix is shown in the two confusion matrices show that the xlm-based classifier is better at detecting literal translations, whereas the linguistic features-based classifier is better at detecting non-literal translations. an oracle study shows that if we could unite the strength of both, only 7 wrong cases remain (1 non-literal translation predicted as literal, and 6 literal translations predicted as non-literal), which will result in an accuracy of 98.01%. therefore, a hybrid classifier could be investigated in future work to improve the performance. ## conclusion and perspectives in this paper, we concentrate on the detection of non-literal translations by fine-tuning pre-trained crosslingual language models (xlm). 13 a subset of ted talks corpus manually annotated with translation techniques is utilized to ensure that we deal with appropriate human non-literal translations. we first train a human-vs-machine translation classifier with different corpus genres and we show that there exists a moderate positive correlation between the prediction probability of being human translation and the non-literal translations' proportion for a sentence. the presence of the translation techniques modulation and modulation transposition tends to make the classification as a human translation more difficult. for detecting whether a sentence contains non-literal translations, resuming the fine-tuning after loading the final trained human-vs-machine translation classifier brings a gain of performance (best accuracy 80.16%) than directly fine-tuning xlm. after adapting the architecture, our xlm fine-tuning at phrase level to distinguish literal and non-literal translations obtains an accuracy of 85.2%. we con-ducted detailed error analysis and compared the xlm-based classifier with a linguistic features-based randomforest classifier. the oracle study shows that there exists a complementarity between the two methods, therefore a hybrid classifier could be investigated in future work to improve the performance. the automatic bilingual word alignment being a non-trivial task in itself @xcite , our detection of non-literal translations at phrase level is currently based on already aligned pairs. in future work, it is important to leverage the advances of automatic alignment to reduce the reliance on manual work. to observe the generalization performance, one could extend these experiments to a more dissimilar language pair, for example english-chinese. the fine-tuned models could be used to help constructing material to teach translation and to analyze the usage of different translation techniques between languages. we aim to automatically construct a corpus containing abundant non-literal translation phenomena based on this study, which we hope to be useful for the research on evaluating automatic word alignment, and for giving inspiration for mt's development to produce more appropriate non-literal translations.