Comparing Feature-Engineering and Feature-Learning Approaches for Multilingual Translationese Classification (2021)

Abstract: Traditional hand-crafted linguistically-informed features have often been used for distinguishing between translated and original non-translated texts. By contrast, to date, neural architectures without manual feature engineering have been less explored for this task. In this work, we (i) compare the traditional feature-engineering-based approach to the feature-learning-based one and (ii) analyse the neural architectures in order to investigate how well the hand-crafted features explain the variance in the neural models’ predictions. We use pre-trained neural word embeddings, as well as several end-to-end neural architectures in both monolingual and multilingual settings and compare them to feature-engineering-based SVM classifiers. We show that (i) neural architectures outperform other approaches by more than 20 accuracy points, with the BERT-based model performing the best in both the monolingual and multilingual settings; (ii) while many individual hand-crafted translationese features correlate with neural model predictions, feature importance analysis shows that the most important features for neural and classical architectures differ; and (iii) our multilingual experiments provide empirical evidence for translationese universals across languages.

## introduction texts originally written in a language exhibit properties that distinguish them from texts that are the result of a translation into the same language. these properties are referred to as translationese @xcite . earlier studies have shown that using various hand-crafted features for supervised learning can be effective for translationese classification @xcite @xcite . however, this approach has a number of limitations. firstly, * equal contribution. moreover, most previous approaches have focused on classifying translationese in the monolingual setting, i.e. translations come from one or multiple source languages, but the language on which to perform the classification is always the same. to the best of our knowledge, the multilingual setting with multiple source and target languages has not been explored yet. if translationese features are language-independent or shared among languages, multilingual translationese classification experiments would show the effect. we perform binary translationese classification not only in mono-, but also in multilingual settings to empirically verify the existence of translationese universals throughout different source and target languages. in our work we investigate: we show that: the paper is organized as follows. section 2 describes related work. section 3 introduces the architectures used in our study. section 4 discusses the data and presents the main classification results. we perform cross-data evaluations in section 5 and analyze feature importance and correlation in section 6. finally, we summarize and draw conclusions in section 7. ## related work recent work on translationese, both on humanand machine-translated texts, explores topics ranging from translationese characterization @xcite @xcite to unsupervised classification @xcite , to exploring insights into the structure of language typologies with respect to different translationese properties @xcite @xcite , to the effects on downstream tasks such as machine translation @xcite @xcite @xcite @xcite , to translationese data collection @xcite @xcite . traditional translationese classification approaches rely on manually designed features, such as n-gram frequencies on tokens, part-of-speech (pos) tags or lemmas @xcite @xcite , function word frequencies @xcite , character-level features @xcite , surface and lexical features @xcite , syntactic features @xcite , morpheme-based features @xcite , information-density-based features @xcite , etc. by contrast, to date neural approaches to translationese @xcite have received less attention. while @xcite have used learned language representations to show that the distance in the representational space reflects language phylogeny, dutta chowdhury et al. @xcite use divergence from isomorphism between embedding spaces to reconstruct phylogenetic trees from translationese data. @xcite train a bilstm for translation direction identification and report accuracy up to 81.0% on europarl data. ## multilinguality and cross-language performance since neural architectures perform better than the non-neural ones, we perform the multilingual and cross-language analysis only with the neural models. we evaluate the models trained on one dataset on the other ones, in order to verify: another trend that can be observed is that in cross testing trg-src 1 and trg-src 2 , the model where the source language is more distant from the target suffers larger performance drop when tested on the test set with the closer-related source language, than the other way around. for instance, the de-es model tested on the de-en data suffers a decrease of 17.8 points, and de-en model tested on the de-es data suffers a decrease of 9.8 points. this may be due to de-en having learned more of the general translationese features, which helps the model to obtain higher accuracy on the data with a different source, while the de-es model may have learned to rely more on the language-pair-specific features, and therefore it gives lower accuracy on the data with the different source. a similar observation has been made by @xcite . for the multi-source monolingual models (trg-all), testing on trg-src 1 and trg-src 2 datasets shows a slight increase in performance for a source language that is more distant from the target, and a slight decrease for the more closelyrelated source language (as compared to testing trg-all on trg-all). we also compare the performance of all-all[3] on different test sets to the original performance of the models trained on these datasets (in parentheses). there is a relatively larger drop in accuracy for the trg-src data, than for trg-all data. the largest drop for neural models is 6.7 accuracy points whilst the smallest performance drop for the handcr.+svm is 12.7. this highlights the ability of the neural models to learn features in a multilingual setting which generalize well to their component languages whereas the handcr.+svm method does not seem to work well for such a case. however, for all-all[8] models, in summary, we observe that: ## feature importance and relation to neural models in this section we aim to quantify the feature importance of the hand-crafted linguistically inspired features used in handcr.+svm according to different multilingual models (all-all[3] setting). as we use a support vector machine with a linear kernel, we can interpret the magnitude of the feature weights as a feature importance measure. @xcite for instance, use the squared svm weights for feature selection. we rank the features by the absolute value of the weight. the feature ranks are listed in appendix a.1. figure 1 shows the top 10 features. paragraph length is the most relevant feature, and we observe that most of the top features correspond to paragraph log probability. these features characterize simplification in translationese. to explore whether there is any correlation between the hand-crafted features and predictions of the trained neural models, we conduct the following experiment in the multilingual setting. we fit a linear regression model for each hand-crafted feature, using the estimated probabilities of neural model as gold labels to be predicted. more formally, with n paragraphs (p i , @xmath0...n) in the test set and d features, for each feature vector x @xmath1, @xmath2...d we fit a model we apply min-max normalization to the features. we find that a large proportion of the linguistically motivated features are statistically significant for predicting the neural models' predicted probabilities, namely 60 features (out of 108) are significant for lstm, 38 for the simplified transformer, and 56 for bert, each with probability 99.9%. we also fit the per-feature linear models to predict the actual gold labels (and not the predictions of the neural models) to investigate which features correlate with the ground truth classes, and find 55 features to be statistically significant with 99.9% probability. the full list of statistically significant features for each model, as well as for the gold labels is given in the appendix a.1. we observe that the features significant for the neural models largely overlap with the features significant for the gold labels: the f 1 -score (as a measure of overlap) is 0.89 for lstm, 0.75 for simplified transformer and 0.99 for bert. this is expected, because highperforming neural models output probabilities that are generally close to the gold labels, therefore a similar correlation with hand-crafted features occurs. the r 2 measure is further used to rank features based on the amount of explained variance in predictions of a model. the top 10 features for predicting the predictions of each neural model and for predicting the actual gold labels are displayed in figure 2 . the order of top features is similar across the neural models (pairwise rank correlations ρ spearman of at least 0.76), and similar to, but not identical to, the gold label results (pairwise rank correlations ρ spearman of at least 0.75). we observe that most of the top features are either posperplexity-based, or bag-of-pos features. these features characterize interference in translationese. it also appears that more importance is attached to perplexities based on unigrams and bigrams than on other n-grams. notably, the order of feature importance for the neural models is highly dissimilar from the order of hand-crafted feature weights for the svm (pairwise rank correlations ρ spearman at most 0.23). this might be connected to an accuracy gap between these models. ## summary and conclusion this paper presents a systematic comparison of the performance of feature-engineering-based and feature-learning-based models on binary translationese classification tasks in various settings, i.e., monolingual single-source data, monolingual multi-source data, and multilingual multi-source data. additionally, we analyze neural architectures to see how well the hand-crafted features explain the variance in the predictions of neural models. the results obtained in our experiments show that, (i) representation-learning-based approaches outperform hand-crafted linguistically inspired feature-selection methods for translationese classification on a wide range of tasks, (ii) the features learned by feature-learning based methods generalise better to different multilingual tasks and (iii) our multilingual experiments provide empirical support for the existence of language independent translationese features. we also examine multiple neural architectures and confirm that translationese classification requires deep neural models for optimum results. we have shown that many traditional hand-crafted translationese features significantly predict the output of representation learning models, but may not necessarily explain their performance due to the weak correlation. our experiments also show that even though single-source monolingual models yield the best performance, they can, to a reasonable extent, be substituted by multi-source mono-and multi-lingual models. our interpretability experiment provides only some initial insight into the neural models' performance. even though there are significant relationships between many of the features and the neural models' predicted probabilities, further experiments are required to verify that the neural models actually use something akin to these features. also our current approach ignores interaction between the features. in the future, we plan to conduct a more detailed analysis of the neural models' decision making.