Explaining Translationese: why are Neural Classifiers Better and what do they Learn? (2022)

Abstract: Recent work has shown that neural feature- and representation-learning, e.g. BERT, achieves superior performance over traditional manual feature engineering based approaches, with e.g. SVMs, in translationese classification tasks. Previous research did not show (i) whether the difference is because of the features, the classifiers or both, and (ii) what the neural classifiers actually learn. To address (i) , we carefully design experiments that swap features between BERT- and SVM-based classifiers. We show that an SVM fed with BERT representations performs at the level of the best BERT classifiers, while BERT learning and using handcrafted features performs at the level of an SVM using handcrafted features. This shows that the performance differences are due to the features. To address (ii) we use integrated gradients and find that (a) there is indication that information captured by hand-crafted features is only a subset of what BERT learns, and (b) part of BERT’s top performance results are due to BERT learning topic differences and spurious correlations with translationese.

## introduction translationese is a descriptive (non-negative) cover term for the systematic differences between translated and originally authored text in same language @xcite . some aspects of translationese such as source interference @xcite are language dependent, others are presumed universal, e.g. simplification, explicitation, overadherence to target language linguistic norms @xcite in the products of translations. while translationese effects can be subtle, especially for professional human translation, corpus-based studies @xcite and, in particular, machine-learning and classifier based studies (rabinovich and wintner, 2015; volansky * equal contribution. et al., 2015; rubino et al., 2016; pylypenko et al., 2021) clearly reveal the differences. while research on translationese is important from a theoretical point of view (translation universals, specific interference), it has a direct impact on machine translation research: @xcite @xcite @xcite @xcite , amongst others, show that translation direction in training and test data impacts on results, that already translated test data are easier to translate than original data, that machine translation and post-editing result in translationese, and that mitigating translationese in mt output can improve results. translationese impacts cross-lingual applications, e.g. question answering and natural language inference @xcite @xcite . in this paper we focus on machine-learningclassifier-based research on translationese. here, typically a classifier is trained to distinguish between original and translated texts (in the same language). until recently, most of this research @xcite @xcite used manually defined, often linguistically inspired, feature-engineering based sets of features, mostly using support vector machines (svm). once a classifier is trained, feature importance and ranking methods are used to reason back to what aspects of the input is responsible for (i.e. explains) the classification (and whether this accords with linguistic theorisation). more recently, a small number of papers explored feature-and representation-learning neural network based approaches to translationese classification @xcite . in a systematic study @xcite show that featureand representation-learning deep neural networkbased approaches (in particular bert-based, but also other neural approaches) to translationese classification substantially outperform handcrafted feature-engineering based approaches using svms. however, to date, two important questions remain: (i) it is not clear whether the substantial performance differences are due to learned vs. handcrafted features, the classifiers (svm, the bert classification head, or full bert), or the combination of both, and (ii) what the neural feature and representation learning approaches actually learn and how that explains the superior classification. the contributions of our paper are as follows: ## related work combining learned and hand-crafted features. @xcite ; lim and tayyar madabushi, 2020) combine bert-based and manual features in order to improve accuracy. @xcite @xcite ) concatenate bert pooled output embeddings with handcrafted feature vectors for classification, often using an svm, where the handcrafted feature vector might be further encoded by a neural network or used as it is. our work differs in that we do not combine features from both models but swap them in order to decide whether it is the features, the classifiers or the combination that explains the performance difference between neural and feature engineering based models. additionally, our approach allows us to examine whether or not representation learning learns features similar to hand-crafted features. explainability for the feature-engineering approach to translationese classification. to date, explainability in translationese research has mainly focused on quantifying handcrafted feature importance. techniques include inspecting svm feature weights @xcite , correlation @xcite , information gain @xcite , chi-square @xcite , decision trees or random forests @xcite , ablating features and observing the change in accuracy @xcite , training separate classifiers on each individual feature (or feature set) and comparing accuracies @xcite . for n-grams, the difference in frequencies between the original and translationese classes @xcite , and the contribution to the symmetrized kullback-leibler divergence between the classes @xcite have been used. explainability for the neural approach to translationese classification. to date, explainability methods for neural networks have not been widely explored. @xcite quantify to which extent handcrafted features can explain the variance in the predictions of neural models, such as bert, lstms, and a simplified transformer, by training per-feature linear regression models to output the predicted probabilities of the neural models and computing the r 2 measure. they find that most of the top features are either pos-perplexity-based, or bag-of-pos features. however, their method treats the neural network as a black-box, whereas we use a method that accesses the internals of the model. ## translationese classification feeding pooled output of bert into the svm model [pretrained-bert-ft+svm], accuracy increases by 19% percentage points absolute over using handcrafted features [handcr. one can assume that bert pretrained to predict the handcrafted features and subsequently frozen [bert-r2c-full-frozen] has learnt to encode the handcrafted features during pretraining (section 3.4). nevertheless, its accuracy, albeit higher than a random guess, is lower by ∼ 13 percentage points than the svm classifier. we perform an additional experiment, in order to check whether the difference in accuracy is due to bert failing to sufficiently encode the handcrafted features during pretraining, or due to the svm classifier being superior to the linear classification head of the bert model. namely, we train the svm classifier on the pooled output of bert-r2cfull-frozen model [bert-r2c-full-frozen+svm] and on the pca-reduced dimensionality [bert-r2c-full-frozen+pca 108 +svm]. the accuracy is around 75% for both settings which is as high as using svm on handcrafted feature vectors. we conclude that bert encodes the handcrafted features sufficiently well, but the linear classifier performs worse than an svm in these conditions. further fine-tuning bert fully pretrained for handcrafted feature prediction [bert-r2c-full-ft] for translationese classification results in accuracy comparable to bert that was not pretrained on this task @xcite . this could suggest that our handcrafted feature set is either a subset of features learned by fromscratch-bert, or that the handcrafted features are discarded during fine-tuning. the model where only the first 6 layers were pretrained [bert-r2c-half-ft], achieves similar accuracy, likely due to the same reasons. by contrast, freezing the 6 handcrafted feature prediction pretrained layers [bert-r2c-half-frozen] largely reduces the accuracy with respect to bert-r2c-half-ft, because the model only has access to the 6th layer embeddings that supposedly encode the information about the handcrafted features, and does not have ability to extract its own features. the remaining (higher) 6 layers are responsible for the increment in accuracy with respect to bert-r2c-full-frozen. the results of bert-f2c models show that bert, when fed the handcrafted features in the form of embeddings, can reach at most the same accuracy as the hancdr.-features+svm approach, which suggests that the bert architecture has no advantage over the svm classifier in utilizing the handcrafted features for classification. this is again evidence that the features, and not the classifier, cause the better performance of the feature and representation learning method. foot_2 ## layer integrated gradients saliency we compare input attributions of the ground truth classification label amongst pretrained-bert-ft, fromscratch-bert and four different settings of the translationese classification models pretrained on the handcrafted feature prediction task: bert-r2c-full-ft, bert-r2c-full-frozen, bert-r2c-half-ft and bert-r2c-half-frozen. we use layer integrated gradients from the captum library @xcite , which computes the attribution for all the individual neurons in the embedding layer, and calculate the salience score for each token by averaging the attributions over the embedding dimension. ## summary and conclusions we address two open questions in classificationbased translationese research: (1) are the substantial performance differences between feature-and representation-learning and classical handcrafted feature based approaches is due to (i) the difference in the features, (ii) the classifiers, or (iii) both, and ( 2 ) what do feature-and representationlearning based approaches actually learn? we address (1) by exchanging features from both models, examining a broad variety of settings, to ensure that this is done in a fair an unbiased way. we show that svms perform as good as bert when fed with features learnt by bert. likewise, the bert classification head and the full bert architecture perform at the level of traditional svm-based classification with handcrafted features, when fed with handcrafted features only. this shows that it is the feature and representation learning and not the classifiers that are responsible for the translationese classification performance difference. to address question (2), we examine bert's input attributions using integrated gradients saliency for various settings and observe that attributions are indeed similar for the model trained from scratch (fromscratch-bert) on just the text data and the fine-tuned models that were pretrained on handcrafted feature prediction (bert-r2c-full-ft and bert-r2c-half-ft). this suggests that pretraining on the handcrafted features does not make a visible difference in attributions, and, together with the accuracy result that also does not change, suggests that no extra information is learnt during pretraining on handcrafted features. based on these findings, and the fact that some attributions appear to overlap for bert pretrained on handcrafted features and where the pretrained layers were subse-quently frozen (bert-r2c-full-frozen), and bert trained from scratch (fromscratch-bert), it is consistent to assume that handcrafted features are a (possibly partial) subset of the features automatically learnt by bert. finally, analysis of top activated tokens suggests that at least part of bert's strong translationese classification accuracy is based on topic differences between the classes as well as on some spurious correlations, rather than "proper" translationese phenomena. we are currently working on quantifying the 'clever hans' behaviour using named entity masking and cleaning/normalizing the data.