Measuring Spurious Correlation in Classification: “Clever Hans” in Translationese (2023)

Abstract: Recent work has shown evidence of “Clever Hans” behavior in high-performance neural translationese classifiers, where BERT-based classifiers capitalize on spurious correlations, in particular topic information, between data and target classification labels, rather than genuine translationese signals. Translationese signals are subtle (especially for professional translation) and compete with many other signals in the data such as genre, style, author, and, in particular, topic. This raises the general question of how much of the performance of a classifier is really due to spurious correlations in the data versus the signals actually targeted for by the classifier, especially for subtle target signals and in challenging (low resource) data settings. We focus on topic-based spurious correlation and approach the question from two directions: (i) where we have no knowledge about spurious topic information and its distribution in the data, (ii) where we have some indication about the nature of spurious topic correlations. For (i) we develop a measure from first principles capturing alignment of unsupervised topics with target classification labels as an indication of spurious topic information in the data. We show that our measure is the same as purity in clustering and propose a “topic floor” (as in a “noise floor”) for classification. For (ii) we investigate masking of known spurious topic carriers in classification. Both (i) and (ii) contribute to quantifying and (ii) to mitigating spurious correlations.

## introduction the term translationese refers to systematic linguistic differences between originally authored texts and translated texts in the same language @xcite . important aspects of translationese have been identified in the linguistic literature @xcite @xcite , including source lan-guage interference, over-adherence to target language, simplification, explicitation, and implicitation. translationese may manifest itself at lexical, syntactic, semantic, and discourse-related levels of linguistic description. while translationese signals are subtle (especially for professional human translation), corpus-based linguistic methods @xcite and machine learning based classification methods @xcite @xcite are able to reliably distinguish between original and translated texts in the same language, genre, and style. while basic research focuses on identifying and categorizing aspects of translationese, research has also shown that translationese clearly impacts practical crosslingual tasks that involve translated data @xcite @xcite . finally, translationese is sometimes regarded as (one of) the final frontier(s) of high-resource machine translation @xcite @xcite . in this paper, we focus on translationese classification (into original o and translated t data) using machine learning based approaches. early work on translationese classification focused on manually engineered and linguistically inspired sets of features (n-grams, pos, discrete lm-based features etc.), using supervised classification models such as decision-trees or support vector machines (svms) @xcite @xcite . more recently, research focused on feature-andrepresentation learning neural network methods for translationese classification @xcite . @xcite show that bert-based approaches outperform handcrafted feature and svm-based approaches by a large margin (15-20 accuracy points absolute). amponsah- @xcite show that this performance difference is due to learned features (rather than the classifiers). using integrated gradient (ig) based input attribution methods @xcite also show that bert @xcite sometimes exploits topic differences between o and t data as spurious correlations with the target classification labels (original o and translation t ) as short cuts, rather than "true" translationese signals: the t part of the (europarl-based) data, translations from spanish into german, happens to contain mentions of spanish locations while german originals o tend to mention german location names. spurious correlations in the data with target classification labels may cause "clever hans" behavior @xcite , where the classifier picks up accidental patterns in the data correlated with but otherwise unrelated to the classification target, in the case at hand, topic/content differences rather than proper linguistic indicators of translationese. to the best of our knowledge, to date, we do not know how to measure spurious correlations between topic signals in the data and target classification labels, such as translationese (o and t ). at the same time, this is an important question, as an answer would allow us to better understand to what extent we can trust a classifier to pick up on information truly relevant to the target classification labels, and to which extent a classifier is exploiting "clever hans", i.e. spurious correlations in the data with the target labels. this is especially pertinent with subtle classification targets and challenging low-resource data settings, as in translationese classification: translationese data sets tend to be small, and translationese signals are subtle while competing with many other signals in the data. we approach our research question of "measuring spurious topic correlation in the data with respect to target classification labels" from two opposing ends: (i) where we assume no prior knowledge about topics in the data and (ii) where we have some idea about spurious topic signals in the data. we refer to (i) as chasing unknown unknowns (section 4 below) 1 and (ii) as chasing known unknowns (section 5 below). for (i) we use unsupervised topic modeling (lda and bertopic) and 1 readers may relate this to a 2002 hearing with the then us secretary of defence donald rumsfeld. in one scenario we do not know the topics and their impact, the unknown unknowns; in the other we have some indication about a spurious topic but again do not know its impact, the known unknowns. puurtinen (2003); ilisei et al. (2010); volansky et al. (2013); rabinovich and wintner (2015); rubino et al. (2016); pylypenko et al. (2021) train classifiers to distinguish between originally authored and translated data. many of them explore handcrafted and linguistically inspired feature sets, manual feature engineering, and a variety of classifiers including decision trees and support vector machines (svms) and use feature ranking or attribution methods to reason back to particular dimensions of translationese and their importance in the data and classification results. feature engineering based translationese classification used svm feature weights @xcite , decision trees or random forests @xcite , training separate classifiers for each individual feature (or feature sets) and comparing accuracies @xcite , to explain results. more recent research uses feature and representation learning approaches (sometimes augmented with hand-crafted features) based on neural networks @xcite . @xcite shows that feature learning based approaches (e.g. a pretrained bert based classifier) outperform hand-crafted and feature engineering based approaches (svm) by as much as 15 to 20 percentage points absolute in classification accuracy. amponsah- @xcite show that the difference in classification accuracy is due to feature learning rather than the classifiers, and, using integrated gradients (igs) @xcite , provide evidence that the feature learning methods exploit some spurious correlations with the classification labels in the data, that are clearly not translationese, but topic related cues: the data are german originals o and translations t (into german) from spanish, and spanish place names are highly ig-ranked, given the trained classifiers. dutta chowdhury et al. (2022) use divergence from isomorphism based graph distance measures to show that translationese is visible even in o and t word embedding spaces. while pos features have been used in feature-engineering-based translationese classification, some experiments in @xcite use pos (instead of the surface words) to mitigate possible topic influences on the graph divergence results. this is an approach we develop further (e.g. in terms of partial masking using nes) in our work below. ## data our experiments use the monolingual german dataset from the multilingual parallel direct europarl (mpde) corpus (amponsah-kaakyire et al., 2021) consisting of 42k paragraphs, with half of the paragraphs german (de) originals (below we call this o) and the other half translations (below we call this t ) from spanish (es) to german. the average length (in terms of tokens) per training example is 80. like all of mpde, the de-es subset contains only data from before 2004, since for post-2004 data, it may not be known whether or not the source language sl is already the result of a translation @xcite . while this limits the amount of data, it ensures that the o and t data are clearly identifiable and "pure" o or t . both o and t are german, but t is german translated from spanish, and both coming from mpde ensure that they are the same europarl genre and style. ## chasing unknown unknowns in this section, we assume that we have no prior information about topics and their distribution in the data. because of this, we use unsupervised topic modeling. we develop a measure that checks whether, and if so to what extent, the topics established in this fashion align with the target classes o and t in our data. the measure quantifies to which extent topic is a giveaway for translationese. ## chasing known unknowns in this section, we assume that we have some knowledge about spurious topics in our data and that we want to both quantify and mitigate this spurious topic information in translationese classification. the difference in classification accuracy between a classifier that has access and one that does not have access to spurious topic information quantifies the impact of the spurious topic information in question. using ig, amponsah-kaakyire et al. @xcite show that high-performance bert-based classifiers use location names (spanish names in the t , and german names in the o data) in the classification, clearly not a proper translationese but a spurious topic signal. similarly, named entities (nes) are often highly ranked in lda topics. therefore, the most straightforward approach towards mitigating specific spurious topic information in translationese classification is identifying nes and masking them in the data. ## conclusion we present a measure that, given a data set and target classification labels, quantifies the possible impact of unknown spurious topic information on classification. the measure is based on aligning unsupervised topics with target labels and is equivalent to purity in clustering. we propose the concept of a "topic floor" (akin to "noise floor") as an upper bound of the impact of spurious topic information on classification in classification. we use masking to quantify and mitigate known spurious topic information. we present empirical results for topic floor and masking to quantify "clever hans" in the translationese data of @xcite . we use ig attribution to show that in masked settings where known spurious correlations are mitigated, bert learns features closer to proper translationese.