Towards Debiasing Translation Artifacts (2022)

Abstract: Cross-lingual natural language processing relies on translation, either by humans or machines, at different levels, from translating training data to translating test sets. However, compared to original texts in the same language, translations possess distinct qualities referred to as translationese. Previous research has shown that these translation artifacts influence the performance of a variety of cross-lingual tasks. In this work, we propose a novel approach to reducing translationese by extending an established bias-removal technique. We use the Iterative Null-space Projection (INLP) algorithm, and show by measuring classification accuracy before and after debiasing, that translationese is reduced at both sentence and word level. We evaluate the utility of debiasing translationese on a natural language inference (NLI) task, and show that by reducing this bias, NLI accuracy improves. To the best of our knowledge, this is the first study to debias translationese as represented in latent embedding space.

## introduction while a number of methods to remove or attenuate human-like biases (e.g., gender, race, etc.) in both static and contextualised word embeddings have recently been proposed @xcite @xcite @xcite , attenuating and eliminating a more implicit signal like translationese in embeddings has yet to be studied. translationese signals are complex and multi-faceted and, unlike e.g. gender and profanity, can in general not be captured in terms of simple lists of contrastive word pairs (woman-man, she-he, etc.), but rather manifest as a complex mix of morphological, lexical, syntactic and semantic phenomena. our study is a first attempt to directly debias translationese encoded as latent representations, based on the recently proposed iterative nullspace projection (inlp) algorithm @xcite . the main contributions of our work are as follows. (i) we propose to reduce the bias induced by translation by extending the inlp approach to translationese debiasing at both the word and sentence level, with findings demonstrating that our debiasing approaches can effectively attenuate translationese biases in both static word embedding spaces as well as sentence representations based on contextualised embeddings. (ii) we use the inlp method with a number of neural sentence-level translationese classification architectures (fasttext, mbert, xlm), and propose two alternative methods for detecting explicit translationese bias in word embeddings, and find that after debiasing, the models' performance on classifying translationese degrades to that of a random classifier. (iii) finally, by integrating the proposed debiasing method within the nli task, we show the effect of translation artifacts and their removal on the task. ## debiasing strategies much previous research @xcite @xcite focuses on eliminating bias in word embeddings. however, existing models use lists of contrastive word pairs (e.g., woman-man, shehe) to detect, capture and mitigate specific biases (e.g., gender, profanity, etc.). while translationese cannot, in general, be captured in simple lists of contrastive word pairs, we do have labeled data at sentence level: translated (translationese) and original sentences. we use this data and the inlp algorithm to directly mitigate translationese at sentence level (section 3). additionally, we explore inlp at word level to debias translationese in section 4. only a few earlier studies @xcite deal with translationese at the level of word embedding spaces leveraging distances between graph-based representations of original and translationese data. for debiasing word embedding spaces, we adapt an idea from @xcite to extract lists of pairs of identical words and examine how their use differs in translationese and original data (rather than contrasting word pairs). if a word is used very differently in translated and original data, this is reflected in differences in original and translated word embedding spaces, and is evidence of translationese in the embeddings. alternatively, we propose a simpler approach that builds on a joint embedding space where words are tagged according to their origin (translationese or original) and without any need for a word list. the iterative nullspace projection algorithm @xcite focuses on removing linearly decipherable features from vector representations originally for gender bias mitigation. given a set of labeled data with data points @xmath0, ..., x n and task labels @xmath1, ..., y n , we use a standard classification setup with a neural network and a simple classifier τ on top. an encoder h encodes x k into a representation vector h(x k ) and τ predicts y k based on h(x k ), i.e., y @xmath2. let t be the trait to be mitigated, also known as the protected attribute. the goal of the inlp method is to neutralise the ability of the classifier τ to linearly predict t from h. τ is parameterised by a matrix w and trained to predict t from h. using w , one can collapse the data onto its nullspace n (w ) with a projection matrix p n (w ) . this guarantees w p n (w ) h(t ) = 0, i.e., the information used to classify t is linearly removed. by repeating this process i times until no classifier achieves above-majority accuracy, inlp can neutralise all features that w i uses for predicting t from h: details of the implementation are the same as in @xcite . foot_0 ## translationese in sentence embeddings in our work, we are interested in adapting inlp to study the impact of removing the translationese attributes t from semantic representations, via a binary classification task. specifically, the binary classifier learns to distinguish between original and translationese sentences. therefore, in our setup, labels y correspond to original and translationese and act as protected attributes. data. we use the europarl corpus annotated with translationese information from amponsah-kaakyire et al. @xcite . we focus on three languages: english (en), german (de) and spanish (es). the corpus provides originals in the three languages (l1) and translations into these three languages that come from original texts in the other two (l2). we use the notation l1-l2 to refer to the different sets in de-en, de-es, en-de, en-es, es-de and es-en. we use 70% of the sentences for training, 15% for development and 15% for testing. classifier. we use a logistic classifier on top of sentence embeddings h obtained with 4 models without any additional fine-tuning to the translationese classification task. (i) fasttext @xcite : we compute an average of all token vectors in the sentence. (ii) mbert cls @xcite : we use the [cls] token in mbert as sentence representation. (iii) mbert pool @xcite : we use mean pooling of mbert's contextualised word embeddings. (iv) xlm cls @xcite : we use the [cls] token from xlm-roberta. results. the first four columns in ## translationese in word embeddings unlike sentence-level debiasing, word-level debiasing needs a seed translationese direction to obtain a debiased space. this is challenging for translationese as unlike, e.g. gender and profanity, translationese cannot in general be captured in terms of simple contrastive word pairs. in what follows, we introduce two approaches for debiasing translationese at the level of word embeddings. stepwise aligned space projection. in order to estimate the seed translationese direction, we derive a list of words (g) used differently in translationese t and original o data using the usage change concept from @xcite . the same word used in different data sets (original and translated) is likely to have different neighbours in the two embedding spaces. we only use words from the intersection of both vocabularies o and t . we compute the score for context change across the embeddings o and t of the two data sets by considering the size of the intersection of two sets where each word in a corpus is represented as its top-k nearest neighbors (nn) in its embedding space: in our experiments we only consider words attested at least 200 times in the data. appendix a.3 shows the top 50 elements for all the word lists. in our experiments, we use the translated and original parts of the data described in section 3 to estimate the word embeddings o and t and use @xmath3. following @xcite , a large value of k results in large neighbors sets for each word in the two corpora, resulting in a more stable translationese wordlist g. next, we create a joint word embedding space j from the concatenation of the translated and original data, t and o. since this joint space j includes both original and translationese signals, we then align the previously unrelated o and t spaces to this embedding space j , using vecmap @xcite , producing aligned spaces õ and t , and resulting in an extended single embedding space where t and o are aligned to j . next, we compute the translationese direction v of the same word w in the two embeddings spaces, õ and t , using, finally, we compute the similarity of words in j along the directions v and -v, to divide them into two subspaces , translationese and nontranslationese, respectively. using equation 3, we initialise the inlp algorithm in two ways: (i) inlp.single: with a direction vector created from the difference between two aligned spaces for the highest ranked word in g, and (ii) inlp.avg: by averaging the differences of all words in g. direct joint space projection. in this approach, we directly build the embeddings of a specific word w from o and t into the same space j by annotating w as either w o and w t on surface in the o and t data. in this way we can easily track and distinguish the two embeddings of the same word w coming from o and t data in the same embedding space j resulting from the simple concatenation of two datasets. this eliminates the need to compute the translationese direction vector v to group the subspaces and the complexity of maintaining and aligning o, t , j , õ and t spaces. figure 1 in appendix a.2.2 shows the t-sne projection (van der @xcite of the tagged tokens before and after debiasing. results. we compare the performance before and after debiasing using our two word-level debiasing methods in translationese word lists. the content of our extracted translationese word lists depends on the language (see appendix a.3). for es, punctuation is clearly used differently in originals and translations into es. for de, pronominal adverbs play an important role, especially when translations come from es. for en, there is no clear trend but, interestingly, only one word in the top-50 list (indeed) overlaps with the words with a highest difference in frequency of usage in the original and translationese corpus as analysed for europarl in @xcite . the number of times and the context where a word appears may reflect two different aspects of translationese. ## application to nli in order to investigate the impact of removing translationese artifacts from the translated data we analyse its impact on the nli task, where machine translation is used. nli predicts the relationship between two sentences, premise and hypothesis, and classifies it into one of the three categories -entailment, contradiction, or neutral. recently, @xcite showed that, in the existing nli datasets, there exists a significant lexical overlap between the premise and the hypothesis, which is utilised by neural nli models to make predictions with high accuracies. however, when the premise and hypothesis are paraphrased independently using translation and back-translation, lexical overlap is reduced, negatively impacting the performance of the models. below we test whether and if so, to which extent our inlp-based translationese debiasing approach can avoid the observed performance loss in the back-translated nli task. we generate back-translated (bt) nli dataset by independently translating premise and hypothesis from the original snli data. then, we train two nli models, one on original and one on bt data. to reduce the translationese artifacts resulting from back-translation, we apply the sentence and word embedding debiasing strategies as described in sections 3 and 4 over the embeddings generated from bt data, and use the resulting debiased-bt embeddings to train a third debiased nli model in finally, we consider two scenarios to evaluate translationese debiasing -(i) symmetric [sym]: where original is tested with original test set, and bt and debiased nli models are tested with bt test data and (ii) asymmetric [asym]: where all these models are tested with original test data. we use (i) to see whether debiasing the model trained on translated train-test can bring its performance closer to that of model trained on original train-test data. in (ii), we examine the asymmetry between original test data and bt-nli training data, and whether our translationese debiasing of bt training data can offset this asymmetry and improve nli performance. data. the large-scale snli dataset (stanford natural language inference) @xcite contains 570k sentence pairs in the training set manually classified as entailment, contradiction, or neutral. we use a subset of 10% of the training data for our experiments. the development and test data are used as in the original snli splits (each containing 10k examples). we generate a back-translated variant of the training and test data using german (de) as the pivot language. for translation, we use the pre-trained models of facebook-fair's wmt-19 news translation task submission @xcite . models. we train three different nli models, each for original, back-translated and debiased versions of back-translated embeddings. for the wordlevel setup, we use a single hidden bilstm layer followed by a standard feedforward output layer on top of frozen fasttext word embeddings for the 3class nli classification. the computation of word embeddings and debiasing follows the setup described in section 4. for sentence-level debiasing, we use the bert pool method explained in section 3 with a linear svc on top to predict the labels. results. ## conclusion in this work, we remove translationese artifacts by extending the debiasing inlp approach at both word and sentence level. to the best of our knowl-edge, this is the first paper that attempts at debiasing sentence and word embeddings for translationese. we introduce two techniques for debiasing translationese at the word level: one (stepwise aligned subspaces) is akin to the subspace construction approach of gender-debiasing proposed by @xcite and @xcite , the second (direct joint subspace) is a simplified approach that operates directly on the joint space without the use of a separate translationese word list and multiple independently computed and subsequently aligned subspaces. our word-based debiasing study provides a systematic view of translationese biases contained in static embeddings. we also explore translationese debiasing at sentence level embeddings computed from contextualised word embeddings. as expected, the inlp-based linear translationese debiasing results on static word embeddings are as "perfect" as our sentence level results, reducing the performance of a linear translationese classifier on the debiased data to chance, demonstrating that our debiasing strategies effectively attenuate translationese signals in both these spaces. further, we evaluate the effects of debiasing translation artifacts on a standard nli task in two settings. even though we achieve "perfect" performance for the translationese classification-based debiasing task with inlp, this translates into just modest improvements resulting from inlp-based debiasing translationese in neural machine translation in an nli task, with slightly better results for sentence than word debiasing. this demonstrates that our debiasing approach is effective in reducing translation artifacts but that there is more to translationese than is visibe to a linear classifier. finally, we acknowledge that while this study is the first to debias translationese encoded as latent representation in (word and sentence) embedding space, the effect of this on the actual surface form of the generated output is not investigated. we hope to account for this in future work.