The Effect of Round-Trip Translation on Fairness in Sentiment Analysis (2021)

Abstract: Sentiment analysis systems have been shown to exhibit sensitivity to protected attributes. Round-trip translation, on the other hand, has been shown to normalize text. We explore the impact of round-trip translation on the demographic parity of sentiment classifiers and show how round-trip translation consistently improves classification fairness at test time (reducing up to 47% of between-group gaps). We also explore the idea of retraining sentiment classifiers on round-trip-translated data.

## introduction it is both unethical and potentially illegal for document classification algorithms to perform significantly better for some groups in society than for others @xcite . many document classification technologies have, however, been shown to be sensitive to protected attributes such as gender and age @xcite @xcite . this also holds for sentiment analysis @xcite @xcite @xcite . at the same time it is known that round-trip machine translation @xcite can be used to normalize text @xcite @xcite . this could potentially remove group specific deviations from normal language. however, @xcite found that machine translation is also prone to potentially introducing gender bias. combined, this leaves it an open question whether round-trip translation can be used to reduce the sensitivity of document classifiers to protected attributes of the authors. in this paper, we evaluate the effect of roundtrip translation on fairness using a representative text sally is a whiz at math. sally es una experta en matemáticas. sally is a math expert. contributions we evaluate round-trip translation as a technique for mitigating sensitivity to protected attributes across two attributes and three document classification architectures. we find that round-trip translation at test time consistently reduces the fairness gap (with up to 47%), but that for our best models (svms stacked on bert representations), the effect disappears when both training and test data are translated into a foreign language and back. ## round-trip translation round-trip machine translation @xcite is the process of machine translating a document to another language and then back to the original language. group characteristics # of reviews 1 male, @xmath0, @xmath1, 42 < @xmath2, 42 < @xmath3, 55 < age 7535 6 female, 55 < age 4489 fairness metric the fairness literature is rich with definitions of fairness @xcite , most of which are interpretations of the rawlsian notion of fairness as equal opportunity @xcite . in this work, we adopt the following definition of fairness: if the maximal difference in empirical risk between any two groups in g is , we say θ is -fair. below we use the f1-score for the negative class as our ℓ(•). 3 note how fairness as equal risk is a generalization of approximately equal conditional risk @xcite . ## results in group kld reviews kld round-trip 1 0.027 0.021 2 0.028 0.023 3 0.011 0.009 4 0.023 0.021 5 0.028 0.022 6 0.027 0.020 test time normalization with round trip translation (b) has an overall positive effect on cross-group generalization, reducing the fairness gap with up to ∼ 27%. the third scenario (c) -i.e., the idea of using round-trip translation for normalizing both the training and the test data -yields mixed results, with fairness gap increases up to ∼ 39% (for bert models) and decreases up to ∼ 47% (for laser models). machine translation introduces its own biases, and some representations may be more sensitive to such biases. note also that the process of round-trip translating the data consistently reduces the overall accuracy of our document classifiers, suggesting a trade-off between fairness and accuracy. ## discussion round-trip translation is a simple technique for testtime input normalization, and we have shown that it can significantly reduce sensitivity to protected attributes at a low performance cost. one advantage of round-trip translation is that it does not require annotation of protected attributes. such datasets are generally only available for english at this point. high-quality machine translation, in contrast, is available for hundreds of languages. in this paper, we experiment with using round-trip translation to reduce group disparity of sentiment classifiers for danish. it is important to note, however, that the overall performance drop that results from round-trip trans-lation, while relatively small, means that the absolute performance on minority group drops. in other words, all users experience worse performance with the more fair sentiment classifiers. this, of course, is unfortunate and potentially introduces an ethical dilemma. in fact, it is only with our laser models that minority group performance improves and fairness is reduced. round-trip translation is orthogonal to other approaches to improving fairness, such as distributionally robust optimization @xcite , invariant risk minimization @xcite , and adversarial training @xcite . round-trip translation can thus easily be combined with any of these approaches, but note that these approaches require annotation of protected attributes. round-trip translation does not and can thus be considered an unsupervised approach to reducing group disparities. the fairness gap was most consistently reduced by test-time round-trip translation, but doing roundtrip translation may be more effective for other machine translation systems. in our experiments, google translate introduced new biases when relying on bert representations, but the approach was successful for document classifiers based on laser representations: here, we saw both reductions of the fairness gap and improvements for minority groups. for 2/4 classifiers, we even saw improvements for the majority groups. ## conclusion sentiment classifiers perform better on reviews written by some demographic groups rather than others, with groups defined by protected attributes such as gender and age. we present a first experiment with round-trip translation as a means of reducing this fairness gap in sentiment classification. specifically, we show that translating danish product reviews into english and back, reduces group disparity across three different classification architectures. while the performance cost may in our case be prohibitive for some architectures, in practice, we believe that round-trip translation can be an important technique for improving the fairness of document classifiers in the future, which is easier to scale to new tasks and languages than approaches that require annotation of protected attributes.