Preventing Author Profiling through Zero-Shot Multilingual Back-Translation (2021)

Abstract: Documents as short as a single sentence may inadvertently reveal sensitive information about their authors, including e.g. their gender or ethnicity. Style transfer is an effective way of transforming texts in order to remove any information that enables author profiling. However, for a number of current state-of-the-art approaches the improved privacy is accompanied by an undesirable drop in the down-stream utility of the transformed data. In this paper, we propose a simple, zero-shot way to effectively lower the risk of author profiling through multilingual back-translation using off-the-shelf translation models. We compare our models with five representative text style transfer models on three datasets across different domains. Results from both an automatic and a human evaluation show that our approach achieves the best overall performance while requiring no training data. We are able to lower the adversarial prediction of gender and race by up to 22% while retaining 95% of the original utility on downstream tasks.

## introduction data collections of natural language utterances bear the risk of disclosing sensitive information about the recorded participants, including their gender, race, or political preferences. unlike explicit mentions of private information, like a user's name or location @xcite , such user traits are often encoded rather subtly in a user's speaking or writing style. nevertheless, they can be predicted with high accuracy by deep learning-based classifiers even when they are not obvious to humans @xcite , enabling third-parties with access to the data sets to profile users without their knowledge. a common method to alleviate this problem is the application of an intermediate transformation step to remove sensitive information via text style transfer. while a number of different style transfer techniques exist @xcite @xcite , they require large amounts of text data labeled with user trait information to perform well. additional annotations need to be provided for every new user trait that the model is expected to handle, multiplying the associated costs and effort. furthermore, the impact that such transformations can have on the utility of the resulting data is often overlooked. conversely, we argue that the privacy-utility dichotomy should be at the heart of all research on this topic because it is fairly easy to consider one of the two but difficult to improve both at the same time. in this paper, we explore a simple yet effective zero-shot text transformation method based on multilingual back-translation. back-translation (bt) is an alternative approach without the prerequisites of labeled training data. sensitive user traits can be significantly obfuscated when translated to another language and back @xcite since many concepts cannot easily be mapped across languages. for example, in languages such as japanese and korean the speaker's gender can be inferred from the choice of certain pronouns. when back-translating them via an intermediate language that does not make such differences, such as english, these gender indicators will be largely obfuscated. results from extensive experiments show that our simple zero-shot text transformer has comparable or even better performance than popular style transfer methods, considering both the privacy and utility of the transformed texts. in summary, we make the following contributions: our approach is zero-shot without the need for additional data to train style transfer models. ## related work attribute information such as gender, age, or race are being captured in the deep learning models. traditional approaches prevented this information leakage via lexical substitution of sensitive words @xcite . in recent years, many text style transfer techniques have been proposed to control certain attributes of generated text (e.g., formality or politeness) while preserving the content. a common paradigm is to disentangle the content and style in the latent space @xcite @xcite ). another stream of work treats text style transfer as an analogy of unsupervised machine translation @xcite @xcite to rephrase a sentence while reducing its stylistic properties @xcite . beyond the end-to-end training methods, the prototype-based text editing approach also attracts lot of attention @xcite @xcite , in which attribute markers of input sentences are deleted and then replaced by target attribute markers. these techniques have been well studied in the text style transfer community, but have never been evaluated for both privacy and utility preservation in downstream tasks. shetty et al. (2018) and xu et al. (2019) make use of adversarial training and evaluate on authorship obfuscation. however, they did not include most recent style transfer methods and predictors based on pretrained language models. 3 multilingual back-translation problem scenario in understanding human behaviors and intents, many machine learning applications need to infer important information from pivot language translated back-translated de danke papi thank you daddy fr merci papi thank you papi zh 谢谢你爸爸 thank you dad in this paper, we explore a simple, zero-shot text transformation method through multilingual back-translation. our assumption is that, as also supported in previous research @xcite ), text styles can be significantly obfuscated when being translated to another language (pivot language) then translated back. one example is shown in ## conclusion in this paper, we propose a zero-shot way to effectively lower the risk of author profiling through multilingual bt using off-the-shelf translation models. we compare our approach with different style transfer models, achieving the best overall performance using an automatic and a human evaluation while requiring no additional training data. in the future, we will (1) analyze how the language choice and translation quality affects the privacy preservation in bt, (2) investigate more on other metrics that can be used to aggregate the the four evaluation metrics corresponding to transfer attribute strength, content preservation, fluency, and utility, and (3) extend the zero-shot bt method with some supervision to improve privacy. we highlight a few limitations of our work. first, back-translation transformation remove content style but does not necessarily replace attribute markers like style transfer models, for example, given a text "me and my husband ...", style trans-fer models are more likely to change "husband" to "wife" but back-translation will not. second, our back-translation technique also inherit some of the problems of machine translation generated texts like hallucination @xcite . we provide examples highlighting these issues in appendix c.