Adversarial Stylometry in the Wild: T ransferable Lexical Substitution Attacks on Author Profiling (2021)

Abstract: Written language contains stylistic cues that can be exploited to automatically infer a variety of potentially sensitive author information. Adversarial stylometry intends to attack such models by rewriting an author’s text. Our research proposes several components to facilitate deployment of these adversarial attacks in the wild, where neither data nor target models are accessible. We introduce a transformer-based extension of a lexical replacement attack, and show it achieves high transferability when trained on a weakly labeled corpus—decreasing target model performance below chance. While not completely inconspicuous, our more successful attacks also prove notably less detectable by humans. Our framework therefore provides a promising direction for future privacy-preserving adversarial attacks.

## introduction the widespread use of machine learning on consumer devices and its application to their data has sparked investigation of security and privacy researchers alike in correctly handling sensitive information @xcite . natural language processing (nlp) is no exception @xcite ; written text can contain a plethora of author information-either consciously shared or inferable through stylometric analysis @xcite . this characteristic is fundamental to author profiling @xcite , and while the field's main interest pertains to the study of sociolinguistic and stylometric features that underpin our language use @xcite , herein simultaneously lie its dual-use problems. author profiling can, often with high accuracy, infer an extensive set of (sensitive) personal information, such as age, gender, education, socio-economic status, and mental health issues @xcite @xcite @xcite . it therefore potentially exposes anyone sharing written online content to unauthorized information collection through their writing style. this can prove particularly harmful to individuals in a vulnerable position regarding e.g., race, political affiliation, or mental health. privacy-preserving defenses against such inferences can be found in the field of adversarial foot_0 stylometry. our research foot_1 concerns the obfuscation subtask, where the aim is to rewrite an input text such that the style changes, and stylometric predictions fail. it is part of a growing body of research into adversarial attacks on nlp @xcite , which various modern models have proven vulnerable to; e.g., in neural machine translation @xcite , summarization @xcite , and text classification @xcite . adversarial attacks on nlp are predominantly aimed at demonstrating vulnerabilities in existing algorithms or models, such that they might be fixed, or explicitly improved through adversarial training. consequently, most related work focuses on white or black-box settings, where all or part of the target model is accessible (e.g., its predictions, data, parameters, gradients, or probability distribution) to fit an attack. the current research, however, does not intend to improve the targeted models; rather, we want to provide the attacks as tools to protect online privacy. this introduces several constraints over other nlp-based adversarial attacks, as it calls for a realistic, in-the-wild scenario of application. firstly, authors seeking to protect themselves from stylometric analysis cannot be assumed to be knowledgeable about the target architecture, nor to have access to suitable training data (as the target could have been trained on any domain). hence, we cannot optimally tailor attacks to the target, and need an accessible method of mimicking it to evaluate the obfuscation success. to facilitate this, we use a so-called substitute model, which for our purposes is an author profiling classifier trained in isolation (with its own data and architecture) that informs our attacks. attacks fitted on substitute models have been shown to transfer their success when targeting models with different architectures, or trained on other data, in a variety of machine learning tasks @xcite . the effectiveness of an attack fitted on a substitute model when targeting a 'real' model is then referred to as transferability, which we will measure for the obfuscation methods proposed in the current research. secondly, for an obfuscation attack to work in practice (e.g., given a limited post history), it should suggest relevant changes -to-the author's writing on a domain of their choice. this implies the substitute models should be fitted locally, and therefore need to meet two criteria: reliable access to labeled data, and being relatively fast and easy to train. to meet the first criterion, the current research focuses on gender prediction, as: i) twitter corpora annotated with this variable are by far the largest (and most common), ii) author profiling methods typically use similar architectures for different attributes; therefore, the generalization of attacks to other author attributes can be assumed to a large extent, and, most importantly, iii) @xcite and @xcite have shown that through distant labeling, a representative corpus for this task can be collected in under a day. this allows us to measure transferability of attacks fitted using realistically collected distant corpora to models using high-quality hand labeled corpora. as for the attacks, we focus on lexical substitution of content words strongly related to a given label, as those have been shown to explain a significant portion of the accuracy of stylometric models (see e.g., @xcite @xcite . to that effect, we extend the substitution attack of @xcite and apply it to author attribute obfuscation. specifically, we explore the potential of training a simple (as to meet the speed criterion), non-neural substitute model f to indicate relevant words to perturb, where retaining the original meaning is prioritized. two transformer-based models are introduced to the framework to propose and rank lexical substitutions towards a change in the predictions of f . we evaluate if the attacks on f transfer across corpora, architectures, and a separately trained target model f (see figure 1 ). finally, we measure the quality of changes using automatic evaluation metrics, and conduct an human evaluation that focuses on detection accuracy of the attacks. ## related work stylometry, the study of (predominantly) writing style, dates back several decades @xcite , and has seen increased accessibility through the introduction of statistical models (see surveys by @xcite and machine learning (e.g., @xcite . computational stylometry distinguishes several subtasks such as determining @xcite and verifying author identity @xcite , and author profiling @xcite ; e.g., predicting demographic attributes. adversarial stylometry (as conceptualized by @xcite intends to subvert these inferences by changing an author's text through imitation, or, as pertains to our research, the obfuscation of writing style @xcite @xcite . these changes, or perturbations, can be produced in several ways, and the task is therefore of-ten conflated with paraphrasing @xcite , style transfer (kabbara and cheung, 2016), and generating adversarial samples or triggers @xcite . regardless of the employed method, the main challenge of obfuscation lies in retaining the original meaning of an input text; its written language medium limits any perturbations to discrete outputs, and unnatural discrepancies are significantly better discernible by humans than, say, a few pixel changes in an image. an additional, persistent limitation is the absence of evaluation metrics that guarantee complete preservation of the original meaning of the input whilst changes remain unnoticed @xcite . this not only inhibits automatic evaluation of obfuscation, but all natural language generation research @xcite -placing an emphasis on human evaluation (van der @xcite . it is perhaps for this reason that most obfuscation work uses heuristically-driven, controlled changes such as splitting or merging words or sentences, removing stop words, changing spelling, punctuation, or casing (see e.g., @xcite . these specific attacks are typically easier to mitigate through preprocessing @xcite . obfuscation through lexical substitution @xcite @xcite provides a middle ground of control, semantic preservation and attack effectiveness; however, they might prove less effective against models relying on deeper stylistic features (e.g. word order, part-of-speech (pos) tags, or reading complexity scores). end-to-end systems have been employed for similar purposes @xcite , or to rewrite entire phrases @xcite using (adversarially-driven) autoencoders. such attacks seem less common, and provide less control over the perturbations and semantic consistency. our work does not assume the attacks to run end-to-end, but with a hypothetical human in the loop. we further opt for techniques that are more likely to find strong semantic mirrors to the original text while making minimal changes. a substitute model (the algorithm, hyper-parameters, and output of which an author can manipulate as desired) is employed to indicate candidate replacement words, and our attacks suggest and rank those against this substitute. moreover, prior work typically attacks adversaries trained on the same data, whereas we add a transferability measure. lastly, while au-thor identification has been investigated in the wild @xcite , our work is, to our knowledge, the first to make a conscious effort towards realistic applicability of obfuscation techniques. ## method our attack framework extends textfooler @xcite in several ways. first, a substitute gender classifier is trained, from which the logit output given a document is used to rank words by their prediction importance through an omission score (section 3.1). for the top most important words, substitute candidates are proposed, for which we add two additional techniques (section 3.2). these candidates can be checked and filtered on consistency with the original words (by their pos tags, for example), accepted as-is, or re-ranked (section 3.3). for the latter, we add a scoring method. finally, the remaining candidates are used for iterative substitution until tf's stopping criterion is met (i.e., the prediction changes, or candidates run out). ## discussion and future work we have demonstrated the performance of author attribute obfuscation under a realistic setting. using a simple logistic regression model for candidate suggestion, trained on a weakly labeled corpus collected in a day, the attacks successfully transferred to different data and architectures. this is a promising result for future adversarial work on this task, and its practical implementation. it remains challenging to automatically evaluate how invasive the required number of changes are for successful obfuscation-particularly to an author's message consistency as a whole. however, in practice such considerations could be left up to the author. in this human-in-the-loop scenario, a more extensive set of candidates could be suggested, and their effect on the substitute model shown interactively. this way, the attacks can be manually tuned to find a balance of effectiveness, inconspicuousness, and to guarantee semantic consistency. it would also show the author how their writing style affects potential future inferences. regarding the performance of the attacks: we demonstrated the general effectiveness of contextual language models in retrieving candidate suggestions. however, the quality of those candidates might be improved with more extensive rule-based checks; e.g., through deeper analyses using parsing. nevertheless, such venues leave us with a core limitation of rewriting language, and therefore more broadly nlp: while the masked attacks seemed more successful in our experiments, after manual inspection of the perturbations dropout was found to often be semantically closer (see also as such, we would argue that future work should focus on making as few perturbations as possible, retaining only the minimum amount of required obfuscation success. given this, the other constraints become less relevant; one could generate short sentences (e.g., a single tweet) that might be semantically or contextually incorrect, but if it is a message in a long post history, it will hardly be detectable or intrusive. this would require certain triggers (as demonstrated by @xcite for example), and ascertaining how well they transfer. ## conclusion in our work, we argued realistic adversarial stylometry should be tested on transferability in settings where there is no access to the target model's data or architecture. we extended previous adversarial text classification work with two transformer-based models, and studied their obfuscation success in such a setting. we showed them to reliably drop target model performance below chance, though human detectability of the attacks remained above chance. future work could focus on further minimizing this detection under our realistic constraints. david i holmes. 1998. the evolution of stylometry in humanities scholarship. literary and linguistic computing, 13(3):111-117. matthew honnibal and ines montani. 2017. spacy 2: natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. to appear, 7. xiaolei huang, xing linzi, franck dernoncourt, and michael j. paul. 2020. multilingual twitter corpus and baselines for evaluating demographic bias in hate speech recognition. in proceedings of the twelveth international conference on language resources and evaluation (lrec 2020), marseille, france. european language resources association (elra). di jin, zhijing jin, joey tianyi zhou, and peter szolovits. 2020. is bert really robust? a strong baseline for natural language attack on text classification and entailment. in the thirty-fourth aaai conference on artificial intelligence, aaai 2020, the thirty-second innovative applications of artificial intelligence conference, iaai 2020, the tenth aaai symposium on educational advances in artificial intelligence, eaai 2020, new york, ny, usa, february 7-12, 2020, pages 8018-8025. aaai press. youngjun joo, inchon hwang, l cappellato, n ferro, d losada, and h müller. 2019. author profiling on social media: an ensemble learning model using various features. notebook for pan at clef. patrick juola and darren vescovi. 2011. analyzing stylometric approaches to author obfuscation. in ifip international conference on digital forensics, pages 115-125. springer. jad kabbara and jackie chi kit cheung. 2016. stylistic transfer in natural language generation systems using recurrent neural networks. in proceedings of the workshop on uphill battles in language processing: scaling early achievements to robust methods, pages 43-47. gary kacmarcik and michael gamon. 2006. obfuscating document stylometry to preserve author anonymity. in proceedings of the coling/acl on main conference poster sessions, pages 444-451. association for computational linguistics. akos kádár, grzegorz chrupała, and afra alishahi. 2017. representation of linguistic form and function in recurrent neural networks. computational linguistics, 43(4):761-780. georgi karadzhov, tsvetomila mihaylova, yasen kiprov, georgi georgiev, ivan koychev, and preslav nakov. 2017. the case for being average: a mediocrity approach to style masking and author obfuscation. in international conference of the cross-language evaluation forum for european languages, pages 173-185. springer. moshe koppel, shlomo argamon, and anat rachel shimoni. 2002. automatically categorizing written texts by author gender. literary and linguistic computing, 17(4):401-412. moshe koppel and jonathan schler. 2004. authorship verification as a one-class classification problem. in proceedings of the twenty-first international conference on machine learning, page 62. alon lavie and michael j denkowski. 2009. the meteor metric for automatic evaluation of machine translation. machine translation, 23(2-3):105-115. hoi le, reihaneh safavi-naini, and asadullah galib. 2015. secure obfuscation of authoring style. in ifip international conference on information security theory and practice, pages 88-103. springer. chris van der lee, albert gatt, emiel van miltenburg, sander wubben, and emiel krahmer. 2019. best practices for the human evaluation of automatically generated text. in proceedings of the 12th international conference on natural language generation, pages 355-368. yitong li, timothy baldwin, and trevor cohn. 2018. towards robust and privacy-preserving text representations. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 25-30, melbourne, australia. association for computational linguistics. bin liang, hongcheng li, miaoqiang su, pan bian, xirong li, and wenchang shi. 2018. deep text classification can be fooled. in proceedings of the 27th international joint conference on artificial intelligence, pages 4208-4215.