A Multifaceted Framework to Evaluate Evasion, Content Preservation, and Misattribution in Authorship Obfuscation Techniques (2022)

Abstract: Authorship obfuscation techniques have commonly been evaluated based on their ability to hide the author’s identity (evasion) while preserving the content of the original text. However, to avoid overstating the systems’ effectiveness, evasion detection must be evaluated using competitive identification techniques in settings that mimic real-life scenarios, and the outcomes of the content-preservation evaluation have to be interpretable by potential users of these obfuscation tools. Motivated by recent work on cross-topic authorship identification and content preservation in summarization, we re-evaluate different authorship obfuscation techniques on detection evasion and content preservation. Furthermore, we propose a new information-theoretic measure to characterize the misattribution harm that can be caused by detection evasion. Our results reveal key weaknesses in state-of-the-art obfuscation techniques and a surprisingly competitive effectiveness from a back-translation baseline in all evaluation aspects.

## introduction authorship obfuscation is the task of masking the writing style of an author of a document to prevent authorship identification techniques from using stylistic patterns to reveal the author's identity @xcite . the motivation for this task is to protect the public from the misuse of authorship identification techniques to suppress freedom of speech or to persecute whistle-blowers. when a new authorship obfuscation technique is proposed, it is crucial to compare its effectiveness to state-of-the-art obfuscation tools in settings that accurately depict the real-life environment where such a tool may be used. one important assumption that should be made is that a de-anonymizer is likely to use the most competitive authorship identification tool available to identify the author of a document. using an inferior or brittle authorship attribution technique, or a weak obfuscation baseline will overstate the performance of such obfuscation tools. for example, previous work on obfuscation has evaluated obfuscation techniques against an identification tool that uses the exact same features and classifier that were used to obfuscate it in the first place @xcite . this could lead to misleadingly high impression of the system's effectiveness. similarly, obfuscation techniques must convey the same intended message both before and after obfuscation. therefore, when a new obfuscation tool is proposed, it is evaluated on both the quality of obfuscation and its ability to preserve the content. with the recent development in language models and their ability to generate text, many automatic measures have been introduced to evaluate the quality of this generated text @xcite , and some of these measures have been used in obfuscation techniques. the problem with existing content-preservation measures, however, is that they only provide an abstract, numerical score that limits the user's ability to pinpoint the part of the text that suffered from loss of information and requires re-modification. recently, question answering-based approaches were proposed and shown to provide meaningful feedback in the form of questions that tell the user which information in the text has been changed and in which part @xcite . the concept of evasion in authorship obfusca-tion has a critical, potential harmful side-effect that we raise for the first time. in a classification setting, the typical setting in which evasion of obfuscation techniques are evaluated, a classifier has to pick one author from the set of candidate authors. an obfuscation technique may obfuscate a document by imitating another potential author, in effect unfairly "framing" another person. to investigate this behavior, we use an information-theoretic approach to evaluate the potential for misattribution of the obfuscating technique. specifically, we propose a new evaluation measure; namely, misattribution harm where the goal is to characterize the confidence in the attribution algorithm rather than its output. in this work, we highlight a number of issues with the existing work on obfuscation with respect to two dimensions: obfuscation effectiveness, and content preservation. we further propose a new evaluation dimension namely, misattribution. our key contributions are the following: ## background authorship obfuscation @xcite techniques aim to hide an author's writing style which can be used by authorship identification tools to reveal the true identity of that author. here, the assumption is that the author has already taken the precautions to hide their identity by removing any identifying information such as their name or address from the text. by using obfuscation techniques, users aim to hide their writing habits which may or may not be known to them. with that in mind, it is important that the obfuscated text contains the same conveyed message after obfuscation. ## a multifaceted evaluation framework ideally, authorship obfuscation should only modify the author's writing style in a document while retaining all the original information. however, due to the topic-writing style entanglement, modifying the document is likely to cause information loss; i,e., some content is not preserved. based on that, obfuscation techniques are evaluated in two dimensions: evasion, and content preservation. in the following subsections, we formally describe obfuscation, evasion and content preservation and we discuss the state of the tools used to evaluate them. finally, we propose a novel evaluation dimension to characterize a potential side effect of a successful detection evasion namely, misattribution. ## experimental setup our overall evaluation procedure is as follows. we started by establishing the authorship identification accuracy on the original datasets. note that, the training and testing split is predefined for each dataset as shown in we followed that by creating different obfuscated copies of the test sets, one for each obfuscation technique. next, we evaluated the detection evasion and misattribution on each obfuscated copy in one step. we concluded our evaluation with content preservation. we provide below the details of each step separately. ## conclusion in this work, we demonstrated the importance of using state-of-the-art evaluation tools to measure the performances of authorship obfuscation techniques. in addition, our experiments revealed that current obfuscation techniques have key weaknesses and have been outperformed by a baseline, namely back translation in multiple evaluation aspects. furthermore, we identified a critical issue with respect to the fairness of obfuscation techniques. our proposed misattribution measure investigates the sideeffect of a successful detection evasion by identifying another author as the most plausible author of the obfuscated text. as a result, we argue that an attack on the confidence of the identification model, by generating text in a generic style would confuse the identification model and make it unusable in real-life applications. finally, we argue that evaluation of authorship obfuscation tools should follow the rapidly evolving domain of evaluation tools while keep the potential users and real-life applications when developing and evaluating novel obfuscation techniques.