Authorship Obfuscation in Multilingual Machine-Generated Text Detection (2024)

Abstract: High-quality text generation capability of latest Large Language Models (LLMs) causes concerns about their misuse (e.g., in massive generation/spread of disinformation). Machine-generated text (MGT) detection is important to cope with such threats. However, it is susceptible to authorship obfuscation (AO) methods, such as paraphrasing, which can cause MGTs to evade detection. So far, this was evaluated only in monolingual settings. Thus, the susceptibility of recently proposed multilingual detectors is still unknown. We fill this gap by comprehensively benchmarking the performance of 10 well-known AO methods, attacking 37 MGT detection methods against MGTs in 11 languages (i.e., 10 × 37 × 11 = 4,070 combinations). We also evaluate the effect of data augmentation on adversarial robustness using obfuscated texts. The results indicate that all tested AO methods can cause evasion of automated detection in all tested languages, where homoglyph attacks are especially successful. However, some of the AO methods severely damaged the text, making it no longer readable or easily recognizable by humans (e.g., changed language, weird characters).

## introduction recent advances in language modeling have birthed large language models (llms) which exhibit significant improvements, including the ability to generate texts easily misconstrued as humanwritten @xcite . in addition, llms have been found to exhibit emergent abilities @xcite , such as performing several nlp tasks at or over the human level, e.g., an llm passing academic exams designed for humans in top 10% of test takers @xcite . while this is a celebratory feat for the nlp community, llms currently have several shortcomings like the generation of hate and biased speech @xcite , 2023; @xcite , the generation of memorized and plagiarized content @xcite @xcite , and the generation of mis/disinformation @xcite @xcite . furthermore, we find that some of these llms like chatgpt, flan-t5 @xcite , falcon @xcite , llama @xcite , etc. can have their alignment tuning bypassed by jailbreaking prompting techniques @xcite . therefore, due to the great potential to be misused, it is imperative that at the minimum, we can accurately distinguish machine-generated texts (mgt) from human-written ones. the mgt detection task has been studied very well for the binary case (turing test), showing that accurate detection of machine-generated texts is possible @xcite @xcite . on the other hand, in this work, we investigate the less comprehensively studied problem for mgt detection-authorship obfuscation (ao), which refers to the process of masking an author's writing style/signature, while preserving semantics. in the mgt detection task, ao represents an attack vector for mgt evading detection, being successful when true positives (i.e., detected mgt) are changed to false negatives (i.e., undetected mgt). we evaluate the robustness of mgt detectors to adversarial perturbations by implementing ao techniques (see figure 1 for an illustration). since the task has been solely studied for monolingual settings, mostly english @xcite , but also chinese or russian @xcite @xcite , we focus on the multilingual perspective. our key contributions are: ## related work with the advancements of llms, the mgt detection methods also adapt to multilingual settings @xcite . however, it is known that mgt detection is susceptible to ao methods (e.g. paraphrasing), which can drop the detection performance even by 90% @xcite @xcite . as previously mentioned, ao methods have been evaluated in monolingual settings only. for some other nlp tasks, adversarial attacks (as a subset of ao) have been evaluated in multilingual set-1 https://doi.org/10.5281/zenodo.13846588 tings @xcite . however, multilingual evaluation of ao in mgt detection task is still missing, thus the severity of a potential threat of detection evasion is unknown. ## methodology the methodology is illustrated in figure 1 , summarizing key components. we benchmark 10 wellknown ao methods attacking 37 mgt detection methods against texts in 11 languages (i.e., 10 × 37 × 11 = 4,070 combinations). all source codes, data, and full results are publicly available to enable full replication of our study foot_2 . ## experiments and results firstly, our experiments focus on finding whether the existing ao methods (so far evaluated primarily for english) are even usable for non-english languages. secondly, we focus on finding how robust the existing mgt detection methods are in regard to susceptibility to ao. lastly, we evaluate the effect of using obfuscated texts for data augmentation to increase their adversarial robustness. to compare mgt detection performance, we use standard metrics, namely auc roc (area under the curve of receiver operating characteristic) as a classification-threshold independent metric (not affected by a threshold calibration on a domain data) and macro avg. f1-score as a metric balancing between a precision and a recall of the classification commonly used for the mgt detection task using imbalanced datasets. we further provide macro avg. f1-score values theoretically achievable on the used data with the classification thresholds calibrated based on the roc curve for optimal (i.e., maximal difference between tpr and fpr, where tpr is true positive rate and fpr is false positive rate), 1% fpr and 5% fpr conditions. we are aware that such a performance is not actually achievable due to calibration on the evaluation data (i.e., data leakage). therefore, we are not using it to compare performance of the mgt detection methods, but to evaluate effect of ao on the detection (i.e., compare ao methods), while not being biased due to sub-optimal predictions. we use two primary metrics to evaluate the effect of ao methods. asr (attack success rate) measures the effectiveness of changing true positives to false negatives, considering all ao methods as attacks on detection. auc roc drop (detection performance decrease) is used for measuring the effect of obfuscated texts on detection performance. before diving deeper into the experiments, let us briefly examine the general detection performance on our data (i.e., difficulty of the data for this task). ## conclusion in this work, we have done the first comprehensive benchmark of authorship obfuscation methods in multilingual settings. we have also done the first evaluation of adversarial robustness (using authorship obfuscation methods) of multilingual machinegenerated text detection methods. we have crafted a new public dataset of 740k obfuscated texts, which has wide usage possibilities, even beyond the purpose outlined in this work (e.g., to study differences in obfuscated human-written and obfuscated machine-generated texts, or to evaluate robustness of multi-class authorship-attribution systems). our results (namely nonzero attack success rate values) indicate that the existing authorship obfuscation methods, although focused primarily on english, are usable in all 11 tested languages, thus posing a potential threat for automated multilingual detection. however, human quality check of the obfuscated texts disqualified two of them as unusable in non-english and/or non-latin script languages. the adversarial robustness results show that there are differences among machine-generated text detection method categories as well as among tested languages. the results confirmed that english monolingual fine-tuning of multilingual detectors is not suitable, since it is the least resistant against obfuscation. as we have shown, the adversarial retraining using a simple data augmentation by obfuscated texts can significantly increase the adversarial robustness of detectors, especially against homoglyph and paraphrasing attacks. future work should be focused on iterative and hybrid authorship obfuscation of multilingual texts to evaluate severity of such threat on machinegenerated text detection.