No offence, Bert - I insult only humans! Multilingual sentence-level attack on toxicity detection networks (2023)

Abstract: We introduce a simple yet efficient sentence-level attack on black-box toxicity detector models. By adding several positive words or sentences to the end of a hateful message, we are able to change the prediction of a neural network and pass the toxicity detection system check. This approach is shown to be working on seven languages from three different language families. We also describe the defence mechanism against the aforementioned attack and discuss its limitations.

## introduction toxicity detection systems have become a crucial part of automoderation solutions. they are now used by most social media platforms, including those with groups of people in dangerously sensitive conditions, e.g. suicide prevention, facebook groups, victims of cyberbullying, etc. vulnerability in such systems may have dreadful, terrific effects on people, especially those in precarious situations. @xcite . on the other hand, such systems can be and are used to silence the voices of criticism, which leads to the creation of echo chambers and amplifies the voice of the (powerful) minority over the voice of the majority, thereby destroying the foundation of democracy and denying the freedom of speech @xcite . this situation can be viewed as a double-edged sword, and we propose another double-edged sword that can be used to parry the blade of toxicity detection systems. ## defence we performed simple adversarial training of the distilbert model on the binary jigsaw toxic comments dataset to improve the model's defence. we performed experiments with word-and sentence-based attacks, attacking both toxic messages and all messages in the dataset. in addition, we picked out only toxic messages and attacked half of them -in this scenario, the task was distinguishing attacked and non-attacked texts. ## discussion all of the attack variants showed decent results in confusing toxicity detection models in every model tested (to varying degrees). the simplistic nature of such attacks could allow virtually any internet user to exploit them to avoid automoderation systems. a possible countermeasure for this type of attack is adversarial training or rule-based filtering. however, a simple rule-based filter can be fooled by char-level adversarial attacks, and adversarial training can be made much more difficult by inn sentences bert roberta electra 0 0,590 0,962 0,898 1 0,001 0,001 0,232 2 0,001 0,002 0,147 ## conclusion in this paper, we demonstrated a novel and effective way of bypassing toxicity detection models by appending positive words or sentences to the end of a toxic message. the introduced concept of the "to each his own" attack is based on the idea of separating the messages addressed to a human and those addressed to an algorithm. the attack exploits the fact that the toxicity detection models are trained on sentence-level labels and do not consider the context or the intention of messages. the described attack in all its variants can be easily used on almost any toxic-detection neural model, with fairly good results. for future research, we suggest looking for a more sophisticated way of constructing insertions, perhaps with respect to the original message, making it a semantically correct addition to human-written messages. we are grateful to the reviewers and the program chair of the emnlp conference for providing constructive feedback, suggesting additional experiments and providing different points of view. this helped us structure the paper and ensure the robustness of our results. sergey berezin wishes to record that he is grateful to mr. mario fritz from saarland university for the introduction to the domain of adversarial attacks through his insightful course on this topic. he is also grateful to mr. maxime amblard from the university of lorraine for encouraging him to pursue research in this direction and validating his ideas.