Don’t be a Fool: Pooling Strategies in Offensive Language Detection from User-Intended Adversarial Attacks (2024)

Abstract: Offensive language detection is an important task for filtering out abusive expressions and improving online user experiences. However, malicious users often attempt to avoid filtering systems through the involvement of textual noises. In this paper, we propose these evasions as user-intended adversarial attacks that insert special symbols or leverage the distinctive features of the Korean language. Furthermore, we introduce simple yet effective pooling strategies in a layer-wise manner to defend against the proposed attacks, focusing on the preceding layers not just the last layer to capture both offensiveness and token embeddings. We demonstrate that these pooling strategies are more robust to performance degradation even when the attack rate is increased, without directly training of such patterns. Notably, we found that models pre-trained on clean texts could achieve a comparable performance in detecting attacked offensive language, to models pre-trained on noisy texts by employing these pooling strategies.

## introduction as the internet becomes an important part of our lives, the prevalence of offensive language on online platforms, particularly social media, has become a serious concern @xcite . deep learning models for filtering offensive languages have been proposed to address this problem. however, malicious users have consistently found ways to avoid them. one such way is the deliberate insertion of additional typographical errors or substitution of certain characters with visually similar alternatives @xcite . despite numerous studies on this phenomenon in english, there has been a comparatively limited exploration in korean, which is a low-resource language characterized by distinct linguistic features @xcite . as the korean communities also suffer from the use of abusive language and cyberbullying @xcite @xcite , it is desirable to investigate the evasion tactics utilized by malicious users and to formulate them. while recent studies have discussed how to avoid offensive language detection @xcite @xcite , their definitions are ambiguous, and no clear solutions have been proposed to defend against the evasions. in this paper, we propose the evasion methods as user-intended adversarial attacks and incorporate them into offensive language from the perspective of malicious users. our proposed attacks are grounded in prevalent forms that can be found in offensive language online, and reflect the distinct features of korean language, wherein a single character can be further subdivided @xcite . we tested the proposed attacks on existing models for offensive language detection, and the results reveal that the performance declines as the rate of the proposed attacks increases. furthermore, we introduce simple yet effective pooling strategies in a layer-wise manner to defend against the proposed attacks. motivated by the exploration of the impact of each layer in a pre-trained language model @xcite , we selectively integrate useful features for the attacked offensive language across all layers 1 . the attacked texts have some changes in the tokens used, differing from the original texts. therefore, we implement pooling strategies to ensure that the model captures not only high-level features but also low-level features, which are related to offensiveness and token embeddings, respectively. this sim-ple modification enriches the understanding of the attacked offensive language, enhancing the robustness of the model against user-intended adversarial attacks without directly training of such patterns. the contributions of our study are as follows: ## conclusion we proposed user-intended adversarial attacks that occur frequently in offensive languages online from the perspective of malicious users. we categorized them into three types: insert, copy, and de-compose, which add special symbols or exploit the distinct features of the korean language. the involvement of attacks significantly affects the tokenization results from the original text. to address the proposed attacks, we introduced the pooling strategies in a layer-wise manner. this extension utilizes not only the last layer, which focuses on offensiveness, but also the preceding layers, which focus more on token embeddings. the experimental results indicated that first-last pooling was the most robust to the proposed attacks and could even achieve a comparable performance to that of models pre-trained on noisy texts, when applied to models pre-trained on clean texts. we especially demonstrated that rather than the middle layers, the first and last layers can be effectively employed to detect attacked offensive languages. furthermore, we experimented with the extent to which the introduced pooling strategies could handle the proposed attacks. we observed that the first-last pooling and max pooling are the most robust, depending on the nature of the texts used for pre-training. it is noteworthy that these strategies, without the explicit training of additional parameters or noisy texts, can effectively defend against user-intended adversarial attacks.