Teacher and Student Models of Offensive Language in Social Media (2023)

Abstract: State-of-the-art approaches to identifying offensive language online make use of large pre-trained transformer models. However, the inference time, disk, and memory requirements of these transformer models present challenges for their wide usage in the real world. Even the distilled transformer models remain prohibitively large for many usage scenarios. To cope with these challenges, in this paper, we propose transferring knowledge from transformer models to much smaller neural models to make predictions at the token- and at the post-level. We show that this approach leads to lightweight offensive language identification models that perform on par with large transformers but with 100 times fewer parameters and much less memory usage

## introduction the presence of offensive content in social media has been linked to suicide attempts and ideation in teenagers, psychological stress, and other undesirable consequences to users @xcite . this motivated researchers to develop models to identify offensive content automatically. there have been several recent studies published on identifying different types of offensive content such as profanity @xcite , pejorative language @xcite , abuse @xcite , hate speech @xcite , cyber-bullying @xcite , and toxicity @xcite . while most studies on this topic deal with the identification of offensive content at the post level, recent work @xcite has addressed the identification of offensive tokens in posts along with their targets @xcite , which can assist human moderators (e.g., news portals moderators) who often review lengthy comments @xcite the two authors contributed equally to this work. @xcite . overall, both token-level and sentence-level tasks are important to real-world applications. recent international competitions on the topic, such as offenseval @xcite , @xcite and hasoc @xcite , have shown that large pre-trained transformer models such as bert @xcite , and xlnet @xcite deliver state-of-the-art performance in offensive language identification at both sentence and token levels. however, the models based on transformers have an extremely large number of parameters and are well-known for demanding computing resources such as disk and ram. this makes inference time slow, posing challenges for real-time inference and their deployment in the real world. making models smaller and more usable in practice is an active area in machine learning and nlp @xcite . one approach is knowledge distillation (kd) which aims to extract knowledge from a top-performing large model (the teacher) into a smaller, yet well-performing model (the student) @xcite . the student model is ideally a model which is less demanding in terms of memory print, computing power and with lower prediction latency. distilbert is such an example where kd has been used to create smaller language models @xcite . the search for computational efficiency is in line with several initiatives such as green ai @xcite and the acl's efficient nlp policy. 1 in this paper, we perform a detailed evaluation of performing kd in offensive language identification. first, we evaluate kd in sentence-level offensive language detection. previous attempts for kd in text classification propose to use computationally augmented synthetic data @xcite to address data scarcity @xcite . we address this limitation by using a very large offensive language identification dataset in the kd process. secondly, we evaluate kd in token-level offensive language identification. we introduce a novel way to perform kd at the token level by using the viterbi algorithm @xcite . in both sentence-level and token-level, we show that smaller student models deliver results on par with kd despite requiring fewer computing resources, and interestingly smaller student models outperform machine learning models based on distilled transformers such as distilbert. to the best of our knowledge, this is the first comprehensive kd research applied to offensive language online opening exciting new avenues for research in this area. the findings of this research will be beneficial not only to offensive language research but also to a wide range of related social media tasks, such as sentiment analysis and fake news detection. the contributions of this paper are as follows: ## methods figure 1 summarizes our teacher and student approach. we follow two different approaches for sentence-level and token-level described in the next sub-sections. ## computational efficiency we observe that the student models provide efficient and greener models @xcite compared to the popular transformers models in terms of parameters, processing speed, and inference time. the kd strategy presented in this paper allows the student models to perform competitively, as presented in section 4.1. ## conclusion and future work in this paper, we showed that kd through a teacherstudent approach that directly distills offensive labels can be effective in building lightweight offensive language identification models in both sentence and token-level. kd allows the lightweight student models to outperform distilled yet large pre-trained state-of-the-art architecture such as dis-tilbert. furthermore, they perform competitively with large state-of-the-art architectures such as bert and xlnet. these students models are 15 times smaller in disk space with 100 times fewer parameters, and 3 times faster in inference speed demonstrating that kd can contribute to greener computing. the search for computational efficiency is in line with several initiatives such as green ai @xcite and the aforementioned acl's efficient nlp policy. as the kd process we used does not involve task specific configurations, the findings of this research can be easily expanded to different tasks involving social media texts such as sentiment analysis and fake news identification. for the kd in token-level offensive language identification, we introduced a novel method involving the viterbi algorithm to reduce the complexity of the task. we showed that the student models can be improved with taking the top 1 label sequence from the viterbi algorithm. however, adding more instances with top 2 etc can also reduce student performance. our approach will benefit multiple applications that have token-level labels. furthermore, our approach can be used to generate more token-level data addressing the scarcity of available token-level datasets. in the future, we would like to experiment with cross-lingual kd architectures. transformer-based cross-lingual approaches have been previously successfully applied for this task @xcite . therefore, we would like to explore large transformer models such as xlm-r that can be used as teachers to distil knowledge from resource rich languages to resource poor languages thus benefit multiple languages around the world for which offensive language datasets and resources are not yet available.