“Define Your Terms” : Enhancing Efficient Offensive Speech Classification with Definition (2024)

Abstract: The propagation of offensive content through social media channels has garnered attention of the research community. Multiple works have proposed various semantically related yet subtle distinct categories of offensive speech. In this work, we explore meta-learning approaches to leverage the diversity of offensive speech corpora to enhance their reliable and efficient detection. We propose a joint embedding architecture that incorporates the input’s label and definition for classification via Prototypical Network. Our model achieves at least 75% of the maximal F1-score while using less than 10% of the available training data across 4 datasets. Our experimental findings also provide a case study of training strategies valuable to combat resource scarcity.

## introduction while a vital channel for the dissemination of crucial information, social media platforms have also become hotbeds for hateful, and harmful expressions. such offensive speech not only detracts from the quality of discourse but also poses tangible threats to marginalized and vulnerable groups, escalating existing social tensions. multiple studies have observed the psychological harms to marginalized communities perpetuated by offensive content in the digital space @xcite ; s , tefȃnit , ȃ and buf, 2021). however, the definition of offensive speech varies between contexts, and across publications that study this problem. a common challenge with offensive speech is the lack of a unifying definition, with conceptually related but definitively distinct categories proposed in literature: hate, abusive, aggressive, toxic, offensive, cyberbullying etc. @xcite . while earlier research focused on binary classification, more current works have explored offensive categories in higher granularity and semantic diversity @xcite @xcite @xcite . though an active area of research in the natural language processing (nlp) community, accurate and reliable detection of offensive speech often requires significant amount of training data @xcite . for these tasks, the typical pipeline of data collection involves gathering a candidate corpus based on a set of relevant keywords, then soliciting task-specific labels for them via crowdsourcing or expert annotation @xcite . demographics of annotators may be different from one dataset to the next, including platforms (e.g. amazon mechanic turk, prolific), payments, levels of education, languages and cultural backgrounds @xcite . as offensive content is frequently linked to real world events, there exists a need for appropriately tailored datasets. nevertheless, constructing a sufficient amount of labelled data often proves a resource-intensive challenge @xcite @xcite . on the other hand, there exists a plethora of available data on similar yet categorically distinct areas of offensive speech. we set out with the objective to discover suitable techniques capable of leveraging existing datasets to efficiently and reliably adapt to new domains of offense content. to this end, we compile from literature a collection of 14 relevant datasets, which allow us to perform a battery of testing on various pre-training and meta-learning approaches to assess their efficacy and robustness in classification of offensive content. we also experiment with different model architectures to incorporate label information to enhance knowledge transference at multiple levels of data availability. we introduce je_protonet, a joint embedding based on prototypical network which utilizes definition of label categories and exhibit competitive performance across 4 test sets while using a fraction of the available training data. to the best of our knowledge, this work is the first in literature that harnesses label definition in offensive speech detection. our experiments provide a case study on the trade-off between sample efficiency and performance, with findings potentially applicable to any classification task where categories entail more nuanced expression beyond simple labels. based on empirical findings, we provide a set of recommendations to leverage our approach to enhance the efficiency of offensive speech classification. ## experimental setup from here on, we refer to the datasets as domains. with the goal of investigating the potential benefits of learning from semantically related but distinct data, our general experiment pipeline consists of first pretraining a model on the 10 reserved domains using different techniques 2 . then, the model is fine-tuned and evaluated on each of the 4 aforementioned test domains. more specifically, we hold out a portion of the test domains using the ratio described in their original publications ( ## results macro f-1 score is chosen as the evaluation metric. we first discuss the performance of models relative to each other in their respective setting, then provide a top-down analysis. figure 2 illustrates the performance for each setting. detailed numerical results are displayed in ## discussion definition matters while many works in offensive speech literature have focused on standard classification techniques, ours is the first to leverage the definition of associated labels. our proposed framework to incorporate definition via the joint embedding is beneficial to boost classification performance over other models, given the same amount of training data. in addition to enhancing annotation quality, this factor is yet another signal to encourage researchers to pay more attention to their terminologies to both enhance downstream tasks and facilitate cross-task studies. more data is not always needed our experiments provide a case study on how much data is necessary to achieve certain results in the area of offensive speech detection. while having more labeled data is always preferable, the annotation process can be expensive, and thus constituting a barrier to researchers not equipped with abundant resources. ## conclusion while we also leverage the existing rich corpora, our work explores a different setting of offensive speech detection compared to other works, such as hatebert or fbert @xcite . the proposed joint-embedding may be adapted to complement other existing architectures. our approach can also be applied to other nlp tasks, such as sentiment analysis and stance detection, where labels extend beyond compact phrases. we invite researchers to explore definitions and the extent of their usefulness in other tasks. offensive content is ever-evolving in today's world. we hope that our findings provide useful pointers for nlp practitioners to more efficiently explore diverse topics in this field.