Abusive Speech Detection in S erbian using Machine Learning (2024)

Abstract: The increase in the use of abusive language on social media and virtual platforms has emphasized the importance of developing efficient hate speech detection systems. While there have been considerable advancements in creating such systems for the English language, resources are scarce for other languages, such as Serbian. This research paper explores the use of machine learning and deep learning techniques to identify abusive language in Serbian text. The authors used AbCoSER, a dataset of Serbian tweets that have been labeled as abusive or non-abusive. They evaluated various algorithms to classify tweets, and the best-performing model is based on the deep learning transformer architecture. The model attained an F1 macro score of 0.827, a figure that is commensurate with the benchmarks established for offensive speech datasets of a similar magnitude in other languages.

## introduction as the number of web and social network users increases, abusive speech and its detection are becoming very important @xcite . the concept of abusive speech, in the context of this paper, is an umbrella term for phenomena such as profanities or offensive and hate speech. @xcite defined abusive language as 'hurtful language that a speaker uses to insult or offend another individual or a group of individuals based 2018b), online sexisms detections semeval 2023 @xcite , etc. here we present our research on identifying abusive speech in tweets in serbian language. as a dataset, we used the abcoser corpus @xcite with the primary focus on detecting whether a tweet contains abusive content or not. to accomplish this task, we employed numerous machine learning algorithms, ranging from traditional machine learning and n-gram features to modern transformer models. the remainder of the paper is structured as follows. related work is given in section 2, containing a short overview of the machine learning algorithms and systems used for abusive speech detection. the description of the dataset used in our study is given in section 3. an overview of the methods used in our research is presented in section 4. the results of abusive speech detection classification algorithms are presented in section 5. in conclusion, we summarize the results of our research and indicate further research directions. ## related work the most common strategy for detecting offensive speech on the web is to train the system to recognize offensive content, which would then be deleted or forwarded to the site moderators @xcite . since the first work on smokey flame detection system @xcite until nowadays, the majority of the approaches to abusive content detection are based on supervised machine learning. this is done either by using traditional approaches that rely on machine learning models with feature extraction methods, or by applying deep learning architectures that have been predominant in recent years. some of the systems employ specialized lexica or blacklists of abusive terminology either as the only or as a supplementary tool for the abusive language detection systems in social media @xcite @xcite @xcite . these lexica can help to detect explicit swear words and profanities in the text directly @xcite . however, they are not a sufficient resource for hate speech detection. when building a classifier for abusive speech detection, the researchers usually employed two types of features. the first group of features is based on n-grams, linguistic and syntactic characteristics of text, which are combined with traditional machine learning algorithms like vowpal wabbit regression model @xcite , logistic regression classifier @xcite , svm @xcite @xcite . the second group of features relies on word embeddings, obtained by feeding deep neural networks with vast amounts of text data, such as glove @xcite , elmo @xcite or word2vec @xcite in combination with gradient boosting decision trees, lstm or combination of cnn and rnn neural network architectures. in recent research, transformer based large language models like bert @xcite have been predominantly used for offensive speech detection as they outperform other methods @xcite . in a comparative study of the application the contemporary large language models for the task of offensive language identification @xcite , the authors used zero-shot prompting with six models and demonstrated that only flan-t5 @xcite reached performance close to but not better than state-of-the-art models from offensval competitions. in addition, that was the only model that supported languages other than english. the first paper dealing with hate speech in serbian language by @xcite presented the results of an information search experiment in quest for attacks which are the result of national, racial, or religious hatred and intolerance on a corpus of newspaper articles. the abcoser was the first abusive speech dataset in serbian language @xcite presented together with ontolex lemon lexicon developed to facilitate abusive speech detection. vujičić and mladenović (2023) curated a hate speech lexicon and a dataset in serbian language to train a classifier for automatic hate speech detection in sports domain. they experimented with bilstm deep neural network, and the results showed high precision of detecting hate speech in sports domain (96% and 97%) and low recall. ## dataset in this research, we have used the abcoser corpus that consists of 6,436 tweets out of which 5,020 with regular speech and 1,416 annotated as abusive speech @xcite . the abcoser corpus contains general abusive speech, meaning that it's not prepared with the focus on a specific type of targets such as racial, lgbt or misogyny speech. the corpus resulted from a random sampling of tweets from a timeline of 111 twitter users, whose profiles were gathered via crowdsourcing and manual search as the ones who are more likely to generate abusive speech. the dataset was annotated by using a hierarchical annotation scheme, similar to @xcite . the scheme is presented in ## research methodology the pre-processing of text data in our dataset is an important step to make it easier to extract information and apply machine learning algorithms. twitter data differs significantly from other types of texts, e.g., books or newspaper articles, meaning that there are specific issues that have to be considered when processing nonstandard serbian language present in twitter @xcite . for all the models we applied the following preprocessing steps: ## results to get an observable picture of the results, we present results of a dummy (all off) classifier that assigns to each record the label of a most frequent class, that could serve as a default baseline model. although we executed a vast number of modelclassifier experiments, for the sake of the scope of this paper, the results are presented in ## conclusion in this paper, we presented the results of various systems performance on the automated abusive speech detection task in serbian language. a number of models were evaluated, ranging from traditional ones using bow, tf-idf and text features combined with machine learning classifier algorithms, over word embeddings and deep learning architectures, to state-of-the-art transformer models. by far the best algorithm was obtained by finetuning bertić @xcite for classification of abusive tweets. the best traditional model in our study was acquired by using tf-idf 3-5 character n-grams and passive aggressive classifier on the dataset with restored diacritics. the surprise was the excellent result of the passive aggressive classifier, which has not been mentioned in relevant literature. deep learning models had lower performances possibly due to the small size of our dataset for these models. in future work, we plan to extend the abcoser corpus with new tweets and short texts from other sources e.g. online news comments, while addressing the issue of labels imbalance on both annotation levels. in addition, we would focus on application of extra methods for text preprocessing such as conversion of abbreviations and emoticons, application of better lemmatizer for serbian @xcite , processing of negation in serbian language @xcite etc. in order to improve the recall rate, which currently stands at 0.6520 for the abusive category, it's important to understand that abusive comments can also include implicit bullying through the use of irony or sarcasm @xcite . therefore, employing a separate classifier, like the one suggested by @xcite , specifically trained to detect irony and sarcasm, could prove to be advantageous. based on error analysis, we envision that a hybrid classification system model which combines traditionally crafted text features, abusive speech lexicon that includes mwes @xcite , with a modern transformer model would provide most robust solution for an abusive speech detection system for serbian language. language off not total f1 score english 4,640 9,460 14,100 0.8290 arabic 1,991 8,009 10,000 0.9017 danish 425 2,865 3,290 0.8119 greek 2,911 7,376 10,287 0.8522 turkish 6,847 28,441 35,288 0.8258 serbian 1,416 5,020 6,436 0.8270 dataset statistics