Hate Speech Classification in B ulgarian (2022)

Abstract: In recent years, we have seen a surge in the propagation of online hate speech on social media platforms. According to a multitude of sources such as the European Council, hate speech can lead to acts of violence and conflict on a broader scale. That has led to in- creased awareness by governments, companies, and the scientific community, and although the field is relatively new, there have been considerable advancements in the field as a result of the collective effort. Despite the increasingly better results, most of the research focuses on the more popular languages (i.e., English, German, or Arabic), whereas less popular languages such as Bulgarian and other Balkan languages have been neglected. We have aggregated a real-world dataset from Bulgarian online forums and manually annotated 108,142 sentences. About 1.74% of which can be described with the categories racism, sexism, rudeness, and profanity. We then developed and evaluated various classifiers on the dataset and found that a support vector machine with a linear kernel trained on character-level TF-IDF features is the best model. Our work can be seen as another piece in the puzzle to building a strong foundation for future work on hate speech classification in Bulgarian.

## introduction the term "hate speech" means public speech that expresses hate or encourages violence toward a person or group based on race, religion, sex, or sexual orientation foot_0 . hate speech is not something new. we can find evidence of it throughout history ranging from ancient greece, through rome and the middle ages up to modern times. it is no surprise that during times when the most prominent thinkers were freely expressing their hateful opinions and discrimination against minorities was part of both the law and religion, hate speech was omnipresent. however, identifying hate speech is a complex problem. who decides what hate speech is? aristotle would probably not consider his writings hateful, but two thousand years later, we might. today, social media platforms can enable people with discriminatory views to express their opinions more openly and under anonymity. furthermore, there have been multiple occasions in which there is a connection between online hate speech and increased violent hate-based activities. two very prominent examples of increased hate speech online following real-world events are a) hate speech towards immigrants and muslims following the manchester and london attacks after the uk left the eu. @xcite ; b) an uptick in racist and xenophobic harassment incidents following the presidential election in the us. @xcite . by the year 2020 hate crime had already achieved global recognition. in total, 118 countries and international organizations have laws on hate speech foot_1 . the connection between hate speech and hate crime has also already been studied more thoroughly in academia @xcite . in general, studying human behavior at scale by utilizing social media data has been the focus of researchers' attention for 15 years @xcite . while much research has been devoted to big platforms like facebook and twitter and focuses on a small number of languages, more recently, research on smaller and more specialized communities @xcite and less popular languages @xcite @xcite has become increas-ingly visible. we follow this branch of research and focus our attention on content in a language underrepresented in scientific research, namely bulgarian. bulgarian is a language spoken by approximately 8 million people around the globe, however, it plays an important historical role as the first slavic language to have an official alphabet. it was created and developed in the 9th century ad by the saints cyril and methodius and their disciples. it was the first slavic language into which the bible was translated. although not as damaging as the examples mentioned above, bulgaria suffers from an extremely high incidence of hate speech towards representatives of ethnic, religious, or sexual minorities @xcite . figure 1 showing the sentiment distribution of 1,475 comments in bulgarian following the syrian refugee wave-illustrates the gravity of the problem. article 162, paragraph 1 of the bulgarian penal code penalizes the more extreme forms of hate speech, hence one can conclude that bulgaria currently suffers from two problems in terms of hate speech prevention. first, the detection of hate speech and encouragement towards violent acts. second, is the enforcement of the law. this paper aims to address the first of these two points by collecting, filtering, and manually annotating real-world data, and by implementing, evaluating, and comparing various supervised learning models. the contributions of this article are: the model achieved a macro f1 score of 0.73. ## related work the rising visibility of hate speech on the online social platform has resulted in a continuously growing rate of published research into different areas of hate speech @xcite . due to the enormous volume of data that needs to be checked, more focus has been put on automatic detection algorithms. detecting hate speech has become an essential topic in the natural language processing community @xcite . as a result, a wide range of approaches to text classification was applied, and new datasets were created @xcite . the issue is that some more minor, less represented languages go under the radar. there have been efforts for language-agnostic text classification @xcite , however, these languages remain mainly ignored by the scientific community. bulgarian, for example, is one such language. some efforts @xcite have been made toward detecting toxicity in news articles in bulgarian, but the datasets tend to be too small. in recent years there have been numerous advances in natural language processing conducted by bulgarian researchers on various topics. in @xcite the authors present a new corpus of national legislative documents. in @xcite a multi-class multi-label classifier for social news is presented. in @xcite the author compares the performance of classifiers trained on features generated by a variety of state-of-theart pre-trained embeddings models for tasks such as named entity recognition and classification (nerc) and part-of-speech (pos) tagging. in @xcite a movie review dataset in bulgarian, sentiment lexicon, and a first-proceedings of clib 2022 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 sentence length 0 1000 2000 3000 4000 5000 6000 7000 number of sentences distribution of sentence length in dataset. of-its-kind fine-grained sentiment classifier are presented. word normalization methods such as stemming @xcite and lemmatization @xcite have also been explored, enabling more advanced natural language processing pipelines, sentiment analysis, and others. to address the problem of hate speech. an automatic detection algorithm has to be created. usually, this is done by training a machine learning model in a supervised manner for which huge amounts of annotated data are required. some authors @xcite also incorporate social network data features in the model training, however, we have abstained from this and focused explicitly on natural language. ## data the data required for our purpose was natural language written informally in bulgarian and, if possible, written as part of a dialogue or a comment on a subject. the biggest portion of data was directly provided by the bulgarian forum bg-mamma foot_2 which is the biggest bulgarian forum and its main user base is mostly comprised of current or future parents. except the data provided by them we also scraped other forums such as bg-jargon foot_3 and bg-nacionalisti foot_4 (bg-nationalists). bg-jargon is a website that collects bulgarian slang words and phrases and includes example sentences of how each word is used in everyday life. we have scraped exactly those sentences. bg-nationalists is an extremist right-wing political forum. about 80% of the data is from bg-mamma. the sentences consist on average of 14.3 words (median 11, standard deviation 12.2). all of the websites above contain mostly informal communication. this is further confirmed by the distribution of the sentence length as seen in figure 2 with moth sentences being short but also having a very long tail. while we can find one "sentence" with 685 words, 75% of sentences are 18 words or less. due to the nature of the main source of the data (bg-mamma) we were expecting predominantly non-hateful sentences. ## models for the classification, we trained classical machine learning models such as the logistic regression, support vector machines @xcite , decision trees @xcite , random forests @xcite and naive bayes classifiers @xcite . we also evaluated the performance of several neural network architectures. the most basic of which is a shallow neural network with keras' @xcite embedding layer and tensor-flow's @xcite textvectorization layer. another architecture we evaluated is the one discussed in @xcite . we used another architecture that was based on pre-trained word2vec embeddings which was fine-tuned on the corpus, and its weight matrix was used to set the weights of a keras embedding layer. after that, the architecture proposed above for the character-level-cnn was used again. lastly, we also used the stacked embeddings model for text classification provided by the flairnlp framework in a similar fashion as in @xcite . ## analysis the disparity in the distribution of the categories made us rethink how we should observe the classifiers' performance. a dummy classifier predicting only one class has 98% accuracy. for this reason, the primary metric we used is macro f1 which is an arithmetic mean of the f1-score for both classes and also the positive f1 score. the imbalance is ignored by using the mean of the two scores, and the two classes are equally weighted. however, other metrics can also be chosen, such as balanced accuracy, if the classifier is to be used in more practical settings (e.g., in the industry). balanced accuracy is defined as the average of recall for all categories in the classification. it is used as a substitute for accuracy for imbalanced datasets. it is also much easier to interpret than f-score. for the evaluation, the dataset was split into a training and testing set (75% train, 25% test). a 75-25 proportion instead of 70-30 was used because it allowed for a better distribution of the main five categories in both datasets. furthermore, although binary classification was performed, due to the data imbalance, the data was still split following a stratified approach for all five categories to achieve a similar distribution in both datasets. that was done to offset any additional bias towards one of the categories. ## discussion hate speech has always been a problem in society. the internet revolution reinforced the problem by providing instant connectivity across social media and anonymity. there also exists mounting evidence of a connection between hate speech online and hate crime. all of this has led to increased attention towards hate speech not only from the general public, but also from governments and private organizations. because of its online nature, and hence the amount of data that is being constantly generated, hate speech lends itself very well to automatic detection by an artificial intelligence model. to do this, however, large and robust datasets are required, and although they do exist, most of them are focused on languages with a strong internet presence such as english. as a result, many of the not so well represented languages-such as bulgarianare mostly ignored. multiple reports have shown that hate speech is an even greater problem in bulgaria than in other countries. for this reason, the scientific community in bulgaria should follow in the footsteps of such communities in other countries and focus on the issue. a first step in doing that is to create datasets that can be used for training purposes of future research. we believe that by sharing our dataset consisting of 108,142 manually annotated sentences, we can contribute making that first step. a further contribution of this paper is the evaluation of a variety of machine learning methods for the task of text classification in bulgarian in an imbalanced setting, including some state-of-the-art approaches. ## future work despite the continuing efforts of the scientific community, there are some fundamental issues with solving hate speech classification. for example, in @xcite , the authors argue that researchers have become overly optimistic about the results of their classifiers. that is because most research papers focus only on datasets coming from one source. that causes the models (usually deep neural networks) to overfit and are rarely able to generalize well on new datasets. therefore, the creation of multiple datasets is compulsory for the development of a robust predictive model. another issue is annotator bias. in @xcite , waseem discusses how much the influence of annotators on the performance of classifiers and suggests that systems trained on data labeled by experts perform better than those labeled by amateurs. that leads us to another fundamental issue with hate speech classification: who defines what hate speech is? to mitigate any annotator bias future datasets should not only be annotated by experts but also, if possible, by different people. another important point that could be addressed in the context of imbalanced text classification is data augmentation (da). data augmentation is the process of creating artificial data to improve the performance of a classifier. one can argue that some aspects of data augmentation are already included by incorporating smote into the preprocessing pipeline; however, smote works on the feature space of the vectorized textual data. what might greatly impact the classifier's performance would be to augment the data at the textual level. there are multiple ways of performing this, from using a thesaurus to substitute words on a synonym level to using model embeddings (for example, word2vec) to sample neighboring words. this approach has been shown to increase the performance of hate speech classifiers. @xcite a further unexplored method to improve the classifiers' performance is employing additional feature engineering methods such as named entity recognition and part-of-speech taggers. these methods would enrich the feature space and result in a better classifier. lastly, if the context in which a model is to be used is social media, a further feature engineering idea would be to take user metadata such as age, location, or gender into account. furthermore, proceedings of clib 2022 a "hate score" could be calculated for each user based on her or his past posts or her/his connections' past posts by utilizing social network analysis techniques.