Exploring A mharic Hate Speech Data Collection and Classification Approaches (2023)

Abstract: In this paper, we present a study of efficient data selection and annotation strategies for Amharic hate speech. We also build various classification models and investigate the challenges of hate speech data selection, annotation, and classification for the Amharic language. From a total of over 18 million tweets in our Twitter corpus, 15.1k tweets are annotated by two independent native speakers, and a Cohen’s kappa score of 0.48 is achieved. A third annotator, a curator, is also employed to decide on the final gold labels. We employ both classical machine learning and deep learning approaches, which include fine-tuning AmFLAIR and AmRoBERTa contextual embedding models. Among all the models, AmFLAIR achieves the best performance with an F1-score of 72%. We publicly release the annotation guidelines, keywords/lexicon entries, datasets, models, and associated scripts with a permissive license.

## introduction in this digital era, social media platforms have become an important part of everyday life for people globally. the 2023 global digital report disclosed that nearly 5.16 billion people use the internet and the number of social media users exceeded 4.76 billion worldwide. over 64.4% of the world's population is already online, and nearly 60% of the people are active users of different social media platforms @xcite . hateful content targeting minorities is rapidly spreading across social media platforms and becoming a major socio-political and cultural challenge in the world @xcite . to tackle the problem, many countries, like ethiopia, crafted hate speech regulation laws even though the regulations have limitations for implementation (ayalew, 1 https://github.com/uhh-lt/ amharichatespeech 2020). moreover, there has been a rising interest among researchers in hate speech detection to expose and regulate this phenomenon with technological solutions. in this regard, researchers like @xcite ; @xcite ; @xcite ; @xcite ; @xcite have proposed several hate speech classification models and datasets for the development of automatic hate speech detection systems. despite many researchers claiming state-of-the-art performance on their own datasets, the models can not be generalized for all languages and datasets @xcite . ethiopia's legal regulations that were designed to counteract hate speech are not very well implemented. this is due to the complex nature of the online community, which is difficult to control by local laws, and the anonymity of online users who spread hateful messages while hiding behind their screens. moreover, the available hate speech classification models built for high-resource languages such as english could not be used for low-resource languages like amharic since such tasks incorporate cultural, social, and political variations in addition to language-specific differences. we have compiled amharic hate speech datasets from twitter and built classification models using different machine-learning approaches. in this paper, we addressed the following research questions, which are formulated for amharic, but also apply to other low-resource languages: the paper presented benchmark hate speech data selection approaches, a dataset consisting of over 15.1k annotated tweets, and various classification models. this work has the following main contributions: we organized the remainder of the paper as follows. the study provides introductory information about the amharic language in section 2. related works are presented in section 3. data collection and preprocessing details are discussed in section 4. data annotation strategies are described in section 5. we present classification models in section 6 and the results and discussion part of the paper in section 7. an error analysis of model results is described in section 8. section 9 concludes and shortly discusses future avenues, limitations are indicated in section 10. ## amharic language amharic is the second-largest widely spoken semitic language next to arabic. it is written from left to right with its own unique 'fidäl' scripts. fidäl is a syllable-based writing system where the consonants and vowels co-exist within each graphic symbol. amharic is the working language of the federal government in ethiopia and many regional states in the country @xcite . in amharic, there are 34 core characters each having seven different variations to represent vowels. besides, it has 51 labeled characters, 20 numerals, and 8 punctuation marks. amharic uses more than 310 unique characters and is a morphologically complex and highly inflected language @xcite . ## related work hate speech refers to language content that targets identity such as ethnicity, gender, disability, or political and religious ideology, which indirectly or directly focuses on their group identity and has the potential to incite violence @xcite . in contrast, offensive speech is a speech that usually targets individuals to be offended but not based on their group identity @xcite . hate speech has been addressed by many researchers using data scraped from online messages on social media. among the various studies, @xcite hovy (2016); davidson et al. (2017); founta et al. (2018); elsherief et al. (2018); ousidhoum et al. (2019); founta et al. (2019); winter and kern (2019); mathew et al. (2021); röttger et al. (2022b); demus et al. (2022); röttger et al. (2022a) conducted hate speech research in languages such as english, german, french, arabic, spanish, portuguese and hindi and published their datasets and models to advance further research. as indicated in ## data collection and preprocessing we have been collecting and storing amharic tweets every day since 2014 and built a twitter dataset in a relational database using the twitter api. our algorithm scrapes large numbers of tweets author size labels best method best score resources available mossie and wang (2018) 6,120 hate, not hate naïve bayes 79.8%: acc no mossie and wang (2020) 14,266 hate, not hate cnn-gru 92.6%: acc no tesfaye and kakeba (2020) 30,000 hate, free lstm 97.9%:acc no abebaw et al. (2021) 2,000 hate, not hate svm 92.5% :f1 dataset only abebaw et al. (2022) 2,000 hate, not hate mc-cnn 68.5% :f1 same dataset ayele et al. (2022b) 5,267 hate, normal, offensive roberta 50.0%: f1 yes ## data annotation previous studies on amharic hate speech classification such as mossie and wang (2018, 2020); abebaw et al. (2021) identified two classification categories (i.e. hate vs non-hate) while studies in english and other languages @xcite used hateful, offensive, and normal class categories. recently, the study by @xcite introduced the "unsure" category and employed four class categories, which are hate, offensive, normal, and unsure. we used the webanno foot_1 annotation tool, which is a web-based annotation framework for all annotations. ## classification models texts on social media platforms are usually unstructured, written in mixed scripts, and lack uniformity in writing styles than texts in the normal context. moreover, social media texts do not follow spelling/grammar rules as well as other language standards that make hate speech detection tasks a complex problem. hate speech is linguistically, culturally, and historically dependent on the context of the speech and requires developing classifiers that capture these dependencies @xcite . ## results and discussion we employed the 80:10:10 data split mechanism for creating the train, development, and test instances. we have used the development dataset to optimize the learning algorithms. all the results reported in the remaining sections are based on the test dataset instances. deep learning algorithms are computed using the following hyperparameters, embedding @xmath0, @xmath1, @xmath2, @xmath3, and @xmath4. f1-score (f1), precision (p), recall (r), and accuracy (acc) are used to compare the performance of the models. we conducted experiments with the classical machine learning models such as lr, lsvm, and nb; deep learning models like rnn, lstm, bilstm; and cnn, and the fine-tuned amharic transformer models such as amflair and amroberta. as presented in ## error analysis from model outputs we examined model-predicted tweets against their corresponding gold labels to observe discrepancies. as indicated in to showcase the possible justifications for the errors, we to took 5 tweets as presented in ## conclusion and future work the paper presented data selection and annotation strategies, and classification models for the amharic twitter dataset. a total of 15.1k tweets were annotated into hate, offensive, normal, and unsure classes. we proposed data selection and sampling strategies, a list of hate and offensive lexicon entries, and an annotated dataset for amharic hate speech research. we also presented both classical and deep learning models trained on a new dataset. the study explored hate speech annotation challenges and revealed that annotation of social media texts for hate speech classification is highly context-dependent. models that have used contextual embedding models such as am-flair and am-roberta outperformed all the models, where am-flair achieved the best scores of all. in future work, we plan to use semi-supervised active learning to select hateful tweets employing the human-in-the-loop annotation approach. exploring the targets of hateful content can also be another future work to deal with. to advance hate speech classification research in amharic and other low-resource languages; the dataset, hate and offensive keyword lexicons, the best-performing models, annotation guidelines, data selection pipelines, and associated source codes are publicly released with a permissive license foot_2 .