R e BERT at HSD -2 L ang 2024: Fine-Tuning BERT with A dam W for Hate Speech Detection in A rabic and T urkish (2024)

Abstract: Identifying hate speech is a challenging specialization in the natural language processing field (NLP). Particularly in fields with differing linguistics, it becomes more demanding to construct a well-performing classifier for the betterment of the community. In this paper, we leveraged the performances of pre-trained models on the given hate speech detection dataset. By conducting a hyperparameter search, we computed the feasible setups for fine-tuning and trained effective classifiers that performed well in both subtasks in the HSD-2Lang 2024 contest.

## introduction in this study, we have explored several fine-tuning strategies to establish bert @xcite models and compared their performances on two separate datasets, one in turkish and the other in arabic. we aimed to outperform the competitor models in the hsd-2lang subtask a and subtask b in detecting hate speech in tweets. the details of these subtasks are explained in the contest paper @xcite . bert is a widely used and accepted approach in the field of natural language processing (nlp) due to its efficiency and high performance in detecting hate speech compared to most conventional model architectures. the original bert paper proposed epoch numbers ranging from 2 to 4 and learning rates 5e-5, 3e-5, and 2e-5 with adam optimizer for fine-tuning bert @xcite . however, during our experimentation, we extended the range of the proposed hyperparameters in the original study. by enlarging the range, we were able to try various combinations of hyperparameters to enhance our model's performance. considering the competitive and limited nature of our task at hand, going beyond the suggested methods can be advantageous and provide a unique solution. our approach is insightful as it applies existing models and frameworks practically, and its competitive results offer valuable insights for future hate speech detection research in arabic, turkish, and other languages. ## related work the introduction of bert (bidirectional encoder representations from transformers) by @xcite has radically changed the field of detecting the nuances of languages contextually. the commonly adapted paradigm associated with bert consists of a pre-training and a fine-tuning step @xcite . the fine-tuning step is intended for the model to specialize on a specific task. the finetuning of a pre-trained bert model is proven to be significantly more robust compared to similar approaches offered in the past @xcite . in our case, this task was hate speech detection. hence, we have chosen our pre-trained models accordingly. mozafari et al. (2020) introduced a transfer learning approach where a pre-trained bert model is used for detecting hate speech in social media. the body of past research has laid a solid foundation upon which our study is constructed, enabling us to train our models with novel insights and methodologies. ## methodology we built and trained our models in python using the pytorch @xcite framework. we had access to google colab's a100 nvidia gpus via subscription, which helped us experiment with several architectures efficiently. the performance of the gpu was crucial since we had time constraints for achieving both tasks. a powerful gpu creates a significant difference, especially when training with relatively high epoch numbers (e.g., 100,1000). the training data consists of 9140 observations for subtask a and 960 observations for subtask b. no training or test data was used besides the datasets provided to us as a part of the competition. we initially tested the base performance of pre-trained models from hugging face foot_0 on each subtask's training data without applying fine-tuning or preprocessing. respectively, the initial models we decided not to proceed with were "arabert hate speech detecter" foot_1 developed by widadawane and "bert base turkish uncased" foot_2 developed by dbmdz. the used models are then selected according to their performance. afterward, we conducted a more structured hyperparameter search for the selected pre-trained models for fine-tuning. while training, we detected a data imbalance issue between the number of negatively (non hate speech) and positively (hate speech) labeled observations: in subtask a, the initial ratio was approximately 70/30, whereas in subtask b, it was even more skewed at 90/10. the fact that the imbalance is more apparent in subtask b is particularly significant due to the limited size of the data, which has the potential to make its effects more impactful. the reduced dataset size in subtask b amplifies the risk of model overfitting to the over-represented class, thus increasing the challenges in achieving a balanced and robust model performance. this situation was preventing the expected performance increase through fine-tuning. due to this observation, we only used 80 percent of the negatively labeled (non-toxic) training data for subtask b. the excluded negatively labeled observations were chosen randomly. we opted not to remove too many rows because the data is already very limited, which necessitated a careful balancing to avoid excessively diminishing our dataset's size. it is essential to mention that our initial goal was to have the best score in the competition rather than have a more general model that can detect hate speech on a wide variety of datasets. our only performance benchmark during training was the unlabeled public test dataset. the public test data covers only 20 percent of the total test data. therefore, we delve into finding a configuration for fine-tuning that performs better than the other competitors in the public dataset-assuming that the performance on the public dataset will carry over to the private test dataset (80 percent of the unlabeled test data). ## experimental setup for the preprocessing step, we didn't alter the grammatical attributes through processes such as lemmatization or stemmization. we therefore have not carried out any part-of-speech (pos) tagging, or similar analyses. this is due to the way that bert and transformer @xcite ) models function in general. from our literature review, we decided that it is best to keep the data as raw as possible, with the presence of punctuation as well as any expressions of tone, except only slight modifications to make it cleaner. we used the original preprocessing function by the turkishbertweet @xcite authors, which we then used as our main model for subtask a. this preprocessing function only converts url and emoticons into tags similar to html format, and doesn't apply any further modifications. for subtask b, no preprocessing was applied. we fine-tuned the submitted turkishbertweet model on nvidia a100 gpu and set the batch size equal to 32. after that, we set the max sequence length to 256, base learning rate to 5e-5, epsilon to 1e-8, and warm-up proportion to 10 percent of the total steps. we use the adamw optimizer, introduced by @xcite , with the default parameters and set the scheduler to polynomial weight decay. for the second subtask, we used the pre-trained arabertv02-twitter model @xcite . we finetuned the model on nvidia gtx 1070 and set the batch size equal to 8. we set the max sequence length to 250, base learning rate to 5e-4, epsilon to 1e-8, output @xmath0, output hidden @xmath1, and correct @xmath2. ## results and discussion model configurations can be seen in hyperparameter search for subtask b involves different combinations of epoch numbers and batch sizes. since the amount of training data for subtask b is limited to 960, fine-tuning with limited data involves the risk of overfitting. with limited training data, there is a higher risk that the model will memorize the training examples rather than learn generalizable patterns. this situation can lead to poor performance on unseen data for hate speech detection. furthermore, arabertv02-twitter is pretrained on various tasks with more than 60 million tweets. therefore, training on a small dataset may not effectively adapt the model's ability for hate speech detection. taking this condition into account, we trained our models with a linear decay scheduler with no warmup. our best-submitted model exceeded expectations performance-wise by scoring 0.89 in the public test set and 0.74 in the private test set. when comparing our top-ranking models with each other, we observed that the fine-tuned turkish-bertweet model performed worse than the finetuned arabertv02-tweet model if our sole consideration as a benchmark was the f1 score. however, there may be other factors to discuss before reaching such a conclusion. one potential factor is the limited data in the arabic subtask. it is unclear whether the f1 score is a sufficient metric by itself to compare models when at least one of those models is trained or evaluated on limited data. we also noticed that our submissions had lower variation in f1 score for subtask a, compared to subtask b, which may also be due to the limited data constraint in subtask b. hence, it may be misleading and out of scope to compare these two types of models to each other. ## conclusion in this paper, we conducted experiments to finetune pre-trained bert models for the hate speech detection task. we explore various approaches to maximize the performance of each algorithm by adjusting the hyperparameters. this paper focuses on the three primary hyperparameters: learning rate, batch size, and epoch length. our final leaderboard rankings in arabic and turkish subtasks turned out to be 1 and 2, respectively. this is a demonstration of consistent success, indicating that finetuning bert is a practical and effective approach for detecting hate speech in various aspects of a particular language. several factors, such as the choice of a pre-trained model and hyperparameters used in fine-tuning, contribute to obtaining a satisfactory result. the selection of the pre-trained model should be done under consideration of the format of the data. models pre-trained with data from twitter can help achieve better results. furthermore, the hate speech detection for this task requires a pre-trained model along with an effective tokenizer that is capable of tokenizing specific features for twitter, such as hashtags, urls, and emojis. such particular features can enhance the performance of the classification task by expanding the perception of toxicity in our model.