Exploring Hate Speech Detection Models for L ithuanian Language (2025)

Abstract: Online hate speech poses a significant challenge, as it can incite violence and contribute to social polarization. This study evaluates traditional machine learning, deep learning and large language models (LLMs) for Lithuanian hate speech detection, addressing class imbalance issue via data augmentation and resampling techniques. Our dataset included 27,358 user-generated comments, annotated into Neutral language (56%), Offensive language (29%) and Hate speech (15%). We trained BiLSTM, LSTM, CNN, SVM, and Random Forest models and fine-tuned Multilingual BERT, LitLat BERT, Electra, RWKV, ChatGPT, LT-Llama-2, and Gemma-2 models. Additionally, we pre-trained Electra for Lithuanian. Models were evaluated using accuracy and weighted F1-score. On the imbalanced dataset, LitLat BERT (0.76 weighted F1-score) and Multilingual BERT (0.73 weighted F1-score) performed best. Over-sampling further boosted weighted F1-scores, with Multilingual BERT (0.85) and LitLat BERT (0.84) outperforming other models. Over-sampling combined with augmentation provided the best overall results. Under-sampling led to performance declines and was less effective. Finally, fine-tuning LLMs improved their accuracy which highlighted the importance of fine-tuning for more specialized NLP tasks.

## introduction online hate speech poses significant challenges as social media platforms, forums other online spaces often contain hateful content that can incite violence @xcite , reinforce discrimination @xcite and contribute to social polarisation @xcite . manually identifying and removing such content is timeconsuming, costly and often inconsistent due to the high volume of online interactions @xcite . also, context @xcite , intent, and linguistic nuances @xcite further complicate the detection of hate speech, making it difficult to distinguish between harmful and non-harmful speech. machine learning and deep learning models can analyze large volumes of text, identify patterns associated with hate speech and improve via datadriven learning @xcite . this study evaluates traditional machine learning algorithms, deep learning architectures and large language models (llms) for the detection of hate speech in lithuanian in a scenario, where data and computational resources were limited. we experimented with different dataset configurations to assess how these models handle class imbalance issue which often occurs in many nlp tasks, including hate speech detection @xcite . therefore, we analyzed the impact of data augmentation and resampling techniques on model performance to get insights into how to improve the detection of hate speech in lithuanian, despite the issue of data imbalance. also, existing hate speech research and resources focus on english or similar large languages and represent other languages much less @xcite , which motivated our work as well. thus the rest of the paper is structured as follows: section 2 in brief describes related work, section 3 presents our data, section 4 introduces models we used in our experiments, section 3 specifies experimental setup, section 6 reports results and section 7 ends the paper with conclusions. ## related work hate speech is characterized by language that targets individuals or groups based on attributes such as gender, race, ethnicity, and disability @xcite . the spread of hate speech is often encouraged by societal biases and conflicts @xcite . despite extensive research, defining hate speech remains challenging as it is dynamic and includes a broad range of concepts, such as incitement, impoliteness, stigmatization, and cyberbullying @xcite @xcite . the interpretation of hate speech is further complicated by cultural, political, and legal differences across different regions, which makes it highly contextdependent @xcite @xcite . popular hate speech detection approaches rely on machine learning and nlp techniques @xcite . while traditional methods like logistic regression are still used (e.g., as in rohith and amanullah (2024)), deep learning architectures, such as long short-term memory (lstm), have demonstrated high accuracy @xcite . hybrid models that integrate machine learning and deep learning techniques have further improved detection in successfully capturing lexical and contextual features of hate speech @xcite . graph neural networks (gnns) have gained attention due to their improved contextual understanding of hate speech @xcite . also, it has been shown that additional contextual information relevant to hate speech detection can be obtained from related nlp tasks, such as sentiment analysis and emotion classification @xcite @xcite . moreover, optimizing self-supervised and supervised learning techniques have been used for improving model accuracy in dual contrastive learning frameworks @xcite . automated hate speech detection is challenging due to its dynamic nature @xcite . therefore, an increasing number of studies focus on multilingual hate speech detection, particularly in low-resource settings @xcite @xcite . cross-lingual transfer learning has been employed to leverage high-resource languages, such as english, for improving detection in languages with limited annotated data @xcite @xcite . furthermore, zeroshot transfer learning has shown promise in handling linguistic variations without requiring labeled data @xcite @xcite . additionally, metalearning frameworks such as hatemaml have been proposed to enhance cross-lingual transfer performance in low-resource settings @xcite , to name just a few. beyond cross-lingual strategies, data-efficient learning techniques have been explored to improve detection, e.g., even minimal fine-tuning data in the target language can significantly improve classification accuracy @xcite . also, data augmentation @xcite , oversampling @xcite and resampling @xcite , ensemble learning @xcite @xcite , cost-sensitive learning @xcite , etc. strategies were applied for solving class-imbalance problem. besides, the use of additional datasets has improved model performance in bilingual hate speech detection @xcite . also, privacy-preserving techniques, such as federated learning, have been included in hate speech detection models to protect user data @xcite . given the prevalence of multimodal hate speech, integrating visual and textual features is also relevant for comprehensive moderation @xcite . for example, multimodal approaches that incorporated textual and visual elements have been introduced for analyzing hate speech in memes @xcite . as deep learning models often function as "black boxes", it raises concerns about their interpretability and decision-making process. therefore, efforts to improve transparency and to increase trust in automated moderation systems have been made, as in @xcite and @xcite . research on lithuanian hate speech detection is still developing. it has a strong focus on linguistic analysis, such as analyzing the features of abusive and hateful comments in lithuanian news @xcite or discussing the definition of hate based on its usage in texts @xcite . also, first attempts in developing a corpus for hate speech detection in lithuanian has been reported in @xcite . in addition, there is some initial research on the application of deep learning models for automatic hate speech detection in lithuanian, which is reported in @xcite . despite all advancements, challenges persist in validation, such as bias in training datasets and model overfitting. therefore, frameworks such as hatecheck have been introduced to improve evaluation across linguistic and contextual settings @xcite . considering the continuous changes in hate discourse, refining detection methodologies, including scenarios and languages with limited resources, retains its importance. ## data to develop a hate speech detection solution for the lithuanian language, an initial dataset of approximately 60,000 comments was collected from various lithuanian news portals, including 15min.lt, alkas.lt, and delfi.lt. additionally, the dataset was supplemented with 226,776 comments from news portal lrytas.lt foot_0 and manually collected hate speech comments from various social media pages and news portals foot_1 . the sources were selected based on their popularity, accessibility of user-generated comments and differences in their audiences (e.g., alkas.lt tends to express nationalist perspectives.) recent comments were gathered according to the specific topics (lgbt+, gender issues, immigrants, etc.) that were the most prevalent in the hate speech data. these themes were extracted on the basis of an initial quantitative and qualitative analysis to ensure that contemporary patterns of hate speech, such as the latest vocabulary, were covered. ## models for hate speech detection in lithuanian, we employed well-known deep learning models, including multilingual bert, litlat bert, electra, rwkv, chatgpt, lt-llama-2, gemma-2, bil-stm, lstm, cnn, as well as traditional machine learning models such as svm, and random forest. lithuanian is considered a lesser-resourced language, therefore still not many pre-trained models can adequately process lithuanian texts. of these models, bilstm, lstm, cnn, svm, and random forest were trained for hate speech detection in lithuanian, while multilingual bert, lit-lat bert, electra, rwkv, chatgpt, lt-llama-2, gemma-2 were further fine-tuned to classify lithuanian user-generated comments, identifying those that may contain hate speech. also, we pretrained electra model for lithuanian from scratch. multilingual bert. this model foot_5 uses the architecture of the bert model and is trained on 104 languages, including lithuanian @xcite . wikipedia foot_6 texts were used to train this model. litlat bert. it is a trilingual model foot_7 that was built using the xlm-roberta-base (zhao and tao, 2021) and trained on lithuanian, latvian, and english language data. electra transformer. it is a transformer model foot_8 that uses a pre-training method which trains two neural network models: a generator and a discriminator. this proposed training method is significantly more efficient than the masked training method used in bert models. this is why the electra model requires fewer data and computer resources for training @xcite . as there was no pre-trained electra for lithuanian, we pre-trained it ourselves. rwkv. it is a language model that combines transformers and recurrent neural networks (rnns) @xcite . rwkv can effectively use past context while avoiding some common challenges of traditional rnns, such as struggling to handle long sequences. however, one of its main weaknesses is its sensitivity to how information is presented, i.e., reordering words in a prompt can significantly impact its performance. ## experimental setup experimental setup includes dataset configurations model training and evaluation as well as a separate llms testing procedure. the code used for these experiments as well as model training parameters are available at https://github.com/card-ai/ lt-hatespeech-models . ## results this section presents the evaluation results of different hate speech detection models trained and tested on lithuanian data. the models were assessed across 3 dataset variants: the original imbalanced dataset, a dataset balanced with over-sampling and a dataset balanced with under-sampling. additionally, an augmented dataset was evaluated using the same data balancing strategies to determine whether it could improve model performance. we used accuracy and weighted f1-score as the primary evaluation metrics. ## conclusions this study evaluates traditional machine learning algorithms, deep learning architectures and llms in detecting hate speech in lithuanian in a scenario where data and computational resources were limited. we experimented with different dataset configurations to assess how these models handle class imbalance issue. therefore, we analyzed the impact of data augmentation via lexical substitution and resampling techniques on model performance. transformer-based models, particularly litlat bert and electra, demonstrated the best performance on the imbalanced original dataset, achieving 0.76 and 0.73 weighted f-1 scores, respectively. traditional machine learning models like svm and random forest performed moderately, while rwkv struggled (0.49 weighted f-1 score). data augmentation improved all models, with litlat bert reaching 0.83 weighted f-1 score. with an over-sampling, accuracy improved further. multilingual bert (0.85 weighted f-1 score) and litlat bert (0.84 weighted f-1 score) outperformed all other models. electra improved to 0.76 weighted f-1 score, while rwkv remained the weakest. traditional models like svm (0.62 weighted f-1 score) and random forest (0.54 weighted f-1 score) suffered from potential overfitting. over-sampling combined with augmentation provided the best results, particularly for transformer-based models. with an under-sampling, performance declined. litlat bert (0.74 weighted f-1 score) and electra (0.69 weighted f-1 score) still performed well, but svm (0.62 weighted f-1 score) and random forest (0.52 weighted f-1 score) dropped significantly. rwkv struggled with 0.41 weighted f-1 score, though augmentation slightly improved it. undersampling was less effective than over-sampling for performance gains. gpt-4 models performed well enough without fine-tuning but at higher computational and financial cost. fine-tuning smaller llms significantly boosted their accuracy. fine-tuned lt-llama-2-7b achieved 0.75 weighted f-1 score, while gemma-2-2b-it improved from 0.46 to 0.55 weighted f-1 score. however, lt-llama-2-13b-q8 (0.16 accuracy) showed that pre-trained models without finetuning perform poorly on specialized tasks. our future plans include increasing our dataset as well as exploring larger variety of models and fine-tuning techniques. we also plan to examine model biases and decision-making processes.