JRC at C limate A ctivism 2024: Lexicon-based Detection of Hate Speech (2024)

Abstract: In this paper we describe the participation of the JRC team in the Sub-task A: “Hate Speech Detection” in the Shared task on Hate Speech and Stance Detection during Climate Activism at the CASE 2024 workshop. Our system is purely lexicon (keyword) based and does not use any statistical classifier. The system ranked 18 out of 22 participants with F1 of 0.83, only one point below a system, based on LLM. Our system also obtained one the highest achieved precision scores among all participating algo- rithms.

## introduction in this paper we report on the participation of the joint research centre team at subtask a: hate speech detection in the shared task stance and hate event detection in tweets related to climate activism at case 2024 @xcite using a simple lexicon -based hate speech detection approach. over the past few years, the convergence of nlp and sociopolitical discourse has led to the development of diverse technologies such as hate speech detection, sentiment analysis, and other opinion detection technologies. at the same time, climate activism has taken a momentum on the social web and has captured the attention of nlp researcher community working in these areas @xcite . as the public discussions in this topic proliferated, the escalation of hate speech started to raise concerns among users. within the climate change discourse, hate speech manifests as a concerning trend, often taking aim at specific entities such as climate activists, influential environmental and political organizations like greenpeace, and even entire governmental bodies responsible for environmental policies. the targeting extends beyond institutions to include environmental initiatives like fridaysforfuture @xcite , amplifying the scope of the issue. adding another layer to this complex scenario, there is a noteworthy phenomenon involving individuals who pretend allegiance to the climate activist cause. these people employ hate speech in a troll like manner as a weapon in defending their version of climate advocacy. this dual nature of hate speech within the climate change discourse unveils the intricate interplay between genuine concerns, political discontent, and the broader socio-political landscape. this highlights the need for nuanced approaches in addressing hate speech, considering its diverse sources and motivations within the context of environmental activism. in this picture, automatic hate speech detection is becoming important, keeping "clean" the space of the social platforms and preventing online users from exposure to extreme content and disinformation. on the other hand, hate speech shows also increase of the discontent and frustration towards certain topics and public personalities. it serves as an indicator of the significance of these issues and people and their public perception; it also plays a crucial role as a marker for a negative bias in the social discourse. in fact, in usa certain hate speech acts are given constitutional protection @xcite under the laws defending the freedom of speech. the purpose of our experiment was to put in comparison a keyword based system with the other shared task participants, which were expected to predominantly exploit machine learning methods. as the simplicity of our method suggests, our system achieved score only little above the average system performance, and ranked 18th out of 22 systems, with f1 score of 0.83. our score was 0.03 lower than the system in the middle of the ranking; our method scored f1 lower by 0.07 from the top ranked system. the experiment proved that why are powerful men so scared of greta thunberg? the fridaysforfuture movement and the idea that we'd all have the gall to conduct a climatestrike every friday frightens and infuriates plutocrats. how billionaires with greta thunberg uproot the system, important thread 2 read: mitigate or die! adaptation, even successful, to today's accelerating climate crisis is a deadly delusion for complacent inaction. possible @xmath0, attacking greta thunberg is the only way to get back into conversation again. in 50 years, no one will know who brendan o'neill was, but greta thunberg will still be known. first we destroy nature the rich keep getting richer the poor are increasing in numbers measly check in the mail when there's still hell to pay bills and pills then we destroy ourselves push back despite all this ## related work hate speech is a topic of debate among lawmakers @xcite and nlp experts (jahan and oussalah, 2023), @xcite . automatic hate speech detection has been predominantly approached as a binary text classification, using machine learning @xcite ; multilingual dimension has also been explored in previous works and shared tasks @xcite lexicon-based hate speech analysis has also been addressed in previous works @xcite , @xcite . according to ( macavaney et al., 2019) , keyword-based approaches offer elevated precision but suffer from insufficient recall due to challenges in resolving word sense ambiguity and handling figurative language. essentially, systems relying on keywords may overlook hateful content that doesn't employ explicit hate terms. in contrast, @xcite presents a lexicon-based approach that contradicts this assertion by demonstrating reasonably high levels of both precision and recall. hate speech detection is also strongly related to sentiment analysis and opinion mining, where lexicon-based approaches are still used: a comprehensive study of these techniques is presented in @xcite . ## dataset and task the purpose of the shared task on detecting hate speech during climate activism was identification of tweets discussing the climate change topic and containing hate speech. the tweets have been retrieved by a team of researchers from delhi tech university, virginia tech, and james cook university, australia. the retrieval and annotation are described in @xcite . the data collection process aimed at tweets posted between january 1, 2022, and december 30, 2022. the selection criteria involved hashtags such as #climatecrisis, #climatechange, #climateemergency, #climatetalk, #globalwarming, as well as activistoriented hashtags like #fridaysforfuture, #actonclimate, #climatestrike, #extinctionrebellion, #cli-matealliance, #climatejustice, #climateaction, etc. only tweets composed in the english language have been considered by the data collection team. in this way above 15,000 tweets have been collected, which were subsequently annotated for presence of hate speech, relevance to the climate change discourse, stance, the direction of hate speech, targets of hate speech, and humor. for our shared task, only three aspects were considered from this annotation: hate speech, target of the hate speech (who or what is targeted) and stance (does the tweet support, oppose or is neutral). given we participated in subtask a: hate speech detection, only the hate speech annotation (1 -presence of hate speech, 0 -absence) was considered. ## methodology in our approach we have used the liu and hu lexicon @xcite , which is ranked as a high performing sentiment analysis lexicon by several studies: it was evaluated on twitter data with information about people and other entities (al-shabi, 2020), as well as on product reviews @xcite . in both cases this lexicon has delivered very competitive results, with respect to other repositories of sentiment keywords. considering our shared task on climate activism, the above mentioned twitter based evaluation showed that the lexicon was relevant for the task. the hu and liu lexicon has been created by two researchers from the department of computer science of the university of illinois at chicago, minqing hu and bing liu. it is composed of two lists of words: 2006 positive keywords and 4783 negative ones. since the task targets hate speech, we have used only the list of negative words. experimenting on the training set, we have identified the minimal optimal number of keywords to appear in a tweet, so that it is considered to contain hate speech. our experiment showed that this minimal number is 4: every tweet with four or more negative words were labeled as containing hate speech. after manually inspecting the training set, we have also identified few entities which were strongly associated with hate speech inside the training and evaluation corpora (one of them "greta thunberg") and added them to the lexicon. ## results and discussion we have participated in sub task a, whose goal was to detect from the test set the tweets, containing hate speech. our system ranked 18 out of 22 participating systems, with f1 score of 0.83. (f1 was the official ranking criteria of this shared task). our score was 0.03 lower than the system in the middle of the ranking. we have obtained a score only 0.01 lower than the preceding in the ranking system, which used a large language model; moreover, our precision was among the highest ones. considering our accuracy, we ranked 13, which is caused by the high precision of the rule based approach and the prevalence of instances, belonging to the negative category (no hate speech). our accuracy was also higher than the accuracy of the established baselines for this task, reported in @xcite . moreover, refining the focus on tweets containing a high number of negative keywords proved to be an effective strategy for achieving high precision in hate speech detection. ## conclusions we introduced a lexicon-based system designed to identify hate speech in tweets related to climate change. despite its simplicity and orientation towards high precision, our system achieved accuracy above the baseline and f1 score comparable to some machine learning approaches. our lexiconbased method achieved one of the highest precision scores of 0.92. however, it ranked in the low part of the leaderboard, primarily attributed to its notably low recall of 0.777. this was due to the simplicity of our approach with respect to other lexicon based works. we invested relatively little time in its development, which did not allow us to exploit the full potential of this class of methods.