LLM generated responses to mitigate the impact of hate speech (2024)

Abstract: In this study, we explore the use of Large Language Models (LLMs) to counteract hate speech. We conducted the first real-life A/B test assessing the effectiveness of LLM-generated counter-speech. During the experiment, we posted 753 automatically generated responses aimed at reducing user engagement under tweets that contained hate speech toward Ukrainian refugees in Poland.Our work shows that interventions with LLM-generated responses significantly decrease user engagement, particularly for original tweets with at least ten views, reducing it by over 20%. This paper outlines the design of our automatic moderation system, proposes a simple metric for measuring user engagement and details the methodology of conducting such an experiment. We discuss the ethical considerations and challenges in deploying generative AI for discourse moderation.

## introduction the full-scale conflict in ukraine that started in february 2022 resulted in a humanitarian crisis and forced millions of people to leave their homes. in neighboring poland, the conflict, its financial aspects, and help towards refugees were heavily discussed on social media platforms such as x (previously twitter). in such a conversational environment, hate speech is used as a tool to reinforce the narratives on each side. it also contributes to the propagation of disinformation by distancing it from facts and highlighting the emotional aspect. due to its strong impact on attitudes and beliefs, hate speech requires neutralization, which should take place as soon as possible after publication. we define hate speech on the guidelines outlined in the resolution of the parliamentary assembly of the council of europe (coe) (parliamentary assembly of the council of europe, 2021). according to these, hate speech includes any form of expression that spreads, incites, promotes, or justifies hatred based on various forms of intolerance. this helps us identify behaviors that aim to dehumanize individuals, making them susceptible to stigmatization, discrimination, and violence. in this work, we outline the design of our automatic detection and response system, describe the experiment's methodology, analyze and provide commentary on the results, and share additional insights that may be valuable for future work in automatic moderation systems. furthermore, we discuss the ethical aspects of such interventions. notably, we define a new engagement metric that can be useful for counter-speech experiments. to our knowledge, this is the first experiment using a/b tests to demonstrate the effectiveness of counterspeech interventions on twitter. our primary contributions in this work include: llm generated responses generative models, especially gpt models @xcite , have proved to perform well in different generative tasks without any additional training or fine-tuning, just by giving a couple of examples in prompt @xcite . this technique of adapting llms for new tasks by demonstrating examples is referred to as few-shot learning or in-context learning @xcite . @xcite show @xcite that by using in-context learning, the gpt-3 (brown et al., 2020) model can generate highquality, informative, and persuasive explanations of why the given text is hateful. authors identify the potential of gpt-3 as a valuable tool for combating hate speech. on the other hand, the gpt models can generate biased @xcite or hallucinated @xcite content, and the generated texts should be approached with caution. the usage of gpt models for content moderation online has already been researched, for ex-ample, by axelsen et al. @xcite . the authors of this work utilize gpt-3.5 models to identify and report toxic content and to reward positive contributions in messages among people. the method proposed in our work goes further by generating the responses that directly fight toxic content. russo et al. @xcite show that with a source of knowledge and correct prompt, llms indeed can generate countering, emotional responses to fake, misleading, or manipulative content online. in a related study conducted concurrently with our experiment, a similar study generating counterspeech using retrieval-augmented generation was conducted @xcite . however, contrary to our paper, neither of these performs live experiments that verify the efficacy of using llms to change the discourse in real social media. counter-speech counter-speech has been highlighted by platforms like facebook as an effective long-term strategy to combat hate speech. facebook publicly stated that counter-speech is a critical tool that can be more impactful over time compared to simply removing offensive content @xcite . several studies have examined the nature of counter-speech, like linguistics aspects @xcite , hate and counter users characteristics @xcite , and strategies @xcite . despite these findings, there is limited research on how counter-speech influences network dynamics over time. we found only one study that addressed this gap by showing that counter-speech can reduce the likelihood of neighboring users adopting hate speech, thus dampening the spread of negativity within social networks @xcite . ## dataset tweets collection to obtain tweets, we used the twitter v2 api with a broad query focused on the russo-ukrainian war. from march 3, 2022, to january 24, 2023, we collected approximately 2 million tweets. however, only a small fraction discussed our specific topic -assistance to ukraine. to address this, we refined our query to focus on humanitarian aid, financial assistance, and refugee rights, resulting in an additional 215,000 relevant tweets. in total, we compiled approximately 2,215,000 tweets, referred to as the twitter dataset. all tweets are in polish and filtered using twitter's language parameter. details of both queries can be seen in appendix a. ## hate classifier model the hate classifier is designed to detect specific types of hate speech against aid for ukraine, as defined in our dataset. given the iterative process described in the dataset section, this model leverages a curated set of tweets. we employed two primary approaches for classification: we fine-tuned an open-source roberta model pre-trained on a polish dataset foot_0 @xcite . this method used transfer learning to adapt the model to our dataset. the results of the models can be found in ## retrieval-augmented generation to generate factual and reliable responses to harmful tweets, we implemented a retrieval-augmented generation (rag) approach @xcite . this method enhances the model's responses by incorporating relevant, verified information from external sources. vector embeddings of articles as described in section 3, we maintained a database of 23 verified articles from reputable sources which were embedded using the text-embedding-ada-002 model. when a harmful tweet was detected, we calculated its relevance to each article in our database using cosine similarity. the top three relevant articles were selected and included in the gpt prompt, ensuring that the responses referenced specific, verified information. additionally, few-shot learning was utilized to achieve higher quality @xcite . ## experiment the main hypothesis of this study is that automatically generated counter-speech can effectively decrease user engagement with harmful tweets, making them less convincing to users. we introduced a novel engagement metric to validate this hypothesis and conducted an a/b test to determine its effectiveness. we aimed to assess whether the mean engagement value is significantly lower in the experimental group (with model intervention) compared to the control group. ## results detected harmful tweets can be the roots of discussions (the original tweet) or replies to another existing tweet (not the original tweet). in the next sections, we will validate our hypotheses in both groups as the results differ. engagement change using engagement metric from section 6, we measure the engagement of tweet i as follows: to validate our hypothesis, we aim to verify if the mean e value is significantly lower in the experimental group compared to the control group. furthermore, to ensure accurate calculation of the e value, we must select the minimum value for the ∆impressions(i). we evaluated the impact of selecting minimum thresholds of 10 and 100 impressions. these four sets are visualized in figure 3 . we excluded tweets with anomalies, where the final number of likes was lower than the initial count (when someone unliked the post during the experiment period, representing 1% of the collected data). we observe that in three of these four scenarios, the mean engagement is lower in the experimental group the most significant change can be seen when we reply to a harmful, original tweet with a minimum of 10 impressions in the monitoring period -in this group, we reduce the engagement by 23%. to test the significance, we use the p-value obtained with the t-test. also, we used the bootstrapping @xcite technique to calculate the p-value. to do that, we sampled (with replacement) 10 4 data samples from the control group and compared the distribution of their means with our experimental mean. welch's t-test @xcite gives a p-@xmath0.05, and the bootstrapped p-value @xmath1.01, making the result statistically significant. it is worth noting that the lower engagement coefficient can also be a result of an increased number of impressions. we want to reduce the engagement change in the number of replies as a result of further analysis of our data, we discovered that the ratio of the number of replies to a harmful tweet to its impressions is greater among the tweets that where analogously, replies(i) and impressions(i) is the number of replies and impressions to the tweet i after detection and replies'(i) and impressions'(i) are the number of replies and impressions at the end of the monitoring. this statistic does not count our bots' replies, so we measure only the change in other users' replies. we compare mean ratios in the experimental and the control groups by selecting our tweets with respect to minimum impression change and whether a harmful tweet was a reply or not. we also exclude tweets for which the number of replies decreased over a period of the monitoring (users deleted their responses) -it is about 0.3% of collected data. in each of the four scenarios, the mean ratio increased. the increase is greater for tweets with a minimum impression change of 10 than for tweets with a minimum impression change of 100. the greatest increase, by 387%, is observed when selecting non-original tweets with a minimum impression change of 10. we also tested the significance using bootstrapping and welsch's t-test with the null hypothesis that the mean of the analyzed ratio is equal among groups. the results are presented in the verified links and their impact let us now explore other interesting observations regarding the verified links in the tweets we found after the experiment. the results may serve as an insight for future automatic response generation systems. our model was prompted to support its responses with links from our base of verified sources whenever they applied to the subject of a tweet it responded to. of the 753 responses posted during our experiment, 398 contained a link to one. most of those responses consisted of a short statement pointing to a fact contradicting the original post and encouraged to check the source provided through a link at the end of the response. one interesting observation is that when we answered original tweets (non-replies) with a response containing a link to a source, the increase of replies to the tweet was 24% higher than in the case of responses that were not backed by a source. we observed this on the set of tweets with an impression change of at least 10. these results have a t-test p-@xmath2.02 and bootstrap p-@xmath3.001, which makes this finding even more interesting. it could be hypothesized that this result has one of many causes: our method of preparing the knowledge base, our data sources considered controversial by x users, or distribution differences between tweets for which our bot posted a link versus tweets for which our bot didn't have a verified knowledge source. for example, tweets discussing financial aspects could have received more replies, and our bot posted more links when intervening with them. this could be a basis for further study. another observation we made was the lack of impact of links to sources on the changes to engagement (likes over impressions). one of our goals when providing links was to reduce user engagement in threads debunked by a verified source. harmful tweets for which our bot replied with a link had a similar engagement ratio during the experiment as those for which our bot replied without a link. as before, this could have many causes and needs further research to give definitive results, but we do not conclude that verified knowledge helps reduce the popularity of harmful content. this paper proposes the first automatic system based on the openai gpt model for detecting hate speech on social media and counteracting it by posting automatically generated responses. the proposed system utilizes verified knowledge through fact-checked articles that the generative model can incorporate into intervention replies. we performed a live experiment with a proposed engagement metric that shows how counter-speech could be effective against hate speech. we showed that harmful tweets that are not a reply and have gained at least 10 impressions during our experiment exhibit a 23% lower engagement rate (measured as likes over impressions) in the experimental group compared to the control group. this statistically significant effect is not observed for tweets that are replies, which typically receive fewer views than the original ones, or for tweets that have become very popular (gaining at least 100 impressions), where the dynamics of the engagement rate are harder to alter due to the presence of numerous other replies, which may dilute the impact of our reply. this suggests that automatic moderation systems such as ours perform better when engaged at the highest level of discussion and primarily with moderate popularity tweets, highlighting our interventions' nuanced effectiveness. as an interesting result of our analysis, we present an effect of our replies on the ratio of change in replies to a harmful tweet to change in its impressions. we conclude that our replies increase this ratio, and we observe the greatest change for non-original tweets with a minimum impression count 10. these results may be explained by a social proof phenomenon, an idea that people are more likely to act if they see that others have already done so @xcite . we verified that at least 39% of our replies to an original tweet with a minimum impression change of 10, and at least 48% of our replies to a nonoriginal tweet with a minimum impression change of 10 were the first reply to the tweet, thus initiating a discussion. it's worth noting that despite these discussions, the number of impressions or popularity (measured as likes/impressions) does not increase. this may suggest that many additional responses come from the author or people already following the topic or discussion. however, we leave this analysis as part of further research. we analyzed the impact of verified links on changes our responses made to the discussion and made a statistically significant observation of a 24% higher increase in the response-to-impression ratio in responses supported by a source compared to those without one. this analysis was purely exploratory, so no concrete conclusions can be derived. however, we believe that further research on it could provide compelling results. we also discussed the ethical aspects of such experiments and automated moderation systems. we are convinced that human supervision during the whole experiment, transparent information about our account being a bot, and anonymization of collected data ensure that the experiment was conducted in an informed and ethical manner. we believe this work can help develop better tools and systems for building safer online communities. ## limitations the method and experiments presented in this paper have several limitations. firstly, the experiment was conducted exclusively in the polish language and focused solely on the war in ukraine. while the llm models used are multilingual and have demonstrated strong generalization capabilities, the behavior of social media users and experimental outcomes may vary significantly across different languages, communities, and discussion topics. therefore, the generalizability of our results remains uncertain, and further research is necessary to evaluate the method's effectiveness in different linguistic and cultural contexts, social media platforms, and discussion topics. secondly, our method's reliance on human supervision is a significant limitation. fully automated content moderation systems are prone to errors and can potentially spread misinformation. in this study, we did not assess our method's performance in a fully automated scenario, which limits our understanding of its applicability and reliability in real-world, unsupervised settings. additionally, our model only processes textual data, ignoring tweets with graphics, which are common in disinformation and hate speech. integrating vision models, such as gpt-4o, which supports vision input, could enhance the capability to detect and respond to such multimedia content. furthermore, the engagement metric used in this study may not be suitable for popular accounts with large followings, as their ability to generate views could skew the metric, making it less effective for assessing the impact of counter-speech interventions. additionally, the popularity of a counter-speech profile could attract trolls who may counteract efforts by increasing the visibility of harmful posts. this underscores the importance of robust countermeasures and continuous monitoring to mitigate such risks. finally, the limitations of the detection and answering model, including potential misclassifications and hallucinations, pose significant ethical challenges. these errors can lead to confusion, frustration, and the spread of misinformation. human oversight remains necessary to ensure automated responses' accuracy and appropriateness.