Finding Authentic Counterhate Arguments: A Case Study with Public Figures (2023)

Abstract: We explore authentic counterhate arguments for online hateful content toward individuals. Previous efforts are limited to counterhate to fight against hateful content toward groups. Thus, we present a corpus of 54,816 hateful tweet-paragraph pairs, where the paragraphs are candidate counterhate arguments. The counterhate arguments are retrieved from 2,500 online articles from multiple sources. We propose a methodology that assures the authenticity of the counter argument and its specificity to the individual of interest. We show that finding arguments in online articles is an efficient alternative to counterhate generation approaches that may hallucinate unsupported arguments. We also present linguistic insights on the language used in counterhate arguments. Experimental results show promising results. It is more challenging, however, to identify counterhate arguments for hateful content toward individuals not included in the training set.

## introduction social media platforms such as twitter and facebook are used by many adults- @xcite report that 72% of u.s. adults use at least one social media site. content in social media can spread fast and help community organizations. for example, the first report of the 2017 las vegas shooting was a tweet posted ten minutes after the event started @xcite . more recently, twitter and hashtags have been credited with facilitating the organization of protests and the social movement that followed the killing of george floyd @xcite . social media has several positive effects including the ability to share information at low cost and being a vehicle for free speech where anyone can express their views. it also has negative effects including the spread of misinformation and synthetic counterhate argument (generic, generated on demand by experts or automatically): figure 1 : hateful tweet (top) and three replies with counterhate. previous work targets synthetic counterhate arguments that tend to be generic. in this paper, we find authentic counterhate arguments that address the hateful claims in the tweet at hand. hate. hateful content in social media is widespread. in 2017, a study conducted on 4,248 u.s. adults showed that (a) 41% were personally subjected to online hate speech and (b) 66% witnessed hate speech directed toward others @xcite . hateful content in social media has consequences in the real world. for example, there is evidence that several celebrities have committed suicide after being the target of hateful content in social media @xcite . additionally, an analysis of the japanese daily death registry and one million tweets between 2010 and 2014 revealed that there is a correlation between reports of celebrity suicides in social media and suicides committed by regular people @xcite . there have been many efforts to identify hateful content online (section 2). social media platforms have reported spending billion per year on these efforts. regardless of how hate is identified, one could keep the content but flag it, delete the content, or ban the author. an alternative compatible with freedom of speech is to counter the hateful content, a strategy shown to be effective at minimizing the spread and limiting the consequences of hateful content @xcite . consider the hateful tweet and three counterhate arguments in figure 1 . previous work, discussed in section 2, has targeted synthetic counterhate arguments that (a) are generated on demand by experts @xcite or (b) condemn the language but do not address the specific hateful claims and as a result generate generic counterhate arguments (e.g., "use of this language is not tolerated and it is uncalled for," @xcite ). in this paper, we instead target authentic counterhate arguments that address the specific hateful claims, as exemplified in the bottom two arguments in figure 1. we are inspired by previous work outside of the counterhate domain pointing out that (a) urls increase perceived trust @xcite and (b) good arguments have to appeal to logic by including factual, testimonial, or statistical evidence @xcite . therefore, authentic counterhate arguments with sources have the potential to be more effective than generic statements condemning hate. the work presented here works with hateful tweets toward public figures and authentic counterhate arguments published online. a hateful tweet is, according to the twitter guidelines, 1 any implicit or explicit tweet that attacks an individual's gender, religion, race, ideology, or social class. our definition of authentic counterhate argument borrows from previous work on counterhate @xcite and argumentation @xcite . we define counterhate as a direct response that counters hate speech and an authentic argument as a paragraph that appeals to logic by including factual, testimonial, or statistical evidence. in the remainder of this paper, we may refer to authentic counterhate argument as counterhate to save space. the work presented here could be understood as a specialized information retrieval task where the goal is to identify authentic counterhate arguments against a hateful tweet in (a) online articles and (b) paragraphs within the articles. the main contributions are: foot_1 • a corpus of 250 hateful tweets toward 50 public figures (5 each) and 2,500 candidate articles for authentic counterhate arguments; • annotations indicating whether the 54,816 paragraphs in the 2,500 candidate articles are authentic counterhate arguments; • analysis characterizing the language used to make authentic counterhate arguments; • experimental results showing that the task can be partially automated; and • qualitative analysis describing when the task is the hardest to automate. ## previous work identifying hateful content in user-generated content has received substantial attention in recent years @xcite . researchers have presented several datasets and models for hate detection in twitter @xcite , yahoo @xcite @xcite , facebook @xcite , gab @xcite , and reddit @xcite . previous efforts have also worked on identifying the target of hate (e.g., a group, an individual, an event or object). for example, @xcite differentiate between groups and individuals, and between immigrant women and other groups. @xcite , differentiate between groups, individuals, and others (e.g., an event or object). similarly, @xcite detect hateful content in several languages and identify the target from a set of 15 groups. countering hate previous work on countering hate can be divided into two groups: detection and generation. counter hate detection consists in identifying whether a piece of text counters hateful content. for example, @xcite identifies counterhate in the replies to hateful tweets. they rely in a simple pattern to detect hateful tweets: i @xmath0. garland et al. (2020) work with german tweets from two well-known groups and define hate and counterhate based on the group the authors belong to. he et al. (2021) work with tweets related to covid-19 and identify hate and counterhate using 42 keywords and hashtags. generating counterhate is arguably more challenging than detecting it. qian et al. (2019) crowd-source a collection of counter hate interventions and present models to generate them. similarly, @xcite direct their focus to other groups including jews, lgbtq+, and migrants. @xcite present conan, a collection of anti-muslim hateful content and counterhate replies. both the hateful content and counterhate replies in conan were written by expert operators on demand; thus, it is unclear whether this synthetic dataset would transfer to content written by regular people. more related to our work, @xcite use external knowledge and gpt-2 to generate counterhate arguments against specific hateful claims, similar to what we define here as authentic counterhate. generating counterhate is certainly a valid strategy, but gpt-2 and other large pretrained models are known for hallucinating facts. in this paper, we bypass this issue and instead retrieve authentic counterhate arguments from online articles. we find counterhate arguments for 72% of hateful content we consider, making this strategy viable. further, all previous efforts are limited to counterhate to fight against hateful content toward groups. unlike them, we explore counterhate for hateful content toward individuals. ## a collection of hateful tweets and authentic counterhate arguments we start our study creating a corpus of hateful tweets and authentic counterhate arguments against the hateful content. we focus on hateful tweets toward 50 public figures which we will refer to as individuals. the supplementary materials list the 50 individuals; we collected their names from online sources. 3 the list of 50 individuals is not gender-balanced (male: 30, female: 20) but includes people working in professions such as politician, comedian, singer, and athlete. it also includes, for example, politicians with opposing views including joe biden and donald trump. appendix a details the 50 individuals. ## analyzing counterhate arguments ## experiments and results we experiment with classifiers to determine (a) whether a paragraph is an authentic counterhate argument against a hateful tweet and (b) whether an article contains a paragraph that is an authentic counterhate argument against a hateful tweet. all the neural classifiers are built using pretrained language models from huggingface @xcite and pytorch @xcite . we tune hyperparameters with the training and development splits, and report results with the test split. we first report results with a random 70/10/20 split. then, we explore a more realistic-and challengingscenario: using in the test set candidate counterhate arguments for hateful tweets toward unseen individuals during training (section 5.2). baselines we also present results with two baselines: majority and random labels. the majority label is no for both paragraphs and articles. paragraph-level neural classifier we experiment with neural classifiers built on top of a bert-based transformer. specifically, we use the roberta transformer @xcite , which is pretrained with 800m words from the bookscorpus @xcite , english wikipedia (2,500m words), cc-news (63m english news articles) @xcite , openwebtext @xcite @xcite . the neural architecture consists of roberta, a fully connected layer with 128 neurons and relu activation, and another fully connected layer with 2 neurons and a softmax activation which outputs the prediction (no or yes). we tried several input choices (individually and combinations): in order to feed to the network with different combinations of inputs, we concatenate them with the separator special @xmath1. article-level neural classifier we experiment with the same architecture as the paragraph-level classifier, however, we use the longformer transformer @xcite instead of roberta. the longformer can handle longer input texts. we also consider several inputs: we also experiment with a system that aggregates the paragraph-level predictions. this system reuses the neural classifier at the paragraph level and outputs yes if any of the paragraphs are identified to be an authentic counterhate argument. ## conclusions countering hate is effective at minimizing the spread of hateful content @xcite . unlike blocking content or banning users, countering hate does not interfere with freedom of speech. previous work (a) works with hateful content toward groups and (b) focuses on generic, synthetic counter arguments written on demand by experts or the crowd. in this paper, we focus on authentic counterhate arguments against hateful claims toward individuals. authentic arguments, unlike generic ones, address specific hateful claims and include factual, testimonial, or statistical evidence. we present a collection of 250 hateful tweets toward 50 individuals and candidate counterhate arguments. candidates come from 54,816 paragraphs from 2,500 online articles. our annotation effort shows that authentic counterhate arguments are rare (4.3%). experimental results show promising classification results. it is more challenging, however, to identify authentic counterhate arguments for a hateful claim toward an unseen individual.