A Community-Centric Perspective for Characterizing and Detecting Anti- A sian Violence-Provoking Speech (2024)

Abstract: Violence-provoking speech – speech that implicitly or explicitly promotes violence against the members of the targeted community, contributed to a massive surge in anti-Asian crimes during the COVID-19 pandemic. While previous works have characterized and built tools for detecting other forms of harmful speech, like fear speech and hate speech, our work takes a community-centric approach to studying anti-Asian violence-provoking speech. Using data from ~420k Twitter posts spanning a 3-year duration (January 1, 2020 to February 1, 2023), we develop a codebook to characterize anti-Asian violence-provoking speech and collect a community-crowdsourced dataset to facilitate its large-scale detection using state-of-the-art classifiers. We contrast the capabilities of natural language processing classifiers, ranging from BERT-based to LLM-based classifiers, in detecting violence-provoking speech with their capabilities to detect anti-Asian hateful speech. In contrast to prior work that has demonstrated the effectiveness of such classifiers in detecting hateful speech ( F_1 = 0.89), our work shows that accurate and reliable detection of violence-provoking speech is a challenging task ( F_1 = 0.69). We discuss the implications of our findings, particularly the need for proactive interventions to support Asian communities during public health crises.

## introduction online platforms struggle with various forms of information pollution, including but not limited to harmful speech and misinformation. these malicious phenomena often intertwine in complex ways, exacerbating real-world issues. a glaring example of this is the dramatic increase in hate crimes against asian communities during the covid-19 pandemic, which included physical assaults and verbal harassment @xcite . rumors about the virus's origins coalesced with pre-existing prejudices, resulting in narratives that portrayed asians as "uncivilized", blamed them for the virus, and labeled them as "spies". this intricate interplay between different forms of malicious content led to a 339% surge in anti-asian crimes in 2021 @xcite . such narratives have had a far-reaching impact, making community members afraid to engage in basic daily activities, from grocery shopping to using public transit @xcite . moreover, this increase in real-world attacks was not limited to one ethnic group but affected diverse asian communities, including chinese, korean, vietnamese, and filipino americans @xcite . in light of the dramatic uptick in anti-asian violence and ensuing community fear, the necessity to accurately identify violence-provoking speech -i.e., speech that could promote real-world violence against members of targeted communities @xcite , becomes paramount. this differs from hateful speech, which is a more subjective form of expression that may not directly incite violence @xcite . while both these phenomena share commonalities -being rooted in prejudice and derogation -violence-provoking speech constitutes a specific subset of hateful speech that explicitly or implicitly encourages acts of aggression. the higher severity of harm associated with violence-provoking speech @xcite calls for targeted approaches for its detection, beyond treating hate as a monolithic entity. recognizing that the perception of what qualifies as violence-provoking is not universal but varies among the targeted communities, we adopt a community-centric approach. we leverage the "insider perspectives" @xcite and the subjective lived experiences @xcite of community members to capture the nuances, slurs, and coded language that may be overlooked by outsiders. our focus is particularly on asian communities in the context of the covid-19 pandemic, who were disproportionately impacted by violence-provoking speech leading to real-world harm. we address two key research questions: rq1: what are the characteristics of violenceprovoking speech that targets asian communities? how is anti-asian violence-provoking speech different from anti-asian hateful speech? rq2: can state-of-the-art natural language processing (nlp) approaches accurately detect violence-provoking speech? how do the detection abilities compare to that of hate speech detection? we address these research questions by developing and validating a codebook for identifying anti-asian violence-provoking speech, while working with anti-defamation league, a leading nongovernmental organization that specializes in tackling real-world hate and extremism (rq1). we then use the codebook to obtain crowd-sourced annotations for violence-provoking and hateful content from individuals who self-identify as asian community members. our dataset demonstrates high inter-rater agreement (fleiss' κ = 0.66 for violenceprovoking speech labels). 1 we then use the annotated data to develop binary classifiers that are trained to distinguish (i) violence-provoking content from not-violenceprovoking content and (ii) hateful content from non-hateful content. we find that while the nlp approaches effectively detect hateful speech (f 1 @xmath0.89), it is relatively more challenging to detect violence-provoking speech (f 1 @xmath1.69) (rq2), perhaps due to its nuanced and subjective nature that relies on victims' own perceptions. we discuss possible reasons why detecting violence-provoking speech is challenging and the implications of lacking capabilities of the detectors. additionally, we discuss how our developed approaches could aid in the development of moderation algorithms that employ tiered penalties for content that violate norms of varying severities. finally, we highlight the need to develop traumainformed approaches to proactively support targeted communities. ## related work we categorize relevant prior work into three categories: (i) studies focusing on different forms of harmful speech on online platforms, including fear speech and hateful speech, (ii) studies of anti-asian content, and (iii) detection methods for different forms of harmful speech. forms of harmful speech: prior studies on harmful speech have mostly focused on instances of hate speech. @xcite show how hate speech is an often neglected driver for election violence. another study by @xcite has shown that online hate victimization is part of a wider process of harm that can begin on social media and then migrate to the physical world. another form of harmful speech is fear speech. according to @xcite , fear speech is defined as an "expression aimed at instilling (existential) fear of a target (ethnic or religious) group." while it cannot be pinpointed if fear speech can cause violence, it can lower the threshold for violence @xcite . @xcite study the prevalence of fear speech in a loosely moderated community (gab.com) and observe that users posting fear speech are more influential as compared to those who post hate speech. the authors argue that this is mainly due to the nontoxic and argumentative nature of the speech posts. violence-provoking speech is closely related to fear speech but is defined more specifically as speech that promotes violence. closest to the concept of violence-provoking speech, @xcite , define dangerous speech as an expression that has "a significant probability of catalyzing or amplifying violence by one group against another, given the circumstances in which they were made or disseminated." in this work, we contextualize the framework by @xcite for anti-asian violence-provoking speech by using a community-centric approach and aim to operationalize its large-scale detection using stateof-the-art classifiers. ## study overview our study involves collecting a large-scale twitter dataset that comprises anti-asian content related to the covid-19 pandemic. it is worth noting that given the harmful impact of violence-provoking speech, it is challenging to find examples that explicitly provoke violence, as they are frequently removed from the platform due to moderation efforts. however, instances of such speech still exist on major platforms. to overcome the challenge of collecting potentially violence-provoking speech examples from twitter, we designed an elaborate pipeline to obtain a dense sample, which we then used for obtaining annotations from the asian community members. as shown in figure 1 , there are 4 key parts to our study. the first part comprises data collection, where we collected about 420k twitter posts containing anti-asian keywords and covid-19related keywords. we then used posts in subsequent parts of our study to answer our rqs: • to facilitate the effective development of the anti-asian violence-provoking speech codebook (rq1), we prepared a set containing a reasonably substantial amount of potentially harmful content. the original ∼ 420k twitter posts comprise a variety of topics, such as hateful expressions, counter to hateful expressions, or instances of someone sharing anecdotes about anti-asian hate. however, since we are particularly interested in violence-provoking speech -speech that implicitly or explicitly promotes violence against the asian community, we further filtered down the collected ∼ 420k twitter posts to find a set of 4, 076 twitter posts that is denser in violence-provoking content. this concentrated sample is essential for the manual analysis required in codebook development, without the impractical inundation of non-violence-provoking posts that risk overburdening annotators in a task that involves reviewing potentially traumatizing content. in section 5, we describe how we used this subset of twitter posts to develop a codebook for anti-asian violence-provoking speech while emphasizing its distinction from hateful speech. ## anti-asian covid-19 data collection we started by collecting a large-scale dataset from twitter using covid-19 and anti-asian keywords. we used the twitter academic api to collect data from january 1, 2020 to february 1, 2023, by querying using covid-19 and anti-asian keywords together. to consider a wide range of data, we expanded the set of keywords used in prior work. ## development of anti-asian violence-provoking speech codebook the existing guidelines developed by the dangerous speech project @xcite provide a general framework for identifying violenceprovoking speech. we followed this guideline and expanded it to develop a comprehensive codebook that allows empirical measurement and categorization. we grounded the defined sub-concepts in real data from twitter and contextualized them in the community-targeting (i.e., anti-asian) framework. to enable effective development of the codebook by qualitatively coding the data, we started by obtaining a subset of the ∼ 420k tweets by filtering based on 'dangerous' keywords so that it is potentially concentrated in violence-provoking speech. preparing a concentrated sample for codebook development: to obtain dangerous keywords, we started with the example phrases mentioned in the practical guide provided by the dangerous speech project @xcite and expanded the phrases by computing similarity scores with the phrases in our dataset. for similarity computation, we used word2vec embeddings fine-tuned on our corpus. of the ∼ 420k twitter posts, 4, 076 twitter posts contained one or more dangerous phrases. we use this potentially rich sample in anti-asian violenceprovoking content for codebook development. developing violence-provoking speech codebook: our theory-guided, data-centric approach was inspired by previous work in understanding minority stress @xcite and credibility indicators @xcite . to operationalize the framework proposed by @xcite within the anti-asian hate context and for empirical tasks, three of the authors first engaged in a discussion concerning its applicability and the potential need for expansions or alterations. the discussion was informed by the authors' lived experiences, with some being members of the asian community. then, through an iterative process, the authors inductively developed categorical codes to characterize violence-provoking speech and deductively assessed the applicability. two authors independently coded 180 randomly sampled instances that contained dangerous phrases and drafted concepts with definitions and examples. with a third author involved, the discrepancies in annotations were discussed, and drafted categories were combined and revised. we tested this revised codebook with the sample until no new themes emerged, resulting in a unified and coherent document that reflected the perspectives and consensus of all contributors. the codebook was then reviewed by all co-authors before being validated by members of the asian community and experts in anti-defamation league (details in the following subsection). validating the codebook: we validated the effectiveness of the codebook through pilot testing with both internal and external annotators. we randomly sampled 100 tweets from the entire twitter dataset, and for each tweet, the annotators were asked if it contained violence-provoking speech or hateful content. we quantified the level of agreement between two annotators using fleiss' κ. internal test-ing was done by the two researchers who drafted initial codebooks, which demonstrated substantial agreement for both categories ( ## community-centric annotations with a violence-provoking-dense sample with the validated codebook, we aimed to get community-centric annotations for 1, 000 twitter posts that contained diverse examples and were also rich in violence-provoking content. to enable this, we trained a few-shot classification model to filter relevant posts, given the remarkable generalizability and accuracy of few-shot learning. prompt-based few-shot learning for creating a dense violence-provoking sample for community-centric annotation: from the twitter posts that were labeled while validating the codebook, we used a representative subset of labeled examples (a total of 64 examples, 32 were violence-provoking, and 32 were not) to train a binary classification model. more specifically, we used a prompt-based few-shot learning method called pattern exploiting training (pet) @xcite . this method converts the input data into cloze-style statements and finetunes a pre-trained language model to predict the missing token. we formulated the classification task using the following prompt: is "<input @xmath2., with the [mask] token serving as a placeholder that could be verbalized as "yes" or "no" during model training. we used the pre-trained deberta-v2-xxlarge language model @xcite as the backbone and fine-tuned it for 200 iterations using a batch size of 16. the rest of the hyper-parameters were set to the values used in prior work @xcite . all the language models in this work were trained on an nvidia quadro rtx 8000; the cumulative training time amounted to about 24 gpu hours. we then used the trained model to predict labels and prediction scores over the entire ∼ 420k twitter posts in our curated dataset. we then randomly selected 1000 twitter posts that had a classification score @xmath3.9. since these twitter posts have a high probability score of being violence-provoking, the set of 1000 twitter posts is likely to contain a higher density of violence-provoking posts than the entire set of ∼ 420k posts (labeling of 200 examples from the filtered set by the authors indicated that about 30% posts were violence-provoking). furthermore, the generalizability of the few-shot learning approach helped in identifying violence-provoking posts that do not necessarily contain dangerous keywords. for instance, "these are not people of god, they are virus-ridden chinks who do not deserve our goodness" does not contain any dangerous keywords in our list (see obtaining community-centric annotations: we designed a custom interface to collect data from a crowd-sourcing platform called prolific ( https: //www.prolific.co/ ). we recruited participants who identified as asians, were located in the united states or canada, spoke fluent english, and were at least 18 years old. the participants were compensated at an hourly rate of 10 usd per hour. to ensure high-quality annotations, we only considered participants who had at least 100 submissions and had a minimum of 95% approval rate across their past submissions. the overall cost of obtaining annotations was about 760 usd. our user-friendly interface required the eligible participants to read the consent form and acknowledge their consent and eligibility before being directed to the codebook that contained all the necessary definitions and examples (see the participants could revise their response to the example posts multiple times, and only after they had correctly answered the 5 examples were they directed to the annotations page. on the annotations page, the participants saw 25 posts, one by one, for which they provided answers to the questions above in yes/no. the annotations page also contained an abridged version of the codebook. the participants could discontinue the study at any point for any reason while still getting compensated for the time spent on the study. we obtained annotations from 3 different participants for each of the 1, 000 twitter posts. in all, we received annotations from 120 unique participants, with a median age of 31, among which 54% identified as males and the rest as females. the country of birth for the participants was distributed across the united states, canada, china, korea, philippines, japan, and vietnam. on average, a participant took 29 minutes to complete one session. data statistics: we assigned final labels for violence-provoking and hateful speech based on majority annotations. in other words, if at least 2 out of 3 annotators considered a twitter post to be either violence-provoking (or hateful), it was violence-provoking and hateful (@xmath4. fucking racist weak pigs • they had it all along trust me, can't trust the asian spies here. they helped spread the virus • fucking dog eaters, asian weasels don't have balls to fight like a man that's why the pandemic • clearly your a hater a democrate or maybe a communist like the communist chinese who launched lied threw a deadly virus at the world these chinks must pay for this evil jealous liars and there will be 4 moreee years trumpster.. 4 moreeee years trumpster • they don't care about schools or kids! these are satanic baby eatin soulless traitors who are at war with america on our soil. bet if you lined em chinks up and shot em all this corona would be over yesterday ## contrasting hateful & violence-provoking speech classifiers we now focus on developing state-of-the-art classifiers to assess their ability to detect violenceprovoking and hateful speech targeting asian communities (rq2). we evaluate and contrast both fine-tuned bert-based language models as well as large language models like mixtral @xcite for the following two tasks: (i) hateful and (ii) violence-provoking speech detection. hateful speech detection: we consider the 954 twitter posts identified as hateful in our community-annotated dataset as positive examples for this task. for negative examples, we augment the 44 'not hateful' examples in our dataset by adding the 'neutral' and 'counter-hate' twitter posts curated by @xcite , resulting in a total of 1861 'not hateful' examples. we then split the dataset into an 80 : 20 ratio to create the training and validation set. in ## discussion and conclusion we developed a comprehensive codebook to enable the conceptual identification of violence-provoking speech and distinguish it from more subjective hateful speech (rq1). we then used the codebook to obtain annotations from asian community members. the high inter-annotator agreement scores demonstrate the effectiveness of our codebook and the quality of the collected data. we then used the annotated data to train classifiers that can be used for detecting hateful and violence-provoking speech. we highlighted the lacking capabilities of nlp classifiers in effectively distinguishing violenceprovoking speech from hateful speech and conducted error analysis to aid future research (rq2). implications: we believe that the findings from our study can enable informed decision-making by different stakeholders: (a) policy-makers who are responsible for regulating harmful speech, including both hateful speech and violence-provoking speech, (b) practitioners who are responsible for algorithmic accountability of online platforms, and (c) the targeted asian community members. our work hints at the need for a tiered penalty system on online platforms that may allow for more nuanced and proportionate responses to varying types of harmful content, enhancing algorithmic accountability. tiered penalties are also more fair as they align penalties with the severity of the offense, thereby offering a balanced deterrent that could encourage more thoughtful online interaction. furthermore, our study underscores the importance of tailored, trauma-informed interventions @xcite to support targeted communities as there is a need to create more holistic and humane approach to protecting targeted communities.