A Novel Methodology for Developing Automatic Harassment Classifiers for T witter (2020)

Abstract: Most efforts at identifying abusive speech online rely on public corpora that have been scraped from websites using keyword-based queries or released by site or platform owners for research purposes. These are typically labeled by crowd-sourced annotators – not the targets of the abuse themselves. While this method of data collection supports fast development of machine learning classifiers, the models built on them often fail in the context of real-world harassment and abuse, which contain nuances less easily identified by non-targets. Here, we present a mixed-methods approach to create classifiers for abuse and harassment which leverages direct engagement with the target group in order to achieve high quality and ecological validity of data sets and labels, and to generate deeper insights into the key tactics of bad actors. We use women journalists’ experience on Twitter as an initial community of focus. We identify several structural mechanisms of abuse that we believe will generalize to other target communities.

## introduction harassment is a significant problem in online spaces. in 2017, one in four americans reported experiencing online harassment, with more than 60% describing it as a "major problem" @xcite . for journalists, a social media presence is essentially a professional requirement, as it is both a mechanism for locating sources and for promoting stories @xcite ; as of 2018, more americans (roughly 20%) get their news from social media than from printed newspapers @xcite . at the same time, journalists receive an inordinate volume of hateful and harassing messages via social media. in a recent survey conducted by the committee to protect journalists (cpj), 90% of american journalists described online harassment as the biggest threat facing journalists today, with women and minority journalists being disproportionately targeted online @xcite . this harassment can have devastating effects. in 2016, 10% of women journalists said that they had considered leaving the profession out of fear @xcite , while others avoided certain coverage areas in an effort to mitigate the risk of harassment. still others may choose not to enter the field at all. at a time when there is a major need to retain skilled journalists and diversify newsrooms @xcite , our goal is to develop a research methodology to address this critical threat facing journalists, and ultimately, our free press. our contributions in this paper include: a) identifying gaps in current anti-harassment tools provided by twitter; b) identifying key strategies used by harassers to circumvent these tools and reach their targets; and c) development of a direct-engagement research process and data collection platform to curate datasets with high ecological validity, which will ultimately be used to train better machine learning classifiers for harassment detection. ## motivation and approach currently, there are limited options available for journalists to deal with harassing messages on twitter. twitter has three primary mechanisms through which a user can control their interactions on the platform: muting, blocking, and the recently introduced "conversations" controls, all of which have a slightly different impact on the content a user can access. for example, muting and blocking can both prevent content from certain users from appearing in some user a's timeline (twitter, c) @xcite . however, muted users can still follow and interact with a, while blocked users are no longer able to see a's tweets, and if they visit a's profile, they will see they have been blocked @xcite . the new "conversations" feature, meanwhile, allows user a to specify whether everyone, everyone they follow, or only specific users can reply to a specific tweet @xcite . while these tools offer impressive granularity, many journalists have both large followings and a professional mandate to interact with their audiences on social media. this makes many of the available controls impractical or ineffective. moreover, two of the three tools twitter offers are only effective retrospectively, meaning the targeted user must still read blocked users' offensive tweets before they can choose to mute or block them. not only does this require journalists to experience harm in order to achieve any potential remediation, if they are targeted by a large number of accounts, the manual effort becomes time-prohibitive. shared blocklists have been touted as a means for addressing some of these issues @xcite . however, for journalists this can result in blocking users who may be sharing legitimate critiques of their work @xcite . as a whole, journalists as a community have expressed desire for more effective user engagement management tools @xcite . furthermore, while many social media platforms do already have automated mechanisms for filtering harassment and hate speech, these are largely based on keyword matching, requiring manual creation with no guarantee of accuracy. due to the large scale of problematic content on social media worldwide, manual efforts by moderators and filters have also been insufficient @xcite . the goal of this work is, therefore, to contribute a robust, generalizable mixed-methods approach to constructing harassment training datasets with strong ecological validity, in order to support the development of truly effective classifiers for proactively identifying real-world abusive, harassing, and demeaning speech towards specific communities on twitter. working with journalists, we are collecting a large-scale corpus of personally-harassing messages they have received on twitter, and have developed an easily-employed annotation method to label messages by degree of observed harassment. using this data, we then build machine learning classifiers to distinguish between hateful, abusive and neutral tweets. ultimately, we plan to integrate our trained models into a tool to help journalists navigate and avoid having to see these unwanted, harassing messages. ## related work prior work on automatic detection of hateful and abusive speech toward journalists is limited. in @xcite , researchers used a manually-validated seed set of journalism-related twitter accounts to generate a list of target accounts across five languages. using the twitter api to conduct keyword-based searches, they then manually annotated hate vs. non-hate tweets. this yielded highly imbalanced corpora, with more "hate" than "non-hate" tweets for each language. deep learning models trained on each language corpus achieved best macro-f1 scores over .80 for english, french and greek but somewhat lower for spanish and german. other work has addressed the more general problem of automatic identification of hate speech and abusive language online. in @xcite , researchers found that crowd-sourced annotations performed poorly. this indicates the importance of expert annotators, which @xcite situates specifically in terms of classifying harassment. in @xcite , researchers using data from yahoo and the american jewish congress found that anti-semitic hate speech differed linguistically from speech that targeted other religious or ethnic groups, highlighting the need for a community-specific approach to studying hate speech. @xcite used content from self-identified hate communities, instead of keywords from hand-coded speech or manually coded hate speech terms, as training data for their work on hate speech detection with some success. in @xcite , researchers studied abusive language in online user comments on news and finance forums using linguistic, syntactic, and distributed semantic features as well as lexicon-based features. their dataset has been used to benchmark performance in hate speech detection, as has @xcite . in @xcite , researchers developed deep learning models for hate speech detection on twitter, using transformed word embeddings to classify hate speech on three public datasets. researchers in journalism have also used more qualitative methods to study abusive and hateful speech towards journalists. for example, ut austin's school of journalism published results from in-depth interviews with 75 female journalists describing how rampant online sexual harassment disrupts their ability to do their jobs @xcite . the committee to protect journalists reported similar findings in 2019 @xcite . finally, we note that developers have created tools (e.g. twitter block chain @xcite and the recently discontinued block together (hoffman-andrews, 2020) and the forthcoming block party app @xcite ) specifically designed to address the manual nature of twitter's muting and blocking functions. while these efforts appear to address an important limitation of twitter's current systems, they remain a reactive, rather than proactive, approach. our proposed methodology for training data collection and annotation incorporates and improves on these approaches as follows: (1) we conduct background interviews with our target community of women journalists in order to identify common heuristics used to carry out harassment on twitter, in order to develop a more nuanced and balanced dataset for annotation; (2) annotations are performed by the targets of harassment, guaranteeing a unique level of ecological validity; (3) our approach takes an empowering rather than exploitative approach to the detection process, promoting harm reduction by allowing harassment targets to participate constructively in the creation of classifiers that can better support their needs. ## methodology we employ a mixed-methods approach that integrates qualitative and quantitative data collection and analysis. we begin by directly engaging with our target group of women journalists who have experienced online harassment. we recruit participants by circulating calls to participation in key networks of women journalists, followed by semistructured pilot interviews with select participants, in which we question them about patterns of harassment that they have experienced or observed, and about potential tools or interventions that would improve their experience on social media. despite our convenience sample, two key themes emerged across several pilot interviews, providing valuable insights about the mechanisms of harassment on twitter, which we describe in section 5. results of these interviews are then integrated into our quantitative data collection pipeline. using patterns of harassing language and behaviors on twitter described by interview participants, we develop computational methods to automatically identify those patterns and then use these methods to sample potentially hateful messages from participants' twitter archives for them to annotate. we describe this data selection process in section 6.1. through the process of direct engagement with our target community, we are able to curate a high quality dataset of labeled tweets to support the development of more robust harassment classifiers. ## pilot interviews to generate a well-balanced training set of tweets, we conducted pilot interviews with several women journalists who have faced significant harassment on twitter. through these interviews we learned about specific forms of the "sub-tweeting" and "snitch-tweeting" heuristics that are used to target these and other women journalists with abusive and harassing messages. the primary form of "sub-tweeting" described to us consists of perpetrators capturing screenshots that contain the target's twitter profile or username. they then tweet these out with implicit or explicit calls for their followers to tweet at the same target. this behavior constitutes "sub-tweeting" because the absence of the target's username in the text of the original tweet means that target will not be notified of the instigating tweet, and will therefore be caught off-guard by an influx of often abusive tweets, sometimes numbering in the thousands over a period of less than a day. (see @xcite for more details and examples of "sub-tweeting.") we note that none of twitter's currently available tools can mitigate this attack; even if the perpetrator has already been blocked by the target, they can simply log out of twitter and view the target's profile in a web browser in order to obtain the required media. while the effect of sub-tweeting is to mask the identity of the perpetrator, "snitch-tweeting" is a means of drawing the target into a sub-tweeted thread about themselves to expose them to abuse. because sub-tweeting intentionally circumvents twitter's notification systems, targets of abuse will typically be unaware of such sub-tweeting, unless, as described above, it is used to direct traffic to their account. "snitch-tweeting" consists of adding a target's handle to a thread about them, thus triggering a notification. the goal is for the target then to review the notification and thus to view the abusive thread that precedes the snitch-tweet. taken together, these results helped us inform our design for the tweet selection portion of our data pre-processing, as described below. ## platform design in order to curate a high-quality training dataset from participating journalists' tweets, we designed and implemented a two-part, web-based platform to facilitate the data collection and annotation processes. this web platform was designed to balance the proportion of abusive vs. non-abusive tweets that are presented for annotation, without relying on keywords, which are often too coarsegrained to serve as a reliable indicator of abusive content. instead, we develop heuristics using insights from our pilot interviews as well as private data from the participant's account to include a more nuanced and representative range of potentially abusive tweets for annotation. the platform is also designed to maximize the efficiency and accuracy of the annotation process, in order to generate a large volume of high-quality training data for deep learning models. we achieve this via batched contextual annotation: participants annotate tweets within the context of the original conversation or tweet thread, rather than annotating them in isolation, simulating how they would have viewed the conversation initially on twitter. in addition to the annotation tool described above, we have also built a tool for secure data upload, as described below. ## modeling while we are recruiting more journalists as study participants into our data collection pipeline, we have in parallel been building models of both feature engineering and neural network-based approaches, and testing them on historical hate speech datasets. we plan to take the insights we acquire from these experiments and apply them to classifiers built on our own data once we have accumulated a sufficient amount. we also plan to check the cross-performance between models trained on our own and historical corpora as quality assurance. the data which we have accumulated so far gives us a good idea of which historical corpora are most similar to our own. we explored several corpora, including @xcite and @xcite , but focused on task 5 of semeval 2019, "multilingual detection of hate speech against immigrants and women in twitter (hateval)" in english @xcite , as it is most recent and they are all of similar genre. both task 5 subtasks used the same dataset (cicl2018/hateevalteam, 2019) but with different labels. subtask a was a binary classification task to assign a label of "hate" or "non-hate" to each tweet. subtask b was a multi-class classification task to assign two additional label pairs to each tweet in addition to "hate" or "non-hate": "individual" or "group" and "aggressive" or "non-aggressive". the split across train and development datasets was 9000 to 1000 tweets; these have been open-sourced by the organizing team. the true labels for the test set have not, however, so we evaluate only on the development set. we replicated the winning approach @xcite for sub-task a in english, which used smote to over-sample the "hate" class as a preprocessing step, followed by the use of universal sentence encoder @xcite to generate a vector representation of the tweet, and svm (rbf kernel) to classify the tweet. we also implemented a transformer-based approach for this subtask, based on @xcite , which uses pre-trained bert for sequence classification, fine-tuned for 10 epochs. this approach in fact outperforms the aforementioned winning approach. for sub-task b, the multi-classification task, we replicated the winning approach @xcite by training three separate classifiers to classify three label pairs individually; these classifiers used a linear svm on handcrafted syntactic, lexical and bag-of-words features. the optimal hyperparameters were found using grid search. our experiments with these corpora have given us insights about best practices for training effective models of hate speech, which we plan to apply to our new corpus as we collect more data from participating women journalists. we have additionally been exploring experiments on our collected data with various novel model architectures as opposed to data corpora, which are elaborated upon in 10. ## results and discussion although testing of our platform is still in the pilot phase, early users have shared positive feedback regarding its usability, and have also been able to perform the annotation task with good efficiency, on the order of ∼300 tweets per hour. given the size of previously-collected datasets in this space, our methodology is efficient enough to generate sufficient training data in less than 40 hours, making it both a cost-effective and robust approach. given the high fidelity of our labels and the nearperfect ecological validity of the training data, we believe that classifiers trained on data collected using our methods will significantly outperform existing classifiers on hateful and abusive speech in the wild. from early feedback, we have also identified additional labels that participants found relevant, such as "campaign" or "brigade", used to indicate a lexically generic tweet that is still part of a harassment campaign, as in 2019's "learn to code" campaign @xcite . in addition, our pilot interviews suggest that including a fill-in "other" label may be useful for generating more nuanced classifiers, especially as there has historically been a lack of annotator agreement on what constitutes hateful speech, which tends to vary in severity and lexical nature depending on the situation @xcite . ## limitations currently, our approach is limited by its dependence on a feature allowing twitter users to download an archive of their data; this feature was suspended for roughly two months of the research period in response to the social-engineered hacking of more than 100 accounts @xcite . moreover, some blocked or muted users identified in pre-processing may have been suspended by twitter, making it impossible to include their potentially harassing messages in our corpus. finally, while our platform yielded a useful annotation rate, we note that there are inherent limitations to developing classifiers using strictly hand-labeled data. ## directions for future work given the interruption in data collection, we propose to augment our data-access pipeline by building a sufficiently-permissioned twitter app to download the required data directly from participants' accounts. this would not only provide similarly high-quality data with less burden on participants, it would also provide an ongoing source of test data with which we could refine and improve our classifiers in much closer to real-time. by leveraging the methods presented in @xcite , moreover, we also believe we could augment and improve the classifiers built from our hand-labeled data using a combination of machine learning and crowdsourcing. we are in general investigating ways to overcome the inherent shortcomings of manual expert annotation, while retaining its significant benefits; for example, augmenting our data annotation tool with active learning annotation @xcite , so that participants only need to annotate the most unclear instances of hateful/harassing/neutral speech. in regards to model-building, we are exploring ways we can take advantage of the contextual thread annotation scheme present in our annotation platform. specifically, we have investigated methods using lstms @xcite , and are presently investigating graph attention networks @xcite ; these architectures and others like them could allow us to take advantage of the rich metadata and parent tweet text embeddings present in tweet threads, and have the potential to achieve significantly boosted classification performance compared to that of models built on text embeddings of the potentially harassing tweet alone @xcite . for the purpose of building the eventual tool to aid journalists in the field, we could alternatively address the relatively small size of our manuallylabelled datasets for training deep learning classifiers, by augmenting them against the large, popular corpora already in existence. we could investigate whether this addition would boost performance compared to classifiers trained only on those large, crowd-sourced corpora, as a measure of effectiveness of our methodology. finally, we note that while certain semantic features of the classifiers developed using our methodology will differ depending on the community of focus, we hypothesize that by studying several communities with this level of detail and quality, we will eventually be able to identify generalizable features of harassment activities. ## conclusion this work has focused on outlining a novel and generalizable methodology for generating better training datasets for the detection of abusive and harassing speech on twitter, using women journalists as a test community. by directly engaging the targets of harassment in our research, we have not only created an efficient annotation platform using insights about the structural mechanisms of harassment, but we have offered these victims a constructive way to engage with what are otherwise totally negative experiences. we look forward to continuing to work with women journalists to build data-driven tools against abuse and harassment that allow them to maintain their personal needs while working to uphold our free press.