Detecting Perceived Emotions in Hurricane Disasters (2020)

Abstract: Natural disasters (e.g., hurricanes) affect millions of people each year, causing widespread destruction in their wake. People have recently taken to social media websites (e.g., Twitter) to share their sentiments and feelings with the larger community. Consequently, these platforms have become instrumental in understanding and perceiving emotions at scale. In this paper, we introduce HurricaneEmo, an emotion dataset of 15,000 English tweets spanning three hurricanes: Harvey, Irma, and Maria. We present a comprehensive study of fine-grained emotions and propose classification tasks to discriminate between coarse-grained emotion groups. Our best BERT model, even after task-guided pre-training which leverages unlabeled Twitter data, achieves only 68% accuracy (averaged across all groups). HurricaneEmo serves not only as a challenging benchmark for models but also as a valuable resource for analyzing emotions in disaster-centric domains.

## introduction natural disasters cause thousands of deaths and displace hundreds of millions each year (ritchie and roser, 2020). these catastrophic events not only induce material destruction but also stir an integral part of being human: our emotions. disasters adversely affect individuals' mental states @xcite , and therefore it is no surprise that many take to social media (e.g., twitter) to share their feelings. social media websites, as a result, have become an essential platform for understanding the expression and perception of emotions at a significantly larger scale @xcite @xcite @xcite , with far reaching potential influences from academic re-search to public policy @xcite @xcite . while natural language processing methods have been effective for emotion detection @xcite , existing resources struggle in disaster-centric domains, in part due to distributional shifts. emotion detection in natural disasters (e.g., hurricanes) requires implicit reasoning not available as surface-level lexical information. for example, in "of course, @xcite 1 still have the [storm surge] 2 coming," given the context, we can reasonably infer discontent towards the "storm surge" despite the absence of polarizing words. therefore, distantly supervised techniques largely based on lexical units @xcite fail to capture this type of deeper semantic phenomena. our paper presents a comprehensive investigation into perceived emotions in hurricane disasters. to this end, we introduce hurricaneemo, a dataset of 15,000 disaster-related tweets (in english) streamed during hurricanes harvey, irma, and maria, which were devastating tropical storms occurring in the 2017 atlantic hurricane season @xcite . our samples are annotated with fine-grained emotions derived from the plutchik wheel of emotions @xcite , a well-defined ontology of emotion classes commonly used in computational social science @xcite . 1 to measure inter-annotator agreement on fine-grained emotion labels, we conceptualize the plutchik emotion agreement (pea) metric ( §3). pea is intuitively grounded; our human evaluation shows workers agree with pea's rankings 88% of the time. furthermore, we perform insightful analyses on implicit and explicit emotions in hurricane tweets ( §4). quite surpris-ingly, we find consistencies in plutchik-24 emotion distributions across hurricanes harvey, irma, and maria. hurricaneemo also serves as a challenging new benchmark for large-scale, pre-trained language models. we establish baselines for a coarser plutchik-8 emotion detection task using bert @xcite and roberta @xcite . our experiments reveal: (1) bert only achieves 64% (averaged) accuracy; and (2) using "better" pre-trained models (e.g., roberta) does not help, which is a strikingly different trend than most leaderboards @xcite . to better understand their pitfalls, in particular bert, we conduct a comprehensive error analysis of 200 incorrectly predicted samples. in addition, we incorporate stronger inductive biases into bert via pretraining on related tasks, which culminates in (averaged, absolute) +4% accuracy ( §6). finally, we propose unsupervised domain adaptation to bridge the domain gap between existing large-scale emotion datasets (e.g., emonet @xcite ) and hurricaneemo ( §7). our code and datasets are made publicly available. foot_1 2 related work emotion detection has been extensively studied in news headlines @xcite , blog posts @xcite , health-related posts @xcite , and song lyrics @xcite , but only recently, in social media websites (e.g., twitter, facebook) @xcite @xcite @xcite . however, emotion detection in disaster-centric domains, despite its practical importance, is limited. @xcite (singlehandedly) annotate 2,200 hurricane sandy tweets using ekman-6 emotions @xcite ). in contrast, we introduce 15,000 annotated tweets from multiple hurricanes with (much more fine-grained) plutchik-24 emotions. unlike abdul-mageed and ungar (2017), we focus on readers' perceived emotions rather than writers' intended emotions. furthermore, in disaster-centric domains, the lack of labeled data required to train reliable models precludes the use of supervised learning techniques. several works propose to use labeled data from prior (source) disasters to learn classifiers for new (target) disasters @xcite @xcite @xcite . however, due to the unique nature of each disaster (e.g., type, geographical location, season, cultural differences among the affected population), the source disaster may not accurately reflect the characteristics of the target disaster @xcite . domain adaptation techniques address these challenges by efficiently using large amounts of unlabeled target domain data, consequently outperforming the aforementioned supervised techniques @xcite . our work contributes to disaster-centric emotion detection in three ways by: ## dataset construction in this section, we present hurricaneemo, an annotated dataset of 15,000 english tweets from hurricanes harvey, irma, and maria. we detail each component, including the initial preprocessing ( §3.1), annotation procedures ( §3.2), and the formulation and calculation of inter-annotator agreement ( §3.3). ## baseline modeling we now turn to modeling the emotions in hurri-caneemo. because plutchik-24 emotion counts are heavily imbalanced, we group them into plutchik-8 emotions and consequently create 8 binary classification tasks. the tweets are assorted into their respective label buckets; because tweets may be labeled with more than one emotion, each belongs to one or more buckets. these buckets represent positive samples (i.e., tweets labeled with that emotion). to create negative samples, we sample an equal amount from other buckets. from here, we shuffle the positive and negative samples and perform an 80/10/10 split to create the train, validation, and test sets. foot_3 ## task-guided pre-training to improve upon our baselines, we explore pretraining as a means of implicitly incorporating an inductive bias into our bert model. our hope is that these pre-training tasks will not only make bert more robust in the twitter domain, but also provide useful (albeit abstract) knowledge for the end emotion prediction tasks. for brevity, we chiefly focus on bert, although our methods can be generalized to other pre-trained models. setup. we explore, in isolation, supervised and unsupervised pre-training tasks. for the supervised setting, we pre-train on a multi-class emotion task (emonet) @xcite and binary sentiment analysis task (sentiment) @xcite . for the unsupervised setting, we pretrain on dynamic masked language modeling @xcite on (unlabeled) samples from emonet, sentiment, and hurricaneext ( §3.1). for both types of tasks, we further pre-train bert for a fixed number of epochs, then fine-tune it on a hurricaneemo task. we compare these results to no-pretrain, namely the bert results verbatim from ## fine-grained unsupervised domain adaptation when new disasters emerge, it is likely we may not have emotion annotations, as alluded to previously ( §2). nevertheless, these annotations would be valuable for organizations trying to understand the emotional profile of users during a crisis @xcite . in this section, we explore ways to leverage supervision from large-scale emotion datasets (e.g., emonet (abdul-mageed and ungar, 2017)) in providing labels for our hurricane emotion datasets. we frame this problem as unsupervised domain adaptation; emonet is the labeled source domain and our hurricane datasets are the unlabeled target domain. below, we elaborate framework. emonet was conceived as a multiclass classification task for plutchik-8 emotions @xcite . in contrast, we introduce binary classification tasks, one for each plutchik-8 emotion. we split the emonet multiclass task into 8 binary tasks; this creates a oneto-one alignment between each source and target domain task. we separately perform unsupervised domain adaptation for each binary task. methods. we use our bert model (without taskguided pre-training) as the underlying classifier. following @xcite , we chiefly focus on using strategic pre-training techniques that enable effective transfer between disparate domains. the systems for comparison are: (1) src-only: bert is trained in the source domain and evaluated in the target domain; results. ## conclusion we present hurricaneemo, an annotated dataset of perceived emotions spanning 15,000 tweets from multiple hurricanes. tweets are annotated with finegrained plutchik-24 emotions, from which we analyze implicit and explicit emotions and construct plutchik-8 binary classification tasks. comprehensive experiments demonstrate our dataset is a challenging benchmark, even for large-scale pre-trained language models. we release our code and datasets as a step towards facilitating research in disastercentric domains.