Identifying Worry in T witter: Beyond Emotion Analysis (2020)

Abstract: Identifying the worries of individuals and societies plays a crucial role in providing social support and enhancing policy decision-making. Due to the popularity of social media platforms such as Twitter, users share worries about personal issues (e.g., health, finances, relationships) and broader issues (e.g., changes in society, environmental concerns, terrorism) freely. In this paper, we explore and evaluate a wide range of machine learning models to predict worry on Twitter. While this task has been closely associated with emotion prediction, we argue and show that identifying worry needs to be addressed as a separate task given the unique challenges associated with it. We conduct a user study to provide evidence that social media posts express two basic kinds of worry – normative and pathological – as stated in psychology literature. In addition, we show that existing emotion detection techniques underperform, especially while capturing normative worry. Finally, we discuss the current limitations of our approach and propose future applications of the worry identification system.

## introduction knowing what an individual or society at large, worry about -e.g., unemployment, health issues, ageing, the rise of ai -is an indicator of people's well-being. capturing information like the possible source and nature (e.g., type, intensity) of people's worry is used by many governmental agencies 1,2,3 to guide their policy decisions. this can range from minor decisions such as initiation of information campaigns (e.g., to counter false information being spread during the covid-19 pandemic) to major policy changes like introduction and amendment of laws (e.g., increase of minimum wage, mandatory health insurance, data privacy and fake news). many private companies (e.g., toyota) listen to the worries of their customers using techniques like "voice-of-customer" @xcite as part of their quality function deployment (qfd) process @xcite . measuring worry, however, is quite challenging. traditional approaches rely on time-consuming and costly user surveys and polls (e.g., on a large scale, the world happiness report @xcite , the global risk report (wef, 2020)). these surveys, although very well-structured, suffer from some severe limitations. firstly, being resourceconsuming, (large) surveys are generally conducted periodically (e.g., 1-2 times a year). this, in turn, creates knowledge gaps as it is hardly possible to track short-term trends following significant events (e.g., pandemic outbreak, natural disasters, terrorist attacks). secondly, most of the surveys tend to have a narrow scope with a specific, pre-defined purpose. for example, surveys conducted by the public housing agency are most likely to be limited to the worries of the residents of those societies. lastly, surveys involving sensitive subjects (e.g., racism, immigration, lgbt rights, abortion, religion, politics) tend to suffer from non-response bias of the participants who might divulge their true opinions -even if they are ensured anonymity -in order to adhere to political correctness. in contrast, social media provides a platform for individuals to freely and continuously express their thoughts, feelings and experiences as well as to share information with other members of the society. content on platforms like twitter is generally public and can be easily collected on a large scale, thereby, making social media mining a promising approach to observe and analyze people's worries. social media, however, comes with its own set of challenges in form of self-censorship bias and the presence of scripted bots. besides this, unlike well-structured surveys that often have structured questions (single or multiple-choice, rating and ranking), social media content is quite unstructured. sophisticated analytics, most of them using state-of-the-art machine learning algorithms, are often required to extract meaningful and in-depth insights from such unstructured data. many works have been proposed to capture people's well-being using social media; see section 2. in most of these works, worry is associated with notions such as anxiety, fear and nervousness. however, as per psychology literature, worry is primarily considered to be a thought process concerned with future events that often has adverse and uncertain outcomes @xcite . since people are confronted with or affected by events with uncertain outcomes in their day-to-day life, most of them are worried to some extent. only when the extent of these worries becomes uncontrollable and excessive, this normative process becomes pathological and results in anxiety or depression @xcite . with the increase in people's worries, their emotional responses tend to get stronger, typically expressed as fear. thus, while fear implies worry, not all worries necessarily result in fear @xcite . therefore, we argue that the existing approaches used to evaluate people's mental or emotional states (fear, anxiety, depression, etc.) are not sufficient to accurately capture the notion of worry due to these arguably subtle but important differences. figure 1 (using paralleldots foot_3 api) shows evidence that an emotion classifier is unable to predict worry in tweets classified as sad and happy. to validate our hypothesis, we first compare people's perception of worry and emotions using the same underlying data. we use crowdsourcing to re-annotate a well-established twitter dataset curated for emotion prediction for our new target task of identifying worry. we analyze this dataset to establish that emotion is not an adequate predictor of worry. secondly, using the re-annotated dataset, we train different machine learning models -feature-based, word embedding-based and contextual embedding-based -to refine further the subtleties that result in differences between emotion and worry. next, we perform an in-depth analysis of different kinds of worry -normative and pathological -by conducting a user study. lastly, we perform error analysis to highlight current shortcomings as well as challenges while discussing the future work leading to more effective worry prediction. ## related work psychology literature defines worry to be a futureoriented thought process typically concerned with a problem whose potentially negative outcome is uncertain @xcite . most people deal with some degree of worry on a daily basis in the form of normative or non-pathological worry @xcite . however, excessive, pervasive and uncontrollable worry becomes pathological in the form of generalized anxiety disorder @xcite . besides this difference in intensity of worry, psychology also categorizes worry into various life domains (e.g., health, social relations, environment) as well as the object of worry (e.g., self, close friends/relatives, society and the world) @xcite @xcite . as social media has become one of the most popular services online, users on these platforms indulge in widespread sharing of thoughts, opinions and feelings, as well as, events that constitute their everyday lives like check-ins, relationships, and more @xcite . several studies have shown that the language, linguistic style and behavior derived from social media posts often reflect users' personal characteristics @xcite . consequently, many methods have been proposed to use social media content to predict users' personality traits and well-being. most works such as @xcite @xcite , aim to predict the personality of social media users using the big 5 personality traits: ocean (openness, conscientiousness, extraversion, agreeableness, and neuroticism). while personality traits help in understanding how often a user might worry, they do not allow us to predict whether a particular post expresses worry. existing efforts towards evaluating users' wellbeing related to worry have focused on emotion prediction @xcite , depression @xcite , anxiety and stress @xcite , suicidal thoughts @xcite -that is, on pathological causes affecting users' well-being. however, worrying is not necessarily pathological and only becomes so when it grows excessive and uncontrollable @xcite . similarly, since "daily worries" are part of most people's lives, not all worries trigger a (strong) emotional response. worry, as a thought process concerned with future events that have uncertain and potentially (very) negative outcomes, is most closely related to fear and anxiety. in fact, many existing works predicting emotions in users' social media posts @xcite @xcite associate fear with worry. since worry does not necessarily imply fear, emotions alone are not a good predictor for worry, as they are skewed towards strong feelings of worry that more likely yield an emotional response. ## datasets and experimental setup in this section, we firstly describe our dataset for the task of identifying worry in tweets, then briefly discuss the methodology of data pre-processing and finally outline the set of machine learning models used for evaluation. ## experiments and results this section covers four parts: the evaluation of our worry classifier, the effect of emotion on worry prediction, analysis of different kinds of worry using a user study and an error analysis discussing the core challenges towards further improving this task. ## conclusion and future work worry about a personal issue such as health or finance, or a broader external issue such as environmental pollution, technology change or social structure is commonly expressed on twitter nowa-days. most of the existing works utilizing social media for measuring well-being associate worry with emotions. taking cues from the psychology literature that clearly differentiates pathological (uncontrollable with high emotion intensity) and normative (everyday with low/zero emotion intensity) worry, we argue as to why emotion classifier is unable to capture normative worry, thereby, establishing the need to treat worry detection as a separate task. we started out by exploring the effectiveness of a worry classifier by comparing different state-ofthe-art text classification models with/vs. emotion. we then conducted an empirical user study to further strengthen our hypothesis where we discussed the differences between pathological and normative worry in detail. our results support our argumentthat emotion classification can, at best, only be sufficient to predict pathological worries as they yield strong emotional responses. this topic is, however, less explored despite the immense potential in applications such as identifying day-to-day worries like excessive traffic on a certain route or stressful work environment. this paper lays down an initial ground for future work in this direction but is far from perfect. we highlight the current limitations of this task by performing a qualitative error analysis. one of the main challenges is that worry is often only implied and requires access to shared or common knowledge. utilizing such knowledge will be an important next step to improve the identification of worry. looking at the bigger picture, we envision to implement a real-time, automated worry classification system capable of capturing both pathological and normative worries at different levels -local, national and global -to aid policy and decision making processes of organizations all around the world.