W orry W ords: Norms of Anxiety Association for over 44k E nglish Words (2024)

Abstract: Anxiety, the anticipatory unease about a potential negative outcome, is a common and beneficial human emotion. However, there is still much that is not known about anxiety, such as how it relates to our body and how it manifests in language; especially pertinent given the increasing impact of related disorders.In this work,we introduce WorryWords , the first large-scale repository of manually derived word–anxiety associations for over 44,450 English words. We show that the anxiety associations are highly reliable.We use WorryWords to study the relationship between anxiety and other emotion constructs, as well as the rate at which children acquire anxiety words with age. Finally, we show that using WorryWords alone, one can accurately track the change of anxiety in streams of text.WorryWords enables a wide variety of anxiety-related research in psychology, NLP, public health, and social sciences.WorryWords (and its translations to over 100 languages) is freely available. http://saifmohammad.com/worrywords.html

## introduction anxiety is the apprehensive uneasiness or nervousness due to some anticipated or potential negative outcome. even though it is often considered to be an unwanted and negative feeling, psychologists have long posited its evolutionary benefits. as a species, we have benefited from feeling anxious: for example, building safe dwellings to avoid being attacked by wild animals and poisonous insects, being respectful and kind to others to avoid social exclusion, and so on @xcite . yet, there is much we do not know such as the mechanisms of how anxiety develops in our bodies and mind, how normal and clinical anxiety manifest in language, and effective mechanisms to cope with excessive anxiety. further, the dramatic technologi-cal advances over the last few decades have rapidly and substantially changed our environments to such an extent (for instance, the widespread social media usage and human-technology interactions) that there is now a considerable mismatch between the current environment and what our anxiety response slowly evolved to address over millennia @xcite @xcite . this has led some to argue that we are in an 'age of anxiety' @xcite . anxiety has always been a part of normal human life. some might feel anxious occasionally and in particular circumstances (state anxiety), and some might have a more diffuse sense of general anxiety over long periods of time (trait anxiety). some of the major sources of anxiety include tests, sports, technology, and social anxiety @xcite . more recently researchers have also identified a growing amount of climate change anxiety @xcite . mental health practitioners clinically diagnose anxiety disorders when the sense of anxiety is so high or so frequent that it impacts the day-to-day functioning of an individual. in the us alone, 6.8 million adults (3.1% of the population) are believed to have generalized anxiety disorder (gad), 15 million adults (7.1%) have social anxiety disorder (sad), 7.7 million adults (3.6%) suffer from post-traumatic stress disorder (ptsd), and everyone experiences stress from time to time. 1 anxiety disorders are highly co-morbid with depression (kalin, 2020), and together they constitute the internalizing disorders. in addition to impacting well-being and happiness, anxiety (both normal and clinical) has been shown to impact cognitive outcomes such as learning, accurately performing tasks, productivity, as well as our physical bodies @xcite . in contrast, calmness (being in a state of peace and tranquility) has been shown to improve health and cognitive outcomes @xcite . @xcite also show the importance of conceptualizing anxiety as a continuum from high calmness to high anxiety; and that both promoting calmness and decreasing excessive anxiety are important for overall health and well-being. thus, anxiety and calmness are active areas of research in psychology, medicine, public health, social science, economics, and even the humanities. as is the case with many emotions, people express anxiety through language (consciously and unconsciously). in this work, we study how anxiety and calmness are associated with word meaning. we asked over a thousand respondents to rate 44,450 english words on a scale from -3 (associated with high calmness) to +3 (associated with high anxiety). we aggregated the responses to create a word-anxiety association lexicon that has a real-valued anxiety score for each of the words. we refer to this resource by the name worrywords. we show that the scores in worrywords are highly reliable (a split-half reliability score of 0.82, which is in-line with scores for existing resources for other emotions). we use worrywords to study how anxiety is related to other emotion constructs @xcite . we show the rate at which anxiety and calmness words are acquired with age. finally, we show that wor-rywords can be used with simple word-counting methods to accurately track anxiety in text streams. we make worrywords, and its translations in over 100 languages, freely available for research through the project home page. 2 some areas where we envisage its use include: we also created an extensive list of ethical considerations (section 9) that must be taken into consideration when using worrywords. it is generally agreed that the three primary dimensions of affect (our sense of feeling) are valence (positiveness-negativeness / pleasuredispleasure), arousal (active-passive), and dominance (dominant-submissive) @xcite . there is also work showing that certain categorical emotions such as joy, anger, and fear are particularly important @xcite . more recent work shows that we construct and label these emotions (a cognitive process) in our minds from various interoceptive signals (internal signals from our body to the brain) @xcite . anxiety is often considered a low-valence, higharousal, and low-dominance emotion, but there is little work empirically exploring the degree to which it is correlated with these affective dimensions. anxiety is also often compared to fear, but unlike fear which generally tends to be a reaction to a threat from the present (something that has already manifested), anxiety is often associated with the possible negative outcome of a future event @xcite . anxiety and stress are sometimes used synonymously and they are generally agreed to be very close emotions (both impact mind and body through excessive worrying, loss of sleep, etc.), although stress is often associated with some physical stimulus and stress can be one of the causes of anxiety. manually created word-emotion association lexicons exist for valence, arousal, and dominance @xcite and @xcite for english; @xcite for german). the largest among these is the nrc vad lexicon @xcite : version 1 has entries for over 20,000 english words, and version 2 for over 44,000 unigrams and 10,000 bigrams (two-word sequences). the nrc emotion lexicon @xcite is a popular lexicon for categorical emotions; it has entries for over 14,000 english words. we do not know of existing large lexicons for anxiety; however, there exist some datasets of social media posts manually annoatated with anxiety or stress labels. @xcite collected and annotated about 5,488 reddit posts with the labels 0 (no stress) and 1 (stress). @xcite collected a series of questions and answers from quora and reddit, where the question contained the term 'anxiety'. @xcite created a dataset of 4000 5-sentence segments from reddit annotated for presence or absence of stress. these social media datasets were compiled primarily to train and test automatic systems that attempt to detect anxiety from text. for details on automatic anxiety detection, we refer the interested readers to @xcite ; shen and rudzicz (2017); @xcite . more recent work interviewed volunteers and analyzed their responses to identify language features such as the use of personal pronouns, negations, and emotion words with anxiety @xcite . as part of the worrywords project, we created a large word-anxiety association lexicon with a focus on reliable annotations. 3 we used existing emotion lexicons to examine how anxiety is related to other emotions. we also made use of a the rastogi et al. (2022) dataset to show the usefulness of worrywords in tracking anxiety in text streams. ## obtaining human ratings of anxiety the keys steps in creating the anxiety dataset were: consider anxiety to be a broad category that includes: jittery, antsy, insecure, nervous, unease, tense, worried, unnerving, nerve-racking, apprehensive, fretful, troubled, self-doubting, discontented, concerned, and keyed up consider calmness to be a broad category that includes: calm, relaxed, comforted, serene, at ease, self-assured, carefree, composed, collected, untroubled, peaceful, contented, unconcerned, indifferent, and uninvolved select the options that most english speakers will agree with. q1. <@xmath0: 3: very anxious -1: slightly calm 2: moderately anxious -2: moderately calm 1: slightly anxious -3: very calm 0: not associated with feeling anxious or calm 3. quality control measures. about 2% of the data was annotated beforehand by the authors and interspersed with the rest. these questions are referred to as gold (aka control) questions. half of the gold questions were used to provide immediate feedback to the annotator (in the form of a pop-up on the screen) in case they mark them incorrectly. we refer to these as popup gold. this helps prevent the situation where one annotates a large number of instances without realizing that they are doing so incorrectly. it is possible, that some annotators share answers to gold questions with each other (despite this being against the terms of annotation). thus, the other half of the gold questions were also separately used to track how well an annotator was doing the task, but for these gold questions no popup was displayed in case of errors. we refer to these as no-popup gold. 4. crowdsourcing. we setup the anxiety annotation task on the crowdsourcing platform, mechanical turk. in the task settings, we specified that we needed annotations from ten people for each word. we obtained annotations from native speakers of english residing around the world. annotators were free to provide responses to as many terms as they wished. the annotation task was approved by our institution's review board. dataset #words annotators #annotators #annotations mai shr (ρ) shr (r) unigrams 44,450 us, india, uk, canada 1,020 375,796 8.5 0.82 0.89 the final anxiety score for each term is simply the average score it received from each of the annotators. we also created a categorical version of the lexicon by labeling all words that got a @xmath2.5 as associated with high anxiety, ≥ 1.5 @xmath3.5 as moderate anxiety, ≥ 0.5 @xmath4.5 as slight anxiety, > -0.5 @xmath5.5 as neither anxiety nor calmness, and so on. we refer to the list of words along with their real-valued final scores and categorical labels as the worrywords lexicon. ## reliability of the annotations a useful measure of quality is the reproducibility of the end result-repeated independent manual annotations from multiple respondents should result in similar scores. to assess this reproducibility, we calculate average split-half reliability (shr) over 1000 trials. 5 all annotations for an item are randomly split into two halves. two separate sets of scores are aggregated, just as described in section 3 (bullet 6), from the two halves. then we determine how close the two sets of scores are (using a metric of correlation). this is repeated 1000 times and the correlations are averaged. the last two columns in if the real-valued scores are converted to discrete categories (for example, into 3, 7, or 10, classes), then we can make use of another measure of reliability that factors in the classes. for example, if we just want three classes: associated with anxiety, associated with calmness, and neither. then one could simply split the -3 to 3 range into three equal partitions of -3 to -1 (calmness), -1 to 1 (neither), and 1 to 3 (anxiousness). 6 to determine reliability of the discrete labels obtained from annotations, we first randomly split all annotations into two halves (just as before). now, if scores for a term x obtained from the two halves place x in different classes (e.g., if the two scores are 0.36 and 1.72), then this contributes to a lower overall reliability. in contrast, even if the scores for x from the two halves vary a bit, but both scores place x in the same class (e.g., if the two scores are 1.82 and 1.79) then this contributes to a higher overall reliability. thus we can compute reliability from the observe that the shcmp scores are markedly higher than what one would get with just random annotations. further, shcmp scores of over 95% for 5 and fewer classes indicate very high levels of reliability. ## worrywords: research & applications worrywords has a wide variety of uses and applications as listed in the introduction. since this paper introduces the lexicon, we include here: an important question in understanding any emotion is how it relates to other emotions. with the creation of the first large-scale lexicon of wordanxiety associations, we can now study how these norms correlate with other key emotion constructs such as valence (v), arousal (a), and dominance (d), as well as categorical emotions such as sadness, fear, and anger. since anxiety is commonly understood to be a negative (low-valence emotion), we expect it to have a negative correlation with valence (negative situations are often associated with anxiety). however, we have no prior evidence to indicate how strong this negative correlation is expected to be; especially since, there can be many term pairs w1 and w2 such that even though w1 has lower valence than w2, w2 is associated with a higher degree of anxiety than w1. for example, hopeless and frenzy, lethargic and public speaking, weak and test, etc. similarly, we expect anxiety to have a positive correlation with arousal (since high anxiety terms are also expected to be high arousal terms), and a negative correlation with dominance (since higher anxiety tends to co-occur with situations of loss of control). yet, again, it is unclear how strong these correlations truly are across a vast vocabulary of words. understanding these relationships has implications in psycholinguistics and lexical semantics: e.g., whether anxiety moderates word recognition even when controlling for valence. further, in the context of lexicon-creation: if the magnitudes of these correlations are very high, then one can simply take an existing vad lexicon, and keep only the low valence, high arousal, and/or low dominance terms to create an anxiety lexicon. vad: the first row (and also the first column) of we also created an interactive visualization that shows the worrywords terms on valence-arousal and valence-dominance spaces (figure 8 in the appendix shows a screenshot). (hovering over individual points shows the corresponding term as well as anxiety, v, a, and d scores.) we observe that, as expected, a majority of the anxiety words (@xmath6.5) occur in the top-left quadrant of the v-a space: low v and high a. nonetheless, there are still non-negligible number of terms that occur in the top-right quadrant (such as adrenalin, revolutionist, and whizzing) and in the bottom-left quadrant (such as joylessness, disheartenment, and disreputableness). comparatively much fewer anxiety terms occur in the bottom right (high v and low a) quadrants. examples include seatbelt and postoperative. in contrast, the majority of the calmness terms (@xmath7.5) occur in the high v quadrants. examples, of low v terms associated with calmness include pity and aloof. the v-d plots show that anxiety terms tend to be concentrated in the bottom-left quadrant, whereas calmness terms are most commonly found in the top-right quadrant. anxiety terms associated with high d include terms such as skyrocket, canny, gunfight, and tyrant; anxiety terms associated with low d include persecuted and gutlessness. categorical emotions: discussion: overall, we see that while anxiety associations follow some expected trends such as pertaining to low valence, the magnitudes of correlations are moderate with valence and low with arousal and dominance. thus anxiety presents itself in many different ways and cannot be charac-terized simply with vad. the fact that anxiety has the highest correlation with fear (among the four negative categorical emotions) is in line with the belief that fear is a key aspect of anxiety @xcite . ## capturing anxiety arcs in text streams using worrywords a common use of emotion lexicons is to determine emotion arcs and trends in how emotions change in a stream of textual data. for example, determining changes in the degree of anger towards a product in social media over time @xcite , tracking happiness and well-being of different cities and counties through geo-located posts on x (formerly twitter) in public health research @xcite , generating emotion arcs of stories in digital humanities @xcite , etc. the lexicons may be used in combination with supervised systems and large language models (where such resources are available), or they may be used on their own where interpretability, low-carbon foot print, and simpler systems are valued -which is often the case in psychology and public health research. lexicon-only methods are often markedly less accurate at determining emotions of individual sentences. however, teodorescu and mohammad (2023) have shown that on the task of generating emotion arcs (where emotion scores of hundreds, if not thousands, of individual sentences are averaged to determine points on the emotion arc) lexicon-only methods are competitive with more sophisticated neural and large language models. 9 in this section we explore: ## conclusion we created worrywords -a large lexicon of word-anxiety association scores with the help of responses to a simple questionnaire, by over 1000 respondents (mainly from the us, but also from india, uk, and canada). we showed that the responses led to highly reliable scores. we compared how anxiety association scores correlate with the scores for various other emotions. we determined the rate at which anxiety words are acquired with age; specifically, how from ages 1 to 4 children learn more calmness words than anxiety words, and that trend is reversed age 5 onwards. finally, we showed how worrywords can be used to track anxiety in large text streams. we make worry-words freely available to enable a wide variety of anxiety associated research (such as in the seven bullets listed in the introduction) in nlp, psychology, social science, etc.