Comparing panic and anxiety on a dataset collected from social media (2024)

Abstract: The recognition of mental health’s crucial significance has led to a growing interest in utilizing social media text data in current research trends. However, there remains a significant gap in the study of panic and anxiety on these platforms, despite their high prevalence and severe impact. In this paper, we address this gap by presenting a dataset consisting of 1,930 user posts from Quora and Reddit specifically focusing on panic and anxiety. Through a combination of lexical analysis, emotion detection, and writer attitude assessment, we explore the unique characteristics of each condition. To gain deeper insights, we employ a mental health-specific transformer model and a large language model for qualitative analysis. Our findings not only contribute to the understanding digital discourse on anxiety and panic but also provide valuable resources for the broader research community. We make our dataset, methodologies, and code available to advance understanding and facilitate future studies.

## introduction the indisputable importance of mental health is recently reflecting in the increased research interest, putting the emphasis on identifying related issues in the textual sources in social media and focusing mainly on depression @xcite @xcite and suicide @xcite . we contend that anxiety and panic, despite being understudied, are equally significant. panic is conventionally described as a sudden, overwhelming fear @xcite , while anxiety is marked by persistent unease and uncontrollable worry @xcite . despite their shared characteristics, the literature indicates that the differentiation between generalized anxiety and panic is valid (russell @xcite . while anxiety, in the related literature, is typically coupled with depression anxiety: "i honestly don't think it can be even described in words. i think anxiety comes in different ways in different people, so i am just going to say about my experience. having anxiety is not being able to remember the last time that you were relaxed without a disturbing thought in your head. having anxiety means that your brain is extremely creative with coming up with the worst improbable almost impossible scenario. it is like having a brain that thinks every minor chest pain as a heart attack, every headache a brain tumor and every numbness ms. having anxiety means to always overthink every action and saying and every situation. it means that you have to forget about having fun and enjoying life. it means that you have always to doubt yourself. i can literally go on forever." panic: "picture yourself sitting on the couch enjoying a television show. you feel relaxed and decide to get up and get something to drink. you open the cabinet door to get a glass when... our of nowhere you feel this big surge of suffocating fear.. your heart starts pounding out of your chest, reality suddenly doesn't feel real. all you know is that you must be alone. you feel like you are free falling faster and faster into the dark pit of hell. you feel depressed and defeated and when it passes you feel exhausted." ( @xcite , panic is, in general and with few exceptions @xcite , @xcite , far less studied. discriminating between anxiety and panic is important, given that individuals experiencing panic face a higher risk of more profound psychological and psychiatric issues, including acute suicidality and agoraphobia. moreover, the study of panic can be instrumental in identifying individuals with ptsd, making it a highly worthwhile topic. thus, the ability to identify patients undergoing panic is crucial, carrying substantial clini-cal implications. nevertheless, to the best of our knowledge, current literature lacks computational approaches for discerning panic and anxiety in social media textual resources. to bridge this gap, we collected a related dataset of 1,930 user posts originating from two well-known websites, quora foot_0 and reddit foot_1 , and we conduct a versatile analysis to address the following research questions: ## dataset collection we web scraped the data foot_3 , starting from the wellknown question answering website quora and a set of questions related primarily to panic (attacks), extending it to similar questions related to anxiety (or other anxiety-related questions that we came upon in quora). we then switched to another popular blogging website, reddit, and looked for semantically similar questions and answers. we only collected the original question and the first reply to it, without tracking the whole conversation (replying to a reply is frequently common in reddit). question could contain either "anxiety" or "panic" keyword and we applied rule-based annotation based on the keyword presence (considering panic class as positive). it is important noting that questions, albeit used for determining class label, are not considered as the integral part of the dataset. tab. 1 presents basic distribution of posts per class per blog, while the questions used for data collection can be found in appendix a.2, tab. 4. it is noteworthy to mention that we adhered to this predetermined set of questions to guarantee the distinct separation of classes, guided by the rulebased strategy driven by question formulation. ## [rq1] how (dis)similar are panic-to the anxiety-annotated posts? and is there any difference wrt the post source (originating blog)? to this end, we conducted a multifaceted analysis. linguistic perspective using liwc-22 (pennebaker et al., 2022) features, we could see than panic posts have on average more words than those of anxiety (169.87 vs. 160.83), also containing more pronouns and verbs. anxiety posts, on the other hand, contain on average more words per sentence (20.64 vs 18.47) and longer words (23.17 vs. 17.45) than panic (see a.2, fig. 5 ). quora posts contain more words in general (191.64 vs. 67.77), more words per sentence and longer words than reddit ones, which, on the contrary, are richer in verbs, adverbs and pronouns (see a.2, fig. 6 ). readability perspective upon calculating different readability scores we have noticed prominent difference between two classes on average flesch score (fs) for quora posts, with 49.84 for anxiety versus 63.72 for panic, meaning that in quora, posts denoted as anxiety are more difficult to read than those denoted as panic (see a.2, fig. 9 ). reddit texts are in general, more similar with respect to readability and also easier to read (with average fs per both classes being around 70). emotion perspective we analyzed two different aspects related to emotions: their presence and their intensity. to determine how represented the emotions are in a post, we employed a pretrained hug-gingface language model @xcite and used inferred emotion probabilities relative to 28 different emotions, from the go-emotions dataset, as a proxy. when considering only 5 most represented emotions per post (fig. 2 ), we can notice that gratitude, nervousness and pride have higher average rank in the anxiety posts than in the panic ones. similar pattern, although with much less important average rank, can be observed for disgust and surprise, while fear, remorse, caring and love have higher average rank in the panic posts. within the top 5 most represented emotions per post, the 5 most frequent are neutral emotion, nervousness, fear and approval for both panic and anxiety; realization for panic and caring for anxiety (see a.2, fig. 4 ). when considering all the emotions per post, the mann-whitney u-test revealed a statistically significant difference in the mean ranks of fear, admiration, amusement, approval, disgust, gratitude, optimism, realization and surprise emotions between panic and anxiety posts, with a significance level set at α = 0.05. to assess emotion intensities, we used intensity lexicon (mohammad, 2018) considering only 8 emotions: anger, anticipation, disgust, fear, joy, sadness, surprise, trust. as expected, fear is the emotion with the highest intensity in both classes. in panic posts, fear and anger have higher intensity on average than in anxiety posts (fear intensity: 0.46 in anxiety vs. 0.53 in panic posts; anger intensity: 0.34 in anxiety vs. 0.40 in panic posts). conversely, anticipation and sadness have higher average intensity in anxiety posts than in the panic ones (see fig. 7 ). with regard to the post sources, as displayed in a.2, fig. 8 , all considered emotion intensities are amplified in quora as compared to reddit. writer's attitude perspective we investigate writer's attitude from the perspective of convincingness @xcite , persuasiveness and usage of irony @xcite . while almost no difference could be noted in the irony and persuasiveness average scores between the two classes, average score for convincingness somewhat differs (0.76 for anxiety vs. 0.73 for panic). when considering the sources, we observe that panic posts in both sources have similar convincingness level. however, on overall quora posts exhibit higher average convincingness (0.78 vs. 0.7) and persuasiveness scores (0.36 vs. 0.29) than reddit. on the contrary, reddit posts contain more irony (average score 0.17) than quora's (0.12). ## [rq2] how successfully can a classifier discern panic from anxiety posts? in order to answer this question, we build a gradient boosting classifier. apart from already mentioned liwc variables (denoted as l), emotion probabilities (denoted as m), emotion intensities (denoted as i), convincingness, persuasiveness and irony scores (denoted together as w), we calculated also the embeddings (b) of the posts and included them as input features. the obtained results can be seen in tab. 2. fairly good performance of classifier, low variance across 10 runs and relatively small discrepancy between f1-macro and f1-weighted/f1-micro scores, despite the strong class imbalanceness, showcase the good predictive power of features and classifier robustness. additionally, sacrificing a bit of performance by excluding embeddings we can obtain an explainable model. the latter, as depicted in a.2, fig. 10 , showcases that liwc mental, fear intensity and liwc long words are considered important in all 10 runs. probing the rule-based annotation one might argue that the rule-based class separation used for annotation makes the two classes easily distinguishable based on the two keywords. we therefore perform two straightforward yet efficient analysis. first, we investigated whether the imposed onequestion-one-keyword pattern holds for the answers as well. it turned out that it was not the case. more specifically, out of total 1930 blog posts, as much as 386 (20%) contain both "panic"and "anxiety" keywords. the co-occurrence of "panic"and "anxiety" keywords is specifically prominent in quora where 343 out of 1502 posts (22.84%) contain both keywords, while in reddit this is less frequent (only 43 out of 428 posts or 14.06%). second, we replaced all the occurrences of the two keywords with "maskedcontent" token, recalculated the respective features on the obtained texts and retrained the classifier. as could be seen from tab. 3, although, as expected, the performances drop in terms of all considered metrics, the downgrade in the performance is not prominent. this indicates that the presence of the "panic" and "anxiety" keywords in not crucial for classification. ## [rq3] what a qualitative nlp-assisted we conduct a two-fold analysis. first, we aim to determine what the posters were experiencing. to this end, we prompted language model pretrained on mental health discussions @xcite ), requesting a yes/no answer related to stress and open answers related to "experiencing". results are depicted in a.2, fig. 12 and 11 respectively. second, we leveraged large language models (llm), specifically chatgpt-3.5. we initially used it to assess the dataset annotation quality, finding that only 8.4% of collected panic posts were misclassified as anxiety by chatgpt, and merely 3.8% of original anxiety posts were erroneously labeled as panic. we then conducted a comprehensive multidimensional thematic and content analysis. this involved automated coding and key-phrase extraction across various dimensions (see a.2 for details). several noteworthy observations emerged from our investigation. notably, panic attacks exhibited a higher "strong" intensity (21%) compared to anxiety attacks (7%). in the context of relationships, the top four categories were shared among classes. however, the fifth differed, with "work" for anxiety and "romantic partners" for panic. location-wise, the prevalent places were home, school, work, and public places. in the triggers category, social events constituted 33% of anxiety triggers, contrasting with only 9% for panic attacks. traumatic experiences accounted for 11% of panic triggers, as opposed to 5.6% for anxiety. notably, financial stress was among the top five anxiety triggers but was absent in panic (see fig. 3 ). understanding anxiety and panic triggers is vital for uncovering their root causes and informing targeted interventions @xcite @xcite . examining specific triggering events helps elucidate factors contributing to distress. focusing on locations where these experiences occur provides insights into environmental influences @xcite , aiding the development of strategies for supportive environments. investigating relationships contributes valuable insights into the impact of social interactions on these conditions @xcite , facilitating interventions to improve social relationships and support systems. figure 3 emphasizes these dimensions to enhance comprehension. in the category of other_medical_conditions, stress-related conditions were prevalent in both anxiety and panic. however, panic exhibited a unique pattern, with breath-related conditions like asthma, copd, sinus infections, nasal congestion, and covid-19 being the most frequently mentioned. more details on different distributions, including professional inteventions, is provided in a.2, tab. 5. ## conclusion in this study, we collected a dataset on two prevalent and closely related mental disorders: panic and anxiety. our analytical exploration centered around three primary questions. firstly, we focused into uncovering the distinctions and similarities, both in content and sources, between the anxiety and panic classes. to address this, we conducted lexical analyses, emotion assessments, and evaluations of writers' attitudes. secondly, we assessed the efficacy of a classical machine learning classifier, leveraging combinations of the various extracted features, in distinguishing between posts belonging to these classes. importantly, we conducted experiments to ensure the classifiers did not rely solely on the presence of explicit "panic"/"anxiety" keyword. lastly, we sought insights into these disorders through a qualitative analysis of the dataset, leveraging a bidirectional transformer model trained on mental health texts (mental-roberta) and a largelanguage model (chatgpt-3.5). as a part of our contribution, we provide the entire output resulting from the nlp processes, including the collected codings and key-phrase extractions derived from the qualitative analysis. we hope that the obtained insights could be further exploited by domain experts. looking ahead, future explorations aim to contribute to bring more layers of understanding within the context of these mental health disorders, for example, including a more in-depth analysis and clustering of key-phrase excerpts, with a particular focus on those associated with coping strategies, negative behavioral changes, and the impact on daily life.