Cluster Analysis of Online Mental Health Discourse using Topic-Infused Deep Contextualized Representations (2021)

Abstract: With mental health as a problem domain in NLP, the bulk of contemporary literature revolves around building better mental illness prediction models. The research focusing on the identification of discussion clusters in online mental health communities has been relatively limited. Moreover, as the underlying methodologies used in these studies mainly conform to the traditional machine learning models and statistical methods, the scope for introducing contextualized word representations for topic and theme extraction from online mental health communities remains open. Thus, in this research, we propose topic-infused deep contextualized representations, a novel data representation technique that uses autoencoders to combine deep contextual embeddings with topical information, generating robust representations for text clustering. Investigating the Reddit discourse on Post-Traumatic Stress Disorder (PTSD) and Complex Post-Traumatic Stress Disorder (C-PTSD), we elicit the thematic clusters representing the latent topics and themes discussed in the r/ptsd and r/CPTSD subreddits. Furthermore, we also present a qualitative analysis and characterization of each cluster, unraveling the prevalent discourse themes.

## introduction due to their ubiquitous nature, online health communities and social media platforms have emerged as a conducive means of information exchange and social support, especially for people with stigmatized concerns such as mental health. consequently, these platforms provide a rich ecosystem for mental health clinicians, researchers, and practitioners to analyze the cornucopia of user-generated content and study the underlying mechanisms of different mental health conditions. with the rapid headways in artificial intelligence and computational linguistics, an increasingly large number of researchers have leveraged social media content to study various mental health illnesses and psychiatric conditions. while most of the research has focused on using the traditional machine learning models and statistical methods for predicting mental illness from social media posts, the studies addressing the discourse analysis @xcite @xcite and identification of clusters in online mental health communities @xcite has been relatively modest. even with the recent surge of complex attention-based deep learning models, a large chunk of the research regarding mental health issues has focused on building better predictive systems @xcite @xcite with less emphasis on using these models for mental health related corpus analysis or information extraction. with mental health already coalesced as an appreciable public health burden, research to investigate the discourse clusters prevalent on social media is of paramount importance. mining information from the emergent clusters provides a lens over the dominant themes of discussion, the discourse anatomy, and the dialogue structure in the online forums while also helping to comprehend the general public engagement, sentiment, ideas, and views regarding mental health. successful research in this direction can potentially foster identification of high-risk groups, enhanced mental health patient education programs, better diagnostic and therapeutic theory building, as well as an improved understanding of the underlying design of the online mental health communities @xcite . post-traumatic stress disorder (ptsd) is a mental disorder resulting from traumatic experiences that leads to reliving the trauma, avoidance of certain situations, and hyper-vigilance. similar to ptsd, complex post-traumatic stress disorder (c-ptsd) is a condition that formulates the reaction resulted from the trauma, such as uncontrollable emotions, dissociation, negative self-perception, anger, mistrust, and interpersonal difficulties. thus, in this research, we examine the online discourse on reddit, focussing on ptsd and c-ptsd as use cases to elicit different thematic clusters present in them. prior research has shown that the addition of topic information to pre-trained contextualized representations yields performance improvement for semantic textual similarity @xcite . @xcite integrated the two representations by simple concatenation, a better methodology to integrate representations from different embedding spaces is argued by @xcite . the meta-embeddings proposed by @xcite are learned as the intermediate representations generated by various autoencoder variants. thus, building on these two findings, we propose topic-infused deep contextualized representations, a novel data representation technique that uses a concatenated denoise autoencoder to combine deep contextual embeddings with topic information for generating robust document representations. our methodology spawns document representations that subsume the topic information from latent dirichlet allocation @xcite with the contextual embeddings generated by pre-trained roberta model @xcite . we further demonstrate that the proposed methodology achieves improvement for text clustering against the contextual embeddings generated by the pretrained roberta model. in the light of the above discussion, our research makes the following contributions: ## related work traditionally, the research in topic mining and theme extraction from online mental health communities has been focused on the use of probabilistic generative models like the latent dirichlet allocation @xcite and clustering techniques such as the k-means @xcite . @xcite employed lda to extract topics from the internet support group blueboard. results showed that users engaged in discussions with a greater topical focus on experiential knowledge, disclosure, and informational support, a pattern resembling the clinical symptom-focused approach to recovery. in their study, @xcite used the hierarchical dirichlet process (hdp) algorithm to infer latent topics from blog posts of the live-journal (lj) blogging site. the authors applied the non-parametric affinity propagation algorithm to find clusters within the online communities. toulis and golab (2017) compared the recurring themes encountered in private journals with the ones found in the online communities of reddit and found significant similarities in the topics discussed across both the forums. @xcite provide an exhaustive analysis of the thematic overlap, similarity, and difference in online mental health communities of r/depression, r/anxiety, and r/ptsd. their results show a considerable overlap of themes between the mental health groups, attesting that people engaging in such forums face common problems and comorbidity symptoms. since their introduction, transformer-based language models such as bert @xcite have led to impressive performance gains across multiple nlp tasks. recent works show that these contextualized representations can also support type-level clusters, and hence, can be effectively used for modeling topics @xcite . the recent work by @xcite demonstrates that running simple clustering algorithms like k-means on contextualized word representations result in word clusters that share similar properties to the ones generated by lda. an interesting approach is followed by @xcite , where the authors combine the topic models of lda with contextualized word representations of bert for the task of semantic similarity detection. results depict that adding topical information improves performance, especially for examples with domainspecific terms. ## dataset description for this experiment, we selected the r/ptsd foot_0 and r/cptsd foot_1 subreddits which have 54,000 and 97,000 active users, respectively. using the pushift api foot_2 , we crawled all the available posts from these subreddits between 1st august 2015 and 31st july 2020. to ensure that each selected post has community approval, we selected posts that have a minimum of 10 net upvotes. we further filter our dataset by eliminating posts with less than 75% english content foot_3 , posts with less than five words, as well as posts with [deleted], [updated], and [removed] entries. we employed standard text cleaning and normalization techniques for preprocessing the posts, including removing special characters, accented words, wordplays, urls, replacing acronyms with full forms, and expanding contractions foot_4 . this resulted in a comprehensive dataset of 24,930 posts. ## proposed methodology the proposed system consists of two key components: a robust data representation methodology and an efficient clustering algorithm. figure 1 depicts the model architecture. each component is elucidated in detail as follows: ## experimental setup the document topic distributions are generated using the lda mallet python version foot_6 . as the topics of interest centers around entities that are mostly figure 4: calinski harabasz index (chi) comparison in our experiments, each autoencoder is implemented as a single hidden layer neural network. the dimensions d 1 , d 2 , and d 3 are chosen as 10, 379, and 379, respectively. the hidden dimensions are chosen as such so that the topic-infused deep contextualized representations are of dimensions 768, making them comparable with that of roberta. masking noise of 10 percent is applied to the source embeddings before encoding (vincent et al., 2010). leaky rectified linear (leaky relu) (maas et al., 2013) activation is applied to each layer with the default parameter setting. the model is trained end-to-end for 200 epochs, with the adam optimizer (kingma and ba, 2015), a learning rate of 0.001, and a mini-batch size of 128 for minimizing the objective loss. the learning rate is reduced by a factor of 0.1 if validation loss does not decline after 10 successive epochs. the model with the best loss is used for prediction. ## clustering performance evaluation in order to assess the clustering performance, we make use of three measures, namely the silhouette coefficient (sc) @xcite , the calinski-harabasz index (chi) @xcite , and the davies-bouldin index (dbi) @xcite . higher sc, and chi, and lower dbi scores indicate a better separation of the clusters and tighter integration inside the clusters. we compare the clustering performance of three embedding sets, namely: the roberta last layer embeddings, the roberta second last layer embeddings, and our proposed topic-infused deep contextualized embeddings. figures 3 , 4 , and 5 depict the performance comparison of the corresponding three embeddings with respect to the three metrics mentioned above. the y-axis represents the three corresponding metrics, whereas the x-axis represents the umap component variations. the hyperparameter values of the number of neighbours and minimum distance were chosen as 30 and 0.0, respectively, while cosine similarity was used as the metric for computing the distance in the ambient space of the input data. the random state is seeded to an integer value (42). for hdb-scan, the minimum cluster size was selected as 300, with the other parameters set to the default values @xcite . from the comparative analysis, it is evident that the proposed topicinfused deep contextualized representations result in an improved clustering performance across all three metrics. for our study, we choose umap with components 10, as it empirically gives consistent performance across all three metrics. the 2d embedding space after clustering is shown in figure 2 . to reduce the cluster size, we extract the most representative posts or exemplars of each cluster foot_7 . in technical terms, exemplars are the data points that lie at the heart of a cluster, around which the ultimate cluster forms. normalized pointwise mutual information (npmi) @xcite , on the other hand, has been shown to correlated with human judgement @xcite . thus, in this study, we use npmi as an extrinsic measure. experiments by @xcite suggest that clustering contextual embeddings can result in topics with better npmi compared to lda. as evident from the results reported in ## thematic analysis of the clusters for a comprehensive qualitative evaluation, we select the top 100 posts from each clusters which exhibit the highest jaccard index scores foot_9 with that of the top 15 nouns with the highest c-tf-idf scores for each cluster. the qualitative analysis is carried out by the subject expert in clinical psychology to draw out the major discussion themes. total 7 themes were extracted from the 9 clusters generated. they are as follows: abuse stories: cluster 3 reveals self-disclosure stories of abuse that are physical, emotional, or sexual in nature. abusers are mainly parents, siblings, close family members, co-workers, and known people. narrations prominently highlight dysfunctional family dynamics during the victim's childhood as well as the issues related to current family interactions. family dynamics are characterized by alcoholic/ substance abusive parents and abusers with pathological personality traits. posts indicate traumatic experiences, the effect of these experiences, and how they act as triggers. the victim's exposure to the trauma was long and frequent. "my mother was very controlling, scary, abusive and manipulative..." "i've been coming to terms over the past 6 months or so with the fact that my family caused me a significant amount of trauma growing up..." "for as long as i can remember, my father was physically and emotionally abusive..." flashbacks: the primary themes of cluster 6 include flashbacks of traumatic incidences and the emotional disturbances associated with them. feelings of anxiety, panic, nightmares, fear, impulsivity, and anger are prevalent throughout the cluster. narrations reveal the way flashbacks are impeding current life issues. examples include an aversion to touch and sexual experiences, difficulties in romantic relationships, social relationships, daily chores, and work as well as triggering health-related symptoms. working through flashbacks, ways of dealing with it, and help to overcome it are also shared. "...struggling with flashbacks, nightmares, and intrusive thoughts about trauma. can anyone give me reassurance that i'll get through this?..." "...i can stay that way for weeks while i slowly process the old trauma that comes up. i'm in that state right now -just overwhelmed by intense emotional flashbacks..." advice seeking: people have vividly expressed their feelings and are seeking advice about various issues related to ptsd, according to the posts in cluster 8. main themes revolve around advice related to job functioning impacted by ptsd symptoms, social isolation, dependency issues, management of difficult emotions, exhaustion, and frustration. posts in this cluster are brief and direct, seeking advice, help, and support from the reddit community. abuse issues and general ptsd: clusters 7 and 9 mostly contains posts about abuse stories and ptsd in general. the major themes are related to the parent-child issues, insecure parenting styles, emotional and physical abuse, the effect of childhood trauma, loss and grieving for not having healthy parental relations, the impact of insecure parenting, disclosure of trauma incidences, and sharing recovery tips. following are a few illustrations: ".. i hate the days where you long for the childhood that you never had and support you never received. for the belonging and "home" that you never had ..." "... my last therapy session had me divulging my experiences as a young child and what life was like for me back then..." ## discussions cluster structure is in accordance with the formal diagnostic criteria of ptsd and c-ptsd. abuse stories reflected in cluster 1 corresponds to the criterion-a of ptsd in dsm-5 @xcite , that states the experience of trauma. out of the six symptom cluster proffered by icd-11 foot_10 for cptsd (re-experiencing, avoidance, hypervigilance, emotional dysregulation, interpersonal difficulties, and negative self-concept), cluster 6 adequately corresponds to re-experiencing of flashbacks, while cluster 5 is consistent with emotional dysregulation. parent-child relationships portrayed in cluster 2 corresponds to etiological factors. dysfunctional parent-child relation in ptsd has been widely confirmed in the clinical psychology literature @xcite @xcite . on the other hand, clusters 5, 6, and 7 characterizing therapy experience, working through trauma, and advice, respectively, attribute to the treatments and interventions aspect of ptsd. in conclusion, it can be said that these clusters reveal salient features of ptsd. although the clusters more or less convey their discrete independent themes, there exist many common topics running throughout the clusters. the recurrent expressions observed throughout the clusters pertain to personal and social life (family, parents, friends, person, relationships), daily grind (work, home, job, place), and temporal indicators (day, time, year, month, today, week). furthermore, words encompassing cognition (thoughts, think, feel, feeling, lot, hard), emotional and affect expressions (happy, love, good, anxiety, kind, bad, hate), and inhibition expressions (avoid, deny, escape) are perpetual throughout the posts. ## limitations our work has its own drawbacks and limitations. the problem statement at hand is of fine-grained clustering rather than coarse-grained clustering, thus, making it difficult to draw out interpretable topics of discussion. this is attested by the low npmi scores in ## conclusion and future work in this research, we introduce topic-infused deep contextualized representations, a robust data representation methodology that successfully integrates topical information with contextualized embeddings. we collect and analyze large-scale data pertaining to the online discourse on reddit, centered around ptsd and cptsd, and perform densitybased clustering to draw out the prominent clusters and analyze the themes of discussion present in them. despite the perpetual semantic and thematic similarities amongst the posts in the corpus, our methodology, to some extent, is able to draw out the underlying, fine-grained, latent clusters. the qualitative analysis of each cluster revealed the characteristic themes and salient features of ptsd and cptsd, consistent with the clinical psychology literature. through the lens of social media, this study delineates a deeper understanding of ptsd and c-ptsd, fostering further research in early detection of mental illnesses, identification of highrisk groups, enhanced mental health patient education programs, better diagnostic and therapeutic theory building, as well as an improved understanding of the underlying design of the online mental health communities. the future work of this research can take multiple directions. from an nlp standpoint, the robustness of the proposed methodology should be examined by testing them on various other benchmark nlp tasks such as semantic textual similarity, word analogy, and text classification, to name a few. other variants of autoencoders and objective losses could be employed to facilitate tighter integration of topical information with the contextualized embeddings. from the mental health and clinical psychology perspective, such research can be easily extended to other online mental health communities to draw useful insights.