Towards Intelligent Clinically-Informed Language Analyses of People with Bipolar Disorder and Schizophrenia (2022)

Abstract: NLP offers a myriad of opportunities to support mental health research. However, prior work has almost exclusively focused on social media data, for which diagnoses are difficult or impossible to validate. We present a first-of-its-kind dataset of manually transcribed interactions with people clinically diagnosed with bipolar disorder and schizophrenia, as well as healthy controls. Data was collected through validated clinical tasks and paired with diagnostic measures. We extract 100+ temporal, sentiment, psycholinguistic, emotion, and lexical features from the data and establish classification validity using a variety of models to study language differences between diagnostic groups. Our models achieve strong classification performance (maximum F1=0.93-0.96), and lead to the discovery of interesting associations between linguistic features and diagnostic class. It is our hope that this dataset will offer high value to clinical and NLP researchers, with potential for widespread broader impacts.

## introduction schizophrenia and bipolar disorder have been associated with observable language patterns in clinical and sociolinguistic studies @xcite @xcite . some computational studies have sought to replicate these findings or derive novel clinical insights using automated analyses driven by natural language processing techniques @xcite . effectively identifying features associated with these disorders or providing diagnostic aid offers substantial potential for real-world impact @xcite @xcite . however, these studies to date have been constrained by limitations in dataset size and availability @xcite @xcite @xcite , @xmath0, restricting the extent to which they can produce meaningful or generalizable conclusions. we address this gap by introducing a new, large (@xmath1, people with schizophrenia (sz), and healthy control (hc) subjects. we also establish preliminary benchmarking models for automatically distinguishing between these groups using interpretable linguistic features, achieving promising proof-of-concept ranging from 70-96% accuracy in one-versus-one discrimination between subject groups. finally, we conduct preliminary analyses across a large feature set to identify potential linguistic correlates with these groups. our key contributions are as follows: this research was jointly conducted by an interdisciplinary team of researchers from psychiatry and computer science departments to foster translational impact in both communities @xcite . we hope that the data and insights provided will pave the way for new research and subsequently exciting new clinical and computational findings in this domain. ## background social media data has dominated research at the intersection of computational linguistics and clinical psychology @xcite . its popularity owes partially to its size and availability @xcite @xcite . high rates of social media usage are also evident in users who face mental health concerns @xcite , with associations observed between social media use and the occurrence of psychosis @xcite @xcite , mood disorders @xcite , personality disorders @xcite , eating disorders @xcite , and obsessive compulsive disorder @xcite . the observed connections between social media and mental health have triggered meaningful work into leveraging this data for downstream tasks (aich and parde, 2022), such as the automated detection of depression @xcite , schizophrenia and psychosis @xcite , and suicide risk prediction @xcite . reddit posts, a popular resource for this work due to their semi-anonymity and length @xcite , have been used to identify stress @xcite , eating disorders @xcite , depression @xcite , and suicide @xcite , among others @xcite . twitter has been used for the detection of depression and post-traumatic stress disorder @xcite , schizophrenia @xcite , anti-social behavior @xcite , suicidal ideation @xcite , and stress @xcite . most social media datasets for mental health tasks are annotated along binary or linear scales and label users based on analysis of a set number of posts. annotations may be provided by trained human annotators @xcite , annotators with clearly referenced domain expertise (e.g., @xcite 's work employing a clinical psychiatrist and a grad-uate student from northwell health's early treatment program), user disclosures of mental health conditions @xcite @xcite , and crowdsourcing services @xcite . annotation schema for some mental health conditions can be subjective, causing varied inter-annotator agreement. for example, @xcite reported a cohen's kappa score of κ=0.81, whereas @xcite reported a much lower agreement of κ=0.47 for their dataset of stressed and unstressed social media users. turcan and mckeown (2019b)'s dataset also offers an example of how fuzzy label boundaries can affect annotation quality-it is well established that stress is often temporary @xcite ; hence, post labels do not always equate to a user's mental state. finally, independent decision-making when selecting sources may influence annotation outcomes. although social media data has been leveraged for a variety of mental health tasks, data accessibility remains an enormous challenge. in their analysis of more than 100 mental health datasets, @xcite found only three to be available without any restrictions. they found @xmath2. 1 of those that were described in some capacity (48), 13 were removed from public records or limitations made them unavailable. out of the 35 that remained, 12 needed signed agreements or institutional review board (irb) approvals, 18 had instructions and apis to reproduce them, 2 could be obtained directly by emailing the authors, and as mentioned, 3 were available without restrictions. these trends have also been observed on a broader scale with other healthcare data in nlp studies @xcite . moreover, for publicly accessible data, the inherent subjectivity of many mental health annotation tasks and the frequent reliance on user selfdisclosures means that many "gold standard" labels are imperfectly assigned. most datasets fail to capture nuances of mental health @xcite , and medical self-disclosures may be indirect @xcite . for example, @xcite 's dataset labels the following sample as yes, but provides little clarity regarding the user's diagnosis: i have schizophrenia/depression. i am trying to become better by exercise and working i have a job xoxo i love saturday xx issues related to fairness, gender, balance, and representation of racial and ethnic biases in social media datasets have also been found @xcite . we seek to address many of these limitations by providing a publicly accessible dataset of manually transcribed interactions between individuals with clinically diagnosed mental health conditions and trained clinicians. we also provide dataset transparency regarding representational balance through validity of diagnoses and descriptive statistics. ## feature analyses since we computed features across three subject pools (sz, bd, and hc), we analyzed feature correlations, patterns, and trends across subject groups. this investigation provides a starting ground for the more detailed follow-up studies that our new dataset is designed to enable. we make our analysis and visualization scripts publicly available to lower the barrier for others to pursue these studies. 3 in figures 2 and 3 , we present violin plots illustrating score distributions across selected features from major feature groups described in §4.2. we examine trust emotion features (figure 5a ), herdan measures of lexical diversity (figure 5b and 2b ), mean time per dialogue (figure 6a and 3a), and interpersonal conflict features from liwc 2022 (figure 6b and 3b ). class labels are represented using the numeric signifiers @xmath3, @xmath4, and @xmath5, and colors blue, orange, and green, respectively. due to space restrictions we present plots based on the scene 2 transcripts here, and include plots representing the same features from scene 1 as supplemental content in appendix b (figures 5 and 6 ). we observe that hc subjects exhibit larger overall ranges of lexical diversity and trust language than sz or bd subjects (figure 2 ). sz subjects exhibit lower trust scores, and bd subjects exhibit a bimodal score distribution with two large frequency centers (figure 5a and figure 2a ). this differs from patterns associated with lexical diversity. we observe that bd subjects have a single concentrated distribution of mass slightly above a herdan score of 0.85. sz subjects exhibit a similar mean herdan score, but with a wider score distribution. when examining mean time, we observe that both hc and sz subjects have slightly bimodal score distributions, with sz subjects also having the widest score range (figure 6a and 3a ). bd subjects have a single frequency center and relatively consistent frequency spread from 10-30 seconds. finally, we observe that interpersonal conflict features are concentrated near scores of 2 for all subjects, although sz subjects show the largest score range with a relatively large share of subjects with scores of 4 or greater (figure 3b and 6b ). in figure 4 , we present pairwise feature correlations among six selected features across our five broad feature categories: mean time, positive sentiment, liwc analytic score, anger score, herdan lexical diversity, and liwc lack score. 6 we study and compare pairwise correlations between members of different subject groups, with feature correlations for hc, bd, and sz subjects shown in figures 4a , 4b , and 4c, respectively. we observe weakly positive correlations between analytic scores and positive sentiment among hc subjects, but very weakly (bd) to weakly (sz) negative correlations between this same feature pairing among subjects in other groups, suggesting a stronger relationship between logic and optimism in control subjects compared to subjects with bipolar disease or schizophrenia. interestingly, we also observe stronger positive correlations between anger and mean time, as well as between lexical diversity and positive sentiment, in sz subjects than in hc or bd subjects. hc subjects have weakly negative correlations between lexical diversity and positive sentiment. ## classification task to establish learning validity of our dataset, we designed a simple task to predict subject group membership. specifically, we conduct binary classification experiments to discriminate between two classes from the set of hc, sz, and bd subjects. this also creates an additional avenue through which group-level language behaviors can be analyzed (e.g., through learned feature weights). we experiment with both classical ( §6.1) and transformer-based ( §6.2) models. ## conclusion publishing language data collected in clinical settings that is paired with validated psychiatric diagnoses is an essential first step towards realizing more realistic, medically relevant nlp applications in the mental health domain. in this work, we take that step and describe our new corpus developed in close consultation between nlp and psychiatric researchers and clinicians. the corpus includes manually transcribed interactions between clinical interviewers and healthy control subjects or those with diagnosed schizophrenia and bipolar disorder. we describe all data collection procedures, extract a wide range of promising linguistic features from the data, and conduct an extensive first set of analyses to document trends in linguistic behavior among the sz, bd, and hc subject groups. we show that linguistic diversity manifests itself in various ways across subject populations. we hope that our work will diversify nlp research in the mental health domain beyond social media settings, and that it will open the door for more clinically valid studies of language behavior associated with diagnosed psychiatric conditions. all features extracted for this work are freely available on github and can be downloaded without any further permission. 7 the de-identified transcripts can be downloaded from the national institute of mental health data archive, in keeping with the terms of our nih reporting requirements and the corresponding research grant that funded this work. 8 in the future, we plan to extend our study to also investigate spoken language and acoustic properties from the collected audiorecordings.