CASE : Efficient Curricular Data Pre-training for Building Assistive Psychology Expert Models (2024)

Abstract: The limited availability of psychologists necessitates efficient identification of individuals requiring urgent mental healthcare. This study explores the use of Natural Language Processing (NLP) pipelines to analyze text data from online mental health forums used for consultations. By analyzing forum posts, these pipelines can flag users who may require immediate professional attention. A crucial challenge in this domain is data privacy and scarcity. To address this, we propose utilizing readily available curricular texts used in institutes specializing in mental health for pre-training the NLP pipelines. This helps us mimic the training process of a psychologist. Our work presents CASE-BERT that flags potential mental health disorders based on forum text. CASE-BERT demonstrates superior performance compared to existing methods, achieving an f1 score of 0.91 for Depression and 0.88 for Anxiety, two of the most commonly reported mental health disorders. Our code and data are publicly available.

## introduction mental health plays a vital role in the lifestyle of individuals worldwide. as living situations become complex, individuals around the world have expressed signs of disorders more frequently than before. the demography of mental health disorders affects every age group ranging from elementary school kids to senior citizens. in india, approximately 10.6% of the indian population is estimated to have mental disorders @xcite . there is a huge imbalance in the number of psychologists present to address the issues in comparison to the potential number of patients seeking help. garg et. al @xcite state that for every 100,000 population, there are only 0.75 professionals available in india while the required number is 3. there is a similar world statistic with only 0.5 professionals present for 100,000 population while the desired value is 2. this imbalance is further worsened as such professionals are not present in every location possible and at all times. to address the above issues, many mental health discussion forums emerged in recent times where users can express their issues and have a discus-sion. this can be in the form of a closed chat with humans on the other end or a chatbot along with a public forum with anonymized public posts on the front end but tracked by the platform in the backend. counselchat foot_1 , mind.org foot_2 , and manochikitsa foot_3 are some of the platforms that provide such facilities to users. however, the emergence of such a platform brings in the challenge of disorder identification as there is no restriction to the type of text that can be sent. screening such texts in public forums becomes a challenging task, especially in a manual setting introducing the possibility of human error where one might miss a serious post. an automated filtration strategy can help where the suspected posts can be presented to the human in the loop for further scrutiny of the filtered posts. in this work, we aim to present a discriminative model that flags potential mental health disorders based on forum text. this work intends not to replace a mental health professional but to be used as an assistant for the preliminary screening of patients, especially in a public forum setting where previously a manual screening may have been used. however, this manual screening is costly, both in terms of human labor and time, which in the case of detecting severe cases might not be present for screening thousands of public mental health forums. in summary, our work contributes in the following ways: our work builds on the philosophy of how a person gains curricular knowledge. our approach is analogous to the training of a psychology student as mandated by the american psychology association (apa) (apa, 2019) from a theoretical standpoint. building on textbooks are all you need @xcite where they use highquality data for building a small, yet effective generative model for python programming, we pre-train off-the-shelf models using high-quality curricular data provided by psychologists in academia. this high-quality data helps us to tackle domain data scarcity by using a small amount of data to pre-train and finetune state-of-the-art (sota) discriminative assistive psychology expert models. we believe our contribution in the form of curriculum learning for discriminative models can be used in other domains also, given the amount of curriculum data present in the domain is respectable. we compare our work on several mental health detection datasets with several other similar works like mentalbert @xcite , psych-search (nlp4good, 2021), biobert @xcite and clinicalbert @xcite . the structure of the rest of this article is as follows: section 2 presents the background about the baseline models, and related literature and methods used in our approach. section 3 discusses the dataset collection process used for pre-training and fine-tuning all model instances discussed. section 4 discusses our approach for case-bert. section 5 presents the set of experiments performed and evaluation metrics used to support our hypothesis. lastly, section 6 discusses the results obtained along with ethical concerns and presents a line of future work associated with our work. ## background and related work prior work uses data sources like mental health subreddits @xcite , abstracts of scientific articles on psychiatry (nlp4good, 2021), @xcite and full texts of scientific articles on psychiatry @xcite . ## method as mandated by apa (apa, 2019) clinical psychology student undergoes years of training where they refer to various curricular documents to learn the mandated clinical psychology theory, like textbooks, as prescribed by their institution. this is then followed by supervised clinical experience in handling patients as subordinates, in the form of fellowships where they work to diagnose sev-eral (at least 100) patients; and finally becoming practitioners. our approach tries to mimic this. we consider the off-the-shelf bert @xcite model similar to a student graduating from high school, having a general knowledge of the world. next, we use curricular data mentioned in section 3.1. after this pre-training, we consider our model to be a psychology student having theoretical knowledge of psychology and related fields like psychotherapy. however, it does not have practical experience with patients. for this purpose, we further fine-tune it on the counselchat @xcite , dreaddit (turcan and mckeown, 2019) and depression_reddit @xcite . ## results and discussion as the datasets used suffer from a large class imbalance, we focus on the f1 score and recall. studying recall in such a use case is important as we need to avoid false negatives and miss people who suffer from a disorder. having said that, we also need @xcite and dreaddit @xcite . the best-performing numbers are highlighted in bold while the second best are underlined. to maintain a high value for precision and avoid false positives so that we minimise the stress and stigma that might, unfortunately, be associated with a positive mental health disorder diagnosis. the main aim of this work is to introduce our pretraining philosophy and verify the same for the use case of mental health disorder detection. note that our aim is not to achieve the best state-of-the-art numbers on these datasets, but we still manage to have an advantage in terms of model performance over previous work. in all of the experiments we observe that the generative models fail to perform competitively. specifically, we can see that all the models have a higher recall than precision. this indicates that these models are more sensitive towards signs of mental health and are biased towards flagging the users as having a mental health disorder. an argument can be made that the discriminative models are fine-tuned specifically for these datasets leading to their improved performances, however, finetuning these llms with many billions of parameters is far more complex and compute-intensive than fine-tuning discriminative models which have little above a hundred million parameters. we can see how we could easily train the discriminative models on publicly available gpus on kaggle. doing the same for the llms we compared against is not feasible on kaggle. discussion in this work, we present a pretraining method for a discriminative model. by using curricular data for pre-training, we present models -case-bert-base and case-bert-small, two discriminative bert-based models that flag potential mental health disorders based on forum texts. we tackle the lack of granular data used for training classification pipelines and large corpora for pre-training discriminative pipelines by using curricular text data. this method of pretraining achieves much better performance with a tiny fraction of the data as compared to the previous sota methods. our work is a significant step towards building small expert models that need a small amount of data while not requiring a massive amount of computing power to deploy. we believe that the approach of using curricular data to battle data scarcity is a big challenge and has greater implications in building machine-learning solutions for domains where data privacy and security are critical. these are qualities that are essential for a model that is deployed on a mental health forum. having a small model is also advantageous as it reduces or at best avoids the requirement of expensive hardware to run on.