The Best of Both Worlds: Combining Engineered Features with Transformers for Improved Mental Health Prediction from R eddit Posts (2022)

Abstract: In recent years, there has been increasing interest in the application of natural language processing and machine learning techniques to the detection of mental health conditions (MHC) based on social media data. In this paper, we aim to improve the state-of-the-art (SoTA) detection of six MHC in Reddit posts in two ways: First, we built models leveraging Bidirectional Long Short-Term Memory (BLSTM) networks trained on in-text distributions of a comprehensive set of psycholinguistic features for more explainable MHC detection as compared to black-box solutions. Second, we combine these BLSTM models with Transformers to improve the prediction accuracy over SoTA models. In addition, we uncover nuanced patterns of linguistic markers characteristic of specific MHC.

## introduction the last decade has seen a surge in digital mental health research aimed at using linguistic cues from social media data to predict mental health conditions @xcite . the ultimate goal of this research branch is to enable people to lead healthy lives and to support healthcare professionals in the diagnosis and treatment of mental disorders. data from social media platforms are particularly compelling to the research community because of their scale and deep embedding in contemporary culture. the computational research on such data has yielded new insights into population mental health and has shown promise for incorporating data-driven analytics into the treatment of psychiatric disorders (see @xcite for recent general overviews of this research; see @xcite for reviews focusing on methods and data used for mental health status on social media). the current state of the art (sota) in mhc detection from text data in social media is achieved by approaches that employ deep learning techniques that use pre-trained word embeddings, in particular transformer-based models. at the same time, there are increasing calls to move away from such models toward explainable models for critical industries @xcite , and the black-box nature of such models is increasingly being recognized as a major hurdle in the use of ai models in clinical practice @xcite . effective supporting the diagnosis and treatment of mental disorders therefore requires both accurate and interpretable models. in this paper, we make the following contributions to the dynamic area of research on detecting mental health conditions in social media texts: first, we present attention-based blstm models that leverage the within-text distributions ('textual contours') of a large array of engineered language features to predict the presence of six mental health conditions (mhc) from user posts on reddit. we show that these models can perform competitively with sota models (drop in @xmath0, while maintaining explainability. second, we demonstrate that the accuracy of mhc detection can be increased considerably by integrating these models with transformer-based models. in addition, we uncover nuanced patterns of linguistic markers characteristic of specific mhc. ## related work in this section, we focus on the state-of-the-art approaches to mhc detection on the two reddit datasets used in this paper, i.e smhd @xcite and dreaddit @xcite (for details see section 3.1). on the smhd dataset the current state-of-the-art approach is based on a hierarchical attention network (han) @xcite . the proposed model uses two layers of bidirectional gru units with hidden size of 150, each of them followed by a 100 dimensional attention mechanism. the first layer was set up to encode posts, whereas the second one encodes users in terms of a sequence of encoded posts. the input layer was initialized with 300 dimensional glove word embeddings @xcite . the output layer is 50-dimensional fully connected network, with binary cross entropy as a loss function. @xcite perform experiments with the number of posts per user available to the model @xcite @xcite @xcite to examine the amount of data needed for reasonable performance. their best-performing model achieved an average of 67.56% f1 across the five mhc. on the dreaddit dataset the current state-of-the-art approach is based on a 'emotion-infused' bert model @xcite . in the best-performing model presented in the paper, the bert representation was first augmented with emotion knowledge by fine-tuning it on the goemotions dataset @xcite before applying it to the stress detection task. the model reached 80.25% f1 in binary stress classification, performing slightly better than a bert baseline model. it is interesting to note that both sota papers highlight the importance of explainability in mhc detection and suggest ways to increase the interpretability of the predictions of their models: @xcite perform attention weights analyses to identify posts, and words or phrases in those posts, that are relevant for classification, whereas @xcite use lime to identify the most important words used their models for stress classification and mapping these to liwc categories @xcite . ## modeling approach we conducted experiments with a total of five models: (1) a fine-tuned bert model @xcite , (2) a fine-tuned roberta model @xcite , (3) a bidirectional long short-term memory (blstm) classifier trained on measurements of psycholinguistic features described in section 3.2, and ( 4 ) and ( 5 ) two hybrid models integrating bert and roberta predictions with the psycholinguistic features. for (1) and ( 2 ) we used the pretrained 'bert-base-uncased' and 'robertabase' models from the huggingface transformers library (wolf et al., 2020) each with an intermediate blstm layer with 256 hidden units @xcite . for (3) -the model based solely on psycholinguistic features, we constructed a 4-layer blstm with a hidden state dimension of 1024. the input to that model is a sequence cm n 1 = (cm 1 , cm 2 . . . , cm n ), where cm i , the output of cocogen for the ith sentence of a post, is a 435 dimensional vector and n is the sequence length. to predict the labels of a sequence, we concatenate the last hidden states of the last layer in forward ( -→ h n ) and backward directions ( ←h n ). the result vector of concatenation h @xmath1, whose activation function is rectifier linear unit (agarap, 2018). the output of this is then passed to a fully connected layer fc with relu activation function and dropout of 0.2 and it is finally fed to a final fc layer. the output is finally passed through sigmoid function and finally a threshold is used to determine the labels. we trained these models for 100 epochs, with a batch size of 256 and a sequence length of 5. the architecture of the hybrid classification modelsmodels (4) and ( 5 ) -consists of two parts: (i) a pretrained transformer-based model with a blstm layer and fc layer on top of it and (ii) the psycholinguistic features of the text fed into a blstm network and a subsequent fc layer. the fc layers of both parts take the concatenation of last hidden states of the last blstm layer in forward and backward direction. we concatenate the outputs of these layers before finally feeding them into a final fc layer with a sigmoid activation function. the model used to generate predictions for the test set was the roberta-psyling hybrid model with the following configuration: 2-layer blstm, 256 hidden units and a dropout of 0.2; blstm-psyling: 3-layers, hidden size of 512 and dropout 0.2. we trained this model for 12 epochs, saving the model with the best performance (f1-score) on the development set. the optimizer used is adamw with a learning rate of 2e-5 and a weight decay of 1e-4. structure diagrams of the model based solely on psycholinguistic features and the hybrid architectures are presented in figures 2 and 3 in supplementary material. all models were trained using 5-fold cv of the training data as base classifiers and model stacking was performed using logistic regression as a meta-learner to adaptively combine the outputs of the base classifiers. ## results and discussion an overview of the results of our models in comparison to those reported in the previous studies reviewed above is presented in to further investigate the differences in the linguistic markers among mhc, we ran one-way anovas comparing the features across the six mhc and the control group for each of the 435 linguistic features. prior to being entered in the models, all feature scores were z-score normalized and bonferroni correction was used to correct for family-wise type i errors. follow up tukey's hsd post-hoc test were run to measure the differences between all mhc pairs. the results revealed significant differences across groups for 337 of the linguistic features (see table ?? in the appendix). we focused on the linguistic features with the greatest variance among the groups, as indicated by the highest f-statistics (@xmath2, and conducted hierarchical agglomerative cluster analyses over mhc and linguistic features to derive the linguistic profiles associated with the six mhc groups and the control group. the analyses show that the control group is distinctly separated from all mhc. furthermore, while each of the six mhc were characterized by distinct patterns of language use, the linguistic markers of stress were clearly separated from those of the five self-reported diagnosed mhc. in light of space limitations, we present here only some selected general patterns (for details, see figure 4 and