Evaluating Contextual Embeddings and their Extraction Layers for Depression Assessment (2022)

Abstract: Many recent works in natural language processing have demonstrated ability to assess aspects of mental health from personal discourse. At the same time, pre-trained contextual word embedding models have grown to dominate much of NLP but little is known empirically on how to best apply them for mental health assessment. Using degree of depression as a case study, we do an empirical analysis on which off-the-shelf language model, individual layers, and combinations of layers seem most promising when applied to human-level NLP tasks. Notably, we find RoBERTa most effective and, despite the standard in past work suggesting the second-to-last or concatenation of the last 4 layers, we find layer 19 (sixth-to last) is at least as good as layer 23 when using 1 layer. Further, when using multiple layers, distributing them across the second half (i.e. Layers 12+), rather than last 4, of the 24 layers yielded the most accurate results.

## introduction over the past decade natural language processing (nlp) has increasingly set its sights on interdisciplinary tasks, notably those within the computational social sciences @xcite @xcite . as more and more language has been generated on social media sites such as facebook, twitter, and reddit, researchers have had a wealth of personal discourse available to them that spans across thousands of users. many researchers focus on applying these social media datasets to predict user demographics, personality, or mental health @xcite @xcite . those predicting facets of mental health, such as depression and suicide risk, can help an over-burdened mental health industry by using automated screening @xcite . often these automated tools can be applied to forums where a user is an active member and their account could be flagged to be brought to the attention of a moderator. thus, a personalized and potentially early intervention could be provided to the user in question. here, we investigate one prominent aspect of mental health: degree of depression (ddep) as measured by answers to an online questionnaire administered to facebook users. depression assessment of social media users is of interest for the following reasons: (1) depression is often highly correlated with suicidal tendencies @xcite with deaths by suicide on the rise @xcite and ( 2 ) automated assessment of depression is of high importance as it is often an under-diagnosed ailment, where such predictions could be useful to screen individuals who are at risk @xcite . while many recent nlp pipelines have moved onto leveraging large pre-trained language models based on the transformer architecture @xcite , applying these models to human-level analysis, such as predicting a person's states or traits, has received little attention. even the use of extracted embeddings, often called contextual embeddings, has yet to be fully explored in this level of analysis @xcite . we expand this area of research by investigating how best to leverage the individual layers of off-the-shelf transformer models for depression assessment. notably, we are interested in going beyond just a single layer and propose a greedy algorithm for selecting layers to extract contextual embeddings and aggregate them for large user-level embeddings. ## related works one of the downsides when modeling mental health data is often that it is very small, with only a few hundred participants per study @xcite . however, it is sometimes possible to get around this by using data from social media websites where participants can choose to opt in to share past language data and take a small survey or questionnaire @xcite . @xcite applies this technique to facebook users and evaluates their ddep over a continuous scale (1-5) rather than bucketing users into classes such as mild/moderate/severe. even somewhat recent human-level models in nlp have used bag-of-words style approaches for prediction @xcite , while other areas such as word or document-level tasks have adopted contextual embedding representations @xcite @xcite . as these are often output from very large models, with hundreds of millions or more parameters, they are able to encode syntactic and semantic information that transfer to downstream tasks either through word or sentence embeddings @xcite . while there has been some work applying contextual embeddings and transformer language models to human-level predictions, the most in depth has been v ganesan et al. (2021) who investigated the use of contextual embeddings in low-data scenarios across various areas including mental health, demographics, and personality assessment. however, they only focus on using the base-size variants with an emphasis on dimensionality reduction techniques to apply contextual embeddings to small datasets (@xmath0. here, we work with a medium size dataset of 3 million facebook posts across 25 thousand users and apply both base and large sized language models, as well as investigate layer selection beyond using just the second to last layer of the model. ## methods task: a person's degree of depression score is estimated by their response to a subset of neuroticism questions on a personality assessment through facebook's mypersonality app @xcite . the responses were on a scale of 1 to 5 and averaged together to represent a person's overall degree of depression. here, we formulate the task of depression assessment as building a single user representation where each status is processed through a language model as a sentence and then all words from a user are avg-pooled. we evaluate our models using mean squared error(mse) and disattenuated pearson r(r dis ) to account for questionnaire reliability @xcite . we perform all experiments using the dlatk @xcite library. transformer language models: from the wide selection of general purpose language models, we select the following: xlnet, roberta, albert and bert @xcite @xcite . these models are chosen as they cover common language model types (e.g. autoregressor vs autoencoder), have been pre-trained on various corpus sizes, and in the case of albert offer a more lightweight footprint in terms of total model parameters. when comparing which language model to perform our layer analysis on, we first evaluate performance using only the second to last layer on our held-out test set. this allows us to deduce which model may lead to better application to aggregate human-level predictions. layer selection: to decide on which layers to extract for our final model, we perform a 10-fold cross-fold validation, for each individual layer or combination of layers. first we select the best performing layer, once found, we then concatenate all other layers to find the best 2-layer combination. this process is iterated on until we reach a number of layers where we cease to see a performance increase via the cross-folds. once the best model hid. size r dis mse roberta-b l11 768 .542 .7497 roberta-l l23 1024 .543 .7476 distilroberta l5 768 .533 .7545 figure 1 : layer-wise mean squared error performance across the 10-fold validation set with standard error shown by the shaded region for both roberta-base and large. at lower layers (3-6), roberta-base shows a much lower error rate. however at layer 13 and higher of roberta-large there is lower error beyond any available base layer. regression: our model of choice is a regularized linear regression (ridge) with input being the mean aggregate of extracted contextual embeddings. to find the regularization parameter α, we use a 10fold cross-validation technique searching between 10 and 1 million, increasing by powers of 10 each time, then selecting the α that gave the lowest mean squared error. a simple predictive model is chosen to highlight the improvements from the features themselves rather than any specific network architecture. ## dataset & baseline dataset: the dataset is comprised of facebook users who opted in to share their status updates between 2009 and 2011 and completed a personality questionnaire @xcite . there are 25,000 train users and 1,000 test users which are then filtered down to those who wrote at least 1,000 words across all of their status updates. the final result is a training set of 17,599 and test set of 986 users. baseline: we compare to the proposed model of @xcite which leverages both open-vocab and count based lexicons. notably, the model is trained on 1 -3 grams, a 2000 dimensional social media lda topic vector, lexical inquiry and word count (liwc) lexicon, and nrc sentiment lexicon @xcite . we compare our models both to the reported scores in the original publication and to a version we recreated, referred to as open-ridge. ## evaluation our recreated open-ridge came within .019 r dis of the original work, however, both the recreated and original model are outperformed by both bert and roberta base variants, as shown in we also compare against possible variants of roberta, which offer a computation versus performance trade-off, roberta-large (24 layers) and distilroberta (6 layers) in next, we explore the question of how many layers should be used as well as which layers to extract in order to build a user representation. for this, we apply our layer selection technique based on empirical results of the cross-folds. we show results for the top 10 best combinations per layer amount in lastly, we compare our optimized extraction models to the standard approaches on the held-out test set; shown in the 5-layer version has a small improvement in both metrics and is found to be significant(p=.02) compared to our optimized 4-layer model. for the 6-layer model we see an expected drop in performance, based on cross-fold analysis, suggesting that the additional layer has hurt the model's ability to generalize. ## conclusion with many tasks in nlp focused around humanlevel prediction, methods that can use state-of-theart, off-the-shelf models in the best way are of interest to the community at large. in this work, we found that applying pre-trained transformer language models to depression assessment benefited from non-standard extraction techniques. fur-ther, applying a straight forward empirical analysis of layer performance could lead to noticeable boosts in downstream applications. ultimately, we achieved sate-of-the-art performance of r @xmath1.554 and m @xmath2.7206 using a 5-layer user representation from roberta-large.