MANTIS at SMM 4 H ’2022: Pre-Trained Language Models Meet a Suite of Psycholinguistic Features for the Detection of Self-Reported Chronic Stress (2022)

Abstract: This paper describes our submission to Social Media Mining for Health (SMM4H) 2022 Shared Task 8, aimed at detecting self-reported chronic stress on Twitter. Our approach leverages a pre-trained transformer model (RoBERTa) in combination with a Bidirectional Long Short-Term Memory (BiLSTM) network trained on a diverse set of psycholinguistic features. We handle the class imbalance issue in the training dataset by augmenting it by another dataset used for stress classification in social media.

## introduction the global increase in social media use over the past decade has afforded researchers new opportunities to mine health-related information that can ultimately be used to improve public health. the social media mining for health applications (smm4h) shared task involved ten natural language processing challenges of using social media data for health research @xcite . in our submission to the task targeting the classification of self-reported chronic stress on twitter (task 8), we built hybrid models that combine pre-trained transformer language models with bidirectional long short-term memory (bilstm) networks trained on a diverse set of psycholinguistic features. ## data the twitter data provided by the organizers of task 8 comprised of a total of 4,195 tweets whose distributions over training, development and testing sets are shown in ## description of system architecture we conducted experiments with a total of five models: (1) a fine-tuned bidirectional encoder representations from transformers (bert) model @xcite , (2) a fine-tuned roberta model (robustly optimized bert pre-training approach) @xcite , (3) a bidirectional neural network classifiers trained on measurements of psycholinguistic features described in section 2.2, and (4) and ( 5 ) two hybrid models integrating bert and roberta predictions with the psycholinguistic features. for each model, we performed experiments with and without data augmentation. for (1) and ( 2 ) we used the pretrained 'bert-baseuncased' and 'roberta-base' models from the huggingface transformers library @xcite each with an intermediate bilstm layer with 256 hidden units (al-omari et al., 2020). for (3) -the model based solely on psycholinguistic features, we constructed a 2-layer bilstm with a hidden state dimension of 32. the input to that model is a sequence cm n 1 = (cm 1 , cm 2 . . . , cm n ), where cm i , the output of cocogen for the ith sentence of a document, is a 435 dimensional vector and n is the sequence length. to predict the labels of a sequence, we concatenate the last hidden states of the last layer in forward ( -→ h n ) and backward directions ( ←h n ). the result vector of concatenation ## results an overview of the performance metrics on the validation set is presented in training the models on the augmented data led to an average increase in performance of 3.8% f1 over the non-augmented data. highest performance on the validation set (@xmath0. error inspection indicated that the the majority of errors are false positives (see ## conclusion we developed hybrid classification systems for the detection of self-reported chronic stress that integrate pre-trained transformer language models with bilstm networks trained on a diverse set of psycholinguistic features. our experiments show that such hybrid models significantly outperform base transformer models for both augmented and nonaugmented data.