NCUEE - NLP @ SMM 4 H ’22: Classification of Self-reported Chronic Stress on T witter Using Ensemble Pre-trained Transformer Models (2022)

Abstract: This study describes our proposed system design for the SMM4H 2022 Task 8. We fine-tune the BERT, RoBERTa, ALBERT, XLNet and ELECTRA transformers and their connecting classifiers. Each transformer model is regarded as a standalone method to detect tweets that self-reported chronic stress. The final output classification result is then combined using the majority voting ensemble mechanism. Experimental results indicate that our approach achieved a best F1-score of 0.73 over the positive class.

## introduction the social media mining for health application (smm4h) shared tasks involve nlp challenges on social media data for health research. we participated in the smm4h 2022 task 8 @xcite , focusing on automatically differentiating tweets that exhibit self-reported chronic stress (annotated as "1") from those that do not (annotated as "0"). chronic stress is a kind of long-term physiological or psychological response which may damage mental health. deep sparse neural networks were used to detect stress on social media data @xcite . factor graph models were combined with convolutional neural networks to detect user stress @xcite . an emotion-infused language model was proposed to facilitate stress detection @xcite . emotion recognition was investigated as an auxiliary task to improve stress detection @xcite . recently, novel transformer-based neural networks have achieved highly promising results in many nlp tasks. this trend motivates us to explore the use of pre-trained transformer models to detect chronic stress. this paper describes a system proposed by the ncuee-nlp (national central university, dept. of electrical engineering, natural language processing lab) for the smm4h 2022 task 8 using transformer models including bert @xcite , roberta @xcite , albert @xcite , xlnet @xcite , and electra @xcite , followed by finetuning for the classification task. in addition, a majority voting ensemble mechanism is used to enhance system performance. the evaluation metric is f1-score for the positive class (i.e., tweets annotated as "1"). our best f1-score is 0.73 over the positive class. ## the ncuee-nlp system figure 1 shows our ncuee-nlp system architecture for the smm4h 2022 shared task 8. our system is composed of two parts: 1) pretrained transformer models; and 2) ensemble mechanism. we approach the task using five pre-trained transformer models, including bert @xcite , roberta @xcite , albert @xcite , xlnet @xcite , and electra @xcite . we use training, validation and test datasets to fine-tune the ## evaluation the experimental datasets were mainly provided by task organizers @xcite , with a total of 2,936 tweets in the training set, including 1,092 positive and 1,844 negative tweets. the validation set contains 534 tweets (156 positive/264 negative). all tweets were pre-processed using the following steps: 1) converting username into @user; 2) converting emojis into textual representation; 3) removing all urls; 4) removing '#' from hashtags; 5) segmenting hashtags which contained two or more words into their component words; and 6) removing multilingual characters to retain english letters and numbers only. the pre-trained transformer models were downloaded from huggingface @xcite . we selected an odd number (i.e., three of five) of models for majority voting ensemble. a total of 12 groups was combined for the ensemble mechanism. the hyper-parameters used were as follows: training batch size 100, learning rate 1e-5, and maximum sequence length 256. we selected the top three ensemble models that performed well on the validation set as our final testing models. the combination of roberta, xlnet and electra performed achieved the best f1-score of 0.73 on the test set. ## conclusions this study describes the ncuee-nlp system submission in smm4h 2022 task 8 for selfreported chronic stress, including system design, implementation and evaluation. our submitted ensemble pre-trained transformer models achieved a high f1-score of 0.73.