G olden_ D uck at # SMM 4 H 2024: A Transformer-based Approach to Social Media Text Classification (2024)

Abstract: In this paper, we have addressed Task 3 on social anxiety disorder identification and Task 5 on mental illness recognition organized by the SMM4H 2024 workshop. In Task 3, a multi-classification problem has been presented to classify Reddit posts about outdoor spaces into four categories: Positive, Neutral, Negative, or Unrelated. Using the pre-trained RoBERTa-base model along with techniques like Mean pooling, CLS, and Attention Head, we have scored an F1-Score of 0.596 on the test dataset for Task 3. Task 5 aims to classify tweets into two categories: those describing a child with conditions like ADHD, ASD, delayed speech, or asthma (class 1), and those merely mentioning a disorder (class 0). Using the pre-trained RoBERTa-large model, incorporating a weighted ensemble of the last 4 hidden layers through concatenation and mean pooling, we achieved an F1 Score of 0.928 on the test data for Task 5.

## introduction in the past few years, social media has become the primary source of communication. unfortunately, millions of social media users suffer from mental illnesses such as social anxiety, attention-deficit or hyperactivity disorder (adhd), autism spectrum disorders (asd), delayed speech or asthma. the world health organization (who) estimates that around 450 million individuals are impacted by these conditions. according to who, 1 in 7 adolescents from age 10 to 19 experience mental health issues, many of which go unnoticed and untreated. 1 . in this research work, we have presented our findings for the two shared tasks on social anxiety disorder identification (task 3) and mental illness recognition (task 5) organized under smm4h 2024 workshop @xcite . by employing transformer-based pre-trained models alongside different techniques, we obtained f1 scores of 0.596 for task 3 and 0.928 for task 4. by examining user-generated content from social media platforms like reddit and twitter, this research aims to shed light on the potential therapeutic benefits of outdoor environments and enhance our understanding of childhood developmental disorders. ## related work automatic mental illness detection using social media data has been a key research area in literature. the key focus of this study @xcite is to detect and identify conditions such as adhd, asd, delayed speech, and asthma in children using data from twitter. it suggests twitter data could offer a high-performance method for this epidemiological research. they have achieved notable performance using several models: svm with an f1 score of 0.74, bert-base-uncased achieving an f1 score of 0.85, and bertweet-large scoring an f1 score of 0.92. @xcite discuss a system using traditional machine learning models for the early detection of autism spectrum disorder (asd). they have utilized various machine learning techniques, including svm (accuracy of 0.71), logistic regression (accuracy of 0.71), knearest neighbor (accuracy: 0.62), and random forest (accuracy: 0.69). additionally, they have explored the stacked deep learning models like cnn+svm (accuracy of 0.916) and rnn+svm (accuracy of 0.866). @xcite focus on classifying text data related to social disorders in children and adolescents using encoder-decoder models and data augmentation methods, achieving the best f1 scores of 0.627 in task 3 and 0.841 in task 5. the research highlights the importance of employing sophisticated deep learning networks to identify social disorders from social media data. according 1 to @xcite , the paper explores the application of social media for mental health communication. it emphasizes the significance of automating the detection of mental illness using roberta (achieving an accuracy and f1 score of 0.83) and bert (with accuracy and f1 score of 0.78 and 0.80 respectively) models for classifying mental health disorders based on reddit posts. ## task and dataset description these shared tasks have been organized by the smm4h 2024 workshop @xcite , including two tasks related to identifying and classifying medical information from social media data. task 3, entitled "classification of reported effects of outdoor spaces on social anxiety symptoms", entails a multi-class classification challenge. its objective is to sort reddit posts containing specific keywords related to outdoor environments into four distinct categories: (i) positive effect (0), (ii) neutral or no effect (1), (iii) negative effect (2), and (iv) unrelated (keywords not pertaining to actual outdoor spaces or activities) (3). task 5, named "binary classification of tweets reporting children's medical disorders", introduces a binary classification challenge. the goal of this task is to automatically identify tweets from users who disclosed their pregnancy on twitter and report a child with adhd, asd, delayed speech, or asthma (labeled as "1"), versus tweets that just mention these disorders (labeled as "0"). in task 3, the data is divided into 1800 training samples, 600 validation samples, and 1200 test samples, with the test data not publicly provided. similarly, in task 5, there are 7398 training samples, 389 validation samples, and 10,000 test samples, with the test data also not publicly provided. ## methodology 4.1 preprocessing we have applied similar preprocessing steps for both tasks: removed urls, hashtags, and user mentions from text, removed emojis, lowered all english letters, and removed duplicate punctuations. for task 5, we have conducted additional preprocessing by expanding contractions like "wouldn't" to "would not". ## conclusion in this paper, we outline our contributions to task 3 and task 5 for the 2024 smm4h workshop on social media mining for health applications. we utilized several transformer models to perform multiclass classification on reddit posts and binary classification on tweets. utilizing roberta-base with mean pooling, cls, and attention head concatenation, we have reached an f1 score of 0.596 for task 3. for task 5, we have employed roberta-large with a weighted ensemble of the last 4 hidden layers concatenation and mean pooling, obtaining an f1-score of 0.928. our experiments demonstrated our system's reliability and adaptability, and we are working on further improvements.