AAST - NLP @# SMM 4 H ’24: Finetuning Language Models for Exact Age Classification and Effect of Outdoor Spaces on Social Anxiety (2024)

Abstract: This paper evaluates the performance of “AAST-NLP” in the Social Media Mining for Health (SMM4H) Shared Tasks 3 and 6, where more than 20 teams participated in each. We leveraged state-of-the-art transformer-based models, including Mistral, to achieve our results. Our models consistently outperformed both the mean and median scores across the tasks. Specifically, an F1-score of 0.636 was achieved in classifying the impact of outdoor spaces on social anxiety symptoms, while an F1-score of 0.946 was recorded for the classification of self-reported exact ages.

## introduction the widespread use of social media platforms has encouraged the employment of natural language processing (nlp) in all domains, specifically in health-related applications @xcite . these platforms are considered unfiltered, realtime, valuable sources for analysis as millions of users openly share their personal health narratives and experiences on a daily basis. the social media mining for health applications (smm4h-2024) workshop provides a chance to develop natural language processing (nlp) models that automatically extract meaningful information from social media data through seven shared tasks in various contexts, including pharmaceutical, social, health, and clinical. in this paper, we describe our team's participation in both task 3, self-reported exact age classification, and task 6, multi-class classification of the effects of outdoor spaces on social anxiety symptoms in reddit. the former task explores the extraction of patient demographics, enabling large-scale observational studies. similarly, the latter investigates outdoor spaces mentioned in social media and aims to qualitatively assess their effects and relation to social anxiety disorder (sad). ## error analysis in our error analysis of the validation set for task 3, we observed that one of the main reasons for performance degradation was the misclassification of unrelated and neutral instances. upon checking samples of these instances manually, we identified that there exist some examples to be mislabelled or at least confusing even on the human level. eliminating such ambiguous data in following versions of the task will definitely increase the training data quality and reflect such quality on the models' performance. additionally, in task 6, mislabeled examples in both the training and validation sets pose a risk to the effectiveness of the training and evaluation processes. one clear example of that is the post with id: 8812 in the validation set which was wrongfully classified as having no exact age mention. ## conclusion in this work, approaches were presented to address the challenges of self-reported exact age classification and classification of effects of outdoor spaces on social anxiety symptoms. for the first task, an ensemble of roberta, bertweet, and mistral was employed to handle the problem. despite the achieved performance, it is believed that there is room for improvement in the approach in the future. areas of potential improvement include refining the preprocessing procedure and fine-tuning mistral language model.