IMS _medic ALY at # SMM 4 H 2024: Detecting Impacts of Outdoor Spaces on Social Anxiety with Data Augmented Ensembling (2024)

Abstract: Many individuals affected by Social Anxiety Disorder turn to social media platforms to share their experiences and seek advice. This includes discussing the potential benefits of engaging with outdoor environments. As part of #SMM4H 2024, Shared Task 3 focuses on classifying the effects of outdoor spaces on social anxiety symptoms in Reddit posts. In our contribution to the task, we explore the effectiveness of domain-specific models (trained on social media data – SocBERT) against general domain models (trained on diverse datasets – BERT, RoBERTa, GPT-3.5) in predicting the sentiment related to outdoor spaces. Further, we assess the benefits of augmenting sparse human-labeled data with synthetic training instances and evaluate the complementary strengths of domain-specific and general classifiers using an ensemble model. Our results show that (1) fine-tuning small, domain-specific models generally outperforms large general language models in most cases. Only one large language model (GPT-4) exhibits performance comparable to the fine-tuned models (52% F1). Further, we find that (2) synthetic data does improve the performance of fine-tuned models in some cases, and (3) models do not appear to complement each other in our ensemble setup.

## introduction social anxiety disorder is a medical condition that can significantly impact an individual's life @xcite . social media platforms have emerged as spaces where affected individuals can communicate their experiences and seek support. these platforms are rich with biomedical information, providing an opportunity for medical practitioners to gain novel insights the first three authors contributed equally. about medical conditions. however, this data is highly diverse and annotation is expensive, especially for the medical domain. access to a broad variety of classification and generative models has the potential to close this gap as it (1) allows to explore the capability of domain-specific and general models to solve these types of tasks, and (2) opens the possibility to generate synthetic training instances to complement sparse, human-labeled data. as part of #smm4h 2024, shared task 3 focuses on classifying the effects of outdoor spaces on social anxiety symptoms in reddit posts. given the post, the goal is to predict the user's sentiment towards the effect of outdoor space in a multi-class classification setup with the target labels positive, negative, neutral and unrelated. with our contribution, we investigate three research questions (rqs): rq1 given the sparsity of the data, do fine-tuned, domain-specific models outperform general models? rq2 does incorporating synthetic data complement human-labeled data and enhance model robustness? rq3 is reddit's text diversity better captured by a set of different models in an ensemble setup? ## system description we hypothesize that the diversity of texts shared on reddit benefits from aggregating multiple approaches. therefore, we design an ensemble model that takes as input the predictions of individual models varying in architecture and training procedure. the individual models are: fine-tuned language models. we fine-tune bert @xcite , roberta @xcite , distilbert @xcite and socbert @xcite on the training split of the task data to obtain customized models. we set truncation and padding to true, batch size to 4, and train for 3 epochs, with a learning rate of 5•10 -5 using the adamw optimizer. few-shot prompting. to explore the capacity of general large language models (llms), we prompt mistral-7b-v0.1 @xcite , llama-2-7b-chat-hf @xcite , gpt-3.5 (openai, 2022) and gpt-4 (openai, 2023) to generate labels. we provide them with task instructions in a one-shot setup, where the example instance is randomly chosen from the training data. ## conclusion we present our contribution to the #smm4h 2024 shared task 3 which focuses on classifying the effects of outdoor spaces on social anxiety symptoms in reddit posts. we find that fine-tuning models overall show a more robust performance compared to llm prompting. synthetic data may increase the robustness of the models. we can not show a superior performance of an ensemble. this leads to important future work: for the prompting approaches, we have to evaluate the impact of the prompt design for the task. for fine-tuned models, a thorough analysis of the synthetic data is key to gauging the impact of generated instances in more detail. for all models, an in-depth error analysis is crucial to understand model capabilities and the impact the individual predictions may have on the ensemble. further, testing alternative ensemble designs (e.g., gradient-boosted decision trees) is key to understanding the interaction between the probability-based, class predictions we obtain from the fine-tuned models, and the class-only predictions from llms.