CTYUN - AI @ SMM 4 H -2024: Knowledge Extension Makes Expert Models (2024)

Abstract: This paper explores the potential of social media as a rich source of data for understanding public health trends and behaviors, particularly focusing on emotional well-being and the impact of environmental factors. We employed large language models (LLMs) and developed a suite of knowledge extension techniques to analyze social media content related to mental health issues, specifically examining 1) effects of outdoor spaces on social anxiety symptoms in Reddit,2) tweets reporting children’s medical disorders, and 3) self-reported ages in posts of Twitter and Reddit. Our knowledge extension approach encompasses both supervised data (i.e., sample augmentation and cross-task fine-tuning) and unsupervised data (i.e., knowledge distillation and cross-task pre-training), tackling the inherent challenges of sample imbalance and informality of social media language. The effectiveness of our approach is demonstrated by the superior performance across multiple tasks (i.e., Task 3, 5 and 6) at the SMM4H-2024. Notably, we achieved the best performance in all three tasks, underscoring the utility of our models in real-world applications.

## introduction in recent years, the surge in social media usage has transformed these platforms into valuable repositories of public health attitudes and behaviors. users not only share snippets of their daily lives but also discuss a variety of health issues, drug reactions, and treatment outcomes, providing a wealth of realtime data for medical and health research. social media plays an especially crucial role in monitoring adverse drug reactions, tracking diseases, and facilitating public discussions on health conditions. this data aids healthcare organizations and researchers in understanding disease trends and patient needs, enhancing drug safety monitoring and optimizing treatment plans. against this backdrop, the significance of the 9th social media mining for health research and applications workshop (smm4h-2024) @xcite is particularly pronounced. this workshop brings together researchers, developers, and medical professionals from around the world to address the challenges of automating the extraction and analysis of health information from social media. the conference not only serves as a platform for sharing the latest research findings and cutting-edge technologies but also organizes multiple shared tasks targeting specific practical application problems, attracting numerous teams. these tasks are designed to enhance data processing capabilities across languages and cultural contexts, aiming to more accurately parse and utilize healthrelated information from social media, thereby supporting global health research and public health surveillance. in this workshop, our goal is to construct and enhance the given limited data, including both unsupervised and supervised data, to achieve maximum knowledge extension for training large language models. this will enable us to turn the models into specialized experts for each individual task. the core points are summarized as follows: these strategies not only resolved issues related to uneven class distribution and data scarcity but also refined the sentiment analysis process. finally, experimental results confirm the effectiveness of our strategies, as our team (i.e., ctyun-ai) achieved best performance in tasks 3, 5, and 6 among all participating teams. ## related work recent advances in natural language processing (nlp) have significantly enhanced the ability to analyze health-related discussions on social media. zanwar et al. utilized advanced nlp techniques alongside psycholinguistic features to effectively detect chronic stress expressions on social media, addressing data imbalance issues in the process @xcite . liu et al. demonstrated the use of multiple pre-trained models to detect adverse drug reactions on twitter, showcasing methods that tackle the complexities of social media data @xcite . additionally, tamayo et al. developed a transfer learning approach with postprocessing enhancements to accurately extract disease mentions from spanish tweets, improving the robustness of disease monitoring across different languages @xcite . these contributions highlight the evolving capabilities of nlp to provide valuable insights into public health from social media content. concurrently, generative models in the realm of nlp, exemplified by the gpt @xcite series, have exhibited remarkable abilities in comprehending and producing natural language. bai et al. @xcite developed the qwen models, which excel at multiple tasks. consequently, we employ the qwen models as our base model to cultivate further specialized experts. ## system overview in this section, we systematically explicate the knowledge extension strategies employed by our team for the respective sub-tasks. we commence with a descriptive analysis of the datasets pertinent to each sub-task, followed by an exposition of our methodologies, specifically devised and optimized in accordance with the task-specific data characteristics. ## conclusion in this work, we employed large language models and crafted a comprehensive set of knowledge extension techniques for the purpose of analyzing social media content pertaining to mental health concerns. we have provided a detailed account of how to leverage knowledge extension techniques to maximize the utilization of limited data, enabling us to train a general large language model into a domain-specific expert model. specifically, our approach encompasses both supervised data and unsupervised data, including sample augmentation, knowledge distillation and cross-task training. we achieved the best performance at multiple smm4h-2024 tasks (i.e., task 3, 5 and 6), validating the effectiveness of our approach.