Overview of the 9th Social Media Mining for Health Applications (# SMM 4 H ) Shared Tasks at ACL 2024 – Large Language Models and Generalizability for Social Media NLP (2024)

Abstract: For the past nine years, the Social Media Mining for Health Applications (#SMM4H) shared tasks have promoted community-driven development and evaluation of advanced natural language processing systems to detect, extract, and normalize health-related information in publicly available user-generated content. This year, #SMM4H included seven shared tasks in English, Japanese, German, French, and Spanish from Twitter, Reddit, and health forums. A total of 84 teams from 22 countries registered for #SMM4H, and 45 teams participated in at least one task. This represents a growth of 180% and 160% in registration and participation, respectively, compared to the last iteration. This paper provides an overview of the tasks and participating systems. The data sets remain available upon request, and new systems can be evaluated through the post-evaluation phase on CodaLab.

## introduction the number of social media (sm) users continues to grow worldwide: 86% of us adults and 79% in europe use sm (center; @xcite . advances in automated data processing, machine learning and natural language processing (nlp) allow us to incorporate this massive real-time data source from around the world for biomedical and public health applications, providing researchers a venue to address the many methodological challenges unique to this media. the social media mining for health applications (#smm4h) workshop, in its 9th annual iteration, brings together researchers interested in developing and sharing nlp methods that enable the systematic use of sm data for health research. the tasks of this year use data from various platforms (x, reddit, and patient forums such as lifeline 1 or yjqa 2 ) and languages (english, spanish, french, german, and japanese), with a special focus on large language models (llms) for social media nlp. seven tasks organized by experienced research teams from around the world were selected for 2024. we prioritized tasks evaluating the generalizability of the proposed approaches by explicitly creating test sets with out-of-distribution data such as unseen concepts, multi-lingual and multi-source texts. these tasks are: extraction and normalization of adverse drug events in english tweets (task 1), cross-lingual few-shot relation extraction for pharmacovigilance in french, german, and japanese (task 2), multi-class classification of effects of outdoor spaces on social anxiety symptoms in reddit (task 3), extraction of the clinical and social impacts of non-medical substance use from reddit (task 4), binary classification of english tweets reporting children's medical disorders (task 5), self-reported exact age classification with cross-platform evaluation in english (task 6), and identification of whether an llm or a human domain expert annotated data in the context of healthrelated applications (task 7). teams could register for a single task or multiple tasks. teams were provided with gold-standard annotated training and validation sets to develop their systems and, subsequently, an unlabeled test set for the final evaluation. after receiving the test set, all teams were given 5 days to submit the predictions of their systems to codalab -a platform that facilitates data science competitionsfor automatic evaluation, promoting a systematic performance comparison. among the 84 teams that registered, 45 teams submitted at least one set of predictions: 11 teams for task 1, 3 teams for task 2, 15 teams for task 3, 3 teams for task 4, 20 teams for task 5, 7 teams for task 6, and 3 teams for task 7. teams that submitted predictions were invited to submit a short manuscript describing their system, and 38 of the 45 teams did. each of these 38 system descriptions was peer-reviewed by at least 2 reviewers. in this article, we present the annotated corpora, the technical summaries of all the participating systems, and the performance results, providing insights into state-of-the-art methods for mining social media data for health informatics. adverse drug events (ades) are harmful and undesired reactions attributed to the intake of a drug or medication. active post-market surveillance is essential, given clinical trials may not detect all potential ades, particularly for vulnerable populations. social media can complement traditional reporting systems, such as the fda's adverse event reporting system (faers) for pharmacovigilance @xcite . task 1 involved automatically extracting ade text spans in tweets and normalizing them to their standard preferred term ids (ptids) in the medical dictionary for regulatory activities (meddra). ## conclusion this paper provides an overview of the smm4h 2024 shared tasks. this year, seven tasks were proposed, reflecting the growing interest and participation in the smm4h shared tasks. the topperforming teams predominantly used transformerbased models, including encoder-based lms like deberta-v3-large and decoder-based llms like gpt-4 or qwen-72b-chat. these teams frequently employed llm-based data augmentation techniques to address issues such as data imbalance, unseen examples, and domain mismatch in social media data. notably, out of 38 teams that submitted a system description paper, 11 participated in multiple tasks, often using the same systems finetuned for different tasks. this trend signifies an important effort within the community to develop high-performing classifiers and label sequencers that are both generalizable and reusable.