ADE Oracle at # SMM 4 H 2024: A Two-Stage NLP System for Extracting and Normalizing Adverse Drug Events from Tweets (2024)

Abstract: This study describes the approach of Team ADE Oracle for Task 1 of the Social Media Mining for Health Applications (#SMM4H) 2024 shared task. Task 1 challenges participants to detect adverse drug events (ADEs) within English tweets and normalize these mentions against the Medical Dictionary for Regulatory Activities standards. Our approach utilized a two-stage NLP pipeline consisting of a named entity recognition model, retrained to recognize ADEs, followed by vector similarity assessment with a RoBERTa-based model. Despite achieving a relatively high recall of 37.4% in the extraction of ADEs, indicative of effective identification of potential ADEs, our model encountered challenges with precision. We found marked discrepancies between recall and precision between the test set and our validation set, which underscores the need for further efforts to prevent overfitting and enhance the model’s generalization capabilities for practical applications.

## introduction this paper outlines team ade oracle's participation in the 9th social media mining for health research and applications (#smm4h) 2024 workshop's task 1 @xcite , which involved extracting and normalizing adverse drug events (ades) from tweets into medical dictionary for regulatory activities (meddra) terms foot_0 . the task complexity increased in 2024 by combining ade detection with normalization, a challenge heightened by the informal and diverse language used on social media @xcite . addressing ades through social media enhances pharmacovigilance, providing critical data for public health interventions @xcite @xcite @xcite . our approach employed a spacy-based nlp pipeline, retraining a named entity recognition (ner) module to extract ades, and a roberta model for aligning text with meddra standards @xcite , navigating the trade-offs between recall and precision. while our system effectively identified many ades, the prevalence of false positives points to a need for further refinement to enhance the accuracy and utility of our methods for public health surveillance. ## dataset this study employed the #smm4h 2024 task 1 dataset, comprising 30,949 tweets distributed across 18,185 training, 965 validation, and 11,799 test tweets @xcite . ## system description our methodology for the #smm4h 2024 task 1 involves a two-stage process: we use an ner package to extract ades, followed by the normalization of these entities against the meddra using vector similarity techniques @xcite . ## results our system consisting of the ner model for ade extraction and roberta for the normalization task is evaluated in tables 1 and 2 on the validation set and the official test set, respectively. ## conclusion our contribution to #smm4h 2024 task 1 consists of an ner model retrained to identify ades and a similarity-based roberta model to normalize them. the findings from our system underline the challenges and opportunities presented by the use of nlp in detecting and normalizing ades from social media. despite achieving high recall, our model's low precision highlights a significant challenge in accurately identifying relevant ades amid the informal language prevalent on platforms like twitter. furthermore, the task has demonstrated that while our current methodology is capable of initial identification, it falls short in scenarios involving generalizing to data different from the training data, which is crucial for practical applications. for future work, we will investigate enhancing model precision through advanced linguistic analysis, employing models pre-tuned on ade datasets, fine-tuning roberta for vectorization of ade entities and meddra entries, and incorporating additional ade data.