LHS 712_ ADEN ot G ood at # SMM 4 H 2024 Task 1: Deep- LLMADE miner: A deep learning and LLM pharmacovigilance pipeline for extraction and normalization of adverse drug event mentions on T witter (2024)

Abstract: Adverse drug events (ADEs) pose major public health risks, with traditional reporting systems often failing to capture them. Our proposed pipeline, called Deep-LLMADEminer, used natural language processing approaches to tackle this issue for #SMM4H 2024 shared task 1. Using annotated tweets, we built a three part pipeline: RoBERTa for classification, GPT-4-turbo for span extraction, and BioBERT for normalization. Our models achieved F1-scores of 0.838, 0.306, and 0.354, respectively, offering a novel system for Task 1 and similar pharmacovigilance tasks.

## introduction adverse drug events (ades) are significant public health challenges, contributing to substantial morbidity and mortality @xcite . effective pharmacovigilance, essential for ensuring the safe use of medications, struggles with the underreporting of adverse drug reactions (adrs), with estimates indicating that over 90% of adrs go unreported @xcite . social media offers a novel avenue for real-time, patient-centered insights into adrs, supplementing traditional data sources. we developed the deep-llmademiner pipeline for the smm4h-2024 task 1 @xcite to extract and normalize ades from englishlanguage tweets. this study aims to assess the performance of the three-part pipeline in extracting and normalizing ades from tweets. ## the deep-llmademiner pipeline in step 1, we train a classifier to detect the presence of ades in tweets. in step 2, we train a large language model (llm) to extract ade entities and their spans from tweet text. finally, in step 3, we train a classifier to map the extracted ade entities to formal ids in the meddra ontology 1 , 1 meddra® the medical dictionary for regulatory activities terminology is the international medical terminology a standardized hierarchical medical terminology (fig. 1 ). the #smm4h 2024 task 1 dataset @xcite comprised of 17,974 training tweets annotated with 1,711 ade mentions and 959 validation tweets annotated with 87 ade mentions. additionally, 11,799 test tweets were provided for model evaluation. ## conclusion in our submission to the #smm4h 2024 task 1, we evaluated models for ade classification, span extraction, and normalization steps using roberta, gpt-4, and biobert, respectively. despite some model errors in identification and resource limitations, our methods remained efficient and costeffective.. future work will focus on refining these models and addressing data imbalance to improve ade detection and reporting. specifically, for step 1, we plan to implement weighting strategies to correct dataset imbalance for more balanced and improved outcomes. for step 2, increasing the number of training examples will potentially boost model accuracy. for step 3, we aim to fine-tune hyperparameters based on validation datasets to enhance model performance.