SRCB at # SMM 4 H 2024: Making Full Use of LLM -based Data Augmentation in Adverse Drug Event Extraction and Normalization (2024)

Abstract: This paper reports on the performance of SRCB’s system in the Social Media Mining for Health (#SMM4H) 2024 Shared Task 1: extraction and normalization of adverse drug events (ADEs) in English tweets. We develop a system composed of an ADE extraction module and an ADE normalization module which furtherly includes a retrieval module and a filtering module. To alleviate the data imbalance and other issues introduced by the dataset, we employ 4 data augmentation techniques based on Large Language Models (LLMs) across both modules. Our best submission achieves an F1 score of 53.6 (49.4 on the unseen subset) on the ADE normalization task and an F1 score of 52.1 on ADE extraction task.

## introduction the social media mining for health (#smm4h) workshop has served as a competitive platform aimed at promoting the development and evaluation of advanced natural language processing (nlp) systems for the detection, extraction and normalization of health related information in social media texts (tweets, reviews and reddit posts). among the shared tasks in #smm4h 2024 @xcite , ade extraction and normalization has been the longest running task, which requires participants first extract spans of adverse drug events (ades) expressions from tweets and then normalize the spans to meddra foot_0 ontology's preferred terms (pts). this task is evaluated in the following order of priority: the f1 score of ade normalization, the f1 score of ade normalization on the unseen subset foot_1 and f1 score of ade extraction. the challenges of this task lie in: (1) the dataset exhibits extreme imbalance between samples containing ades (positive) and those not containing ades (negative). (2) the validation and test sets contain ades to which the corresponding pts are not seen during training. (3) the given texts are tweets with frequent use of informal grammar as well as irregular vocabulary. in recent years, llms have been widely used for data augmentation @xcite @xcite to improve data quality, especially for the unbalanced and noisy data. to this end, we propose to make full use of llm-based data augmentation using glm-4 foot_2 and gpt-3.5 across both ade extraction and ade normalization modules. for ade extraction, we first enrich the training set from both mentionlevel and context-level by prompting the llms to rewrite ade spans and the other parts of the positive samples. to cover more ades with unseen preferred terms, we prompt llms to generate synthetic tweets and obtain pseudo ade annotations by our previously trained ade extraction models. for ade normalization, we ask the llms to rewrite the given tweets into formal written texts. in addition, we also ask the llms to give an explanation for each lowest level term (llt) and preferred term (pt) in meddra dictionary. these llm-based data augmentation techniques show varying degrees of performance improvement on our system. finally, our system obtained the highest ade normalization f1 score in task 1. ## data augmentation using llms since the dataset poses challenges such as data unbalance, unseen pts and domain mismatch as illustrated before, we utilize 4 different llm-based (glm-4, gpt-3.5) data augmentation techniques across ade extraction and ade normalization. more details are provided in d