RIGA at SMM 4 H -2024 Task 1: Enhancing ADE discovery with GPT -4 (2024)

Abstract: The following is a description of the RIGA team’s submissions for the SMM4H-2024 Task 1: Extraction and normalization of adverse drug events (ADEs) in English tweets. Our approach focuses on utilizing Large Language Models (LLMs) to generate data that enhances the fine-tuning of classification and Named Entity Recognition (NER) models. Our solution significantly outperforms mean and median submissions of other teams. The efficacy of our ADE extraction from tweets is comparable to the current state-of-the-art solution, established as the task baseline. The code for our method is available on GitHub (https://github.com/emukans/smm4h2024-riga)

## introduction the smm4h-2024 task 1, as outlined in the overview @xcite , challenged participants to extract and normalize ades to meddra highlevel term identifiers (hltis). our submission aims to harness the capabilities of large language models (llms) to enhance performance. additionally, we compare the performance of the off-the-shelf submission, which did not involve model training, with a fine-tuned model that combines the original input with the output generated by gpt. the baseline system @xcite utilizes a pipeline method for solving the task. the pipeline involves 3 components and are executed sequentially: (1) the ade classifier for identifying tweets containing ade mentions, (2) the ade span extractor or named entity recognition (ner) for extracting ade mentions, and (3) the ade normalizer, which maps the extracted ade mention to meddra hlt identifiers. in our submission we utilize the same pipeline components. the paper concentrates on integrating the gpt model generation with the original text. a comparable methodology was employed in semeval-2023 @xcite , where the task involved token classification with highly specific tags. to streamline the process, the riga team utilized gpt as a knowledge database for individuals, entities, food items, and other relevant entities mentioned in the text. according to the llm for generative information retrieval survey @xcite , our method can be classified as a form of data augmentation. similar approaches have been independently employed in several studies @xcite @xcite ## data in contrast to the previous version of the task, the new challenge in the most recent dataset lies in the inclusion of negative samples (falling outside meddra categories) in each data split. as presented in ## methodology in order to address the issue of high data imbalance, our pipeline includes tweet classification as the initial step to filter out the majority of the tweets. subsequently, for the filtered tweets, we conduct ner to extract the precise spans that contain an ades. in the final step, we generate a sentence embedding for the span and identify the nearest hltis using cosine similarity. we used four tesla v100 16gb gpus, provided by our institution, for conducting these experiments. ## results in