TL ab at # SMM 4 H 2024: Retrieval-Augmented Generation for ADE Extraction and Normalization (2024)

Abstract: SMM4H 2024 Task 1 is focused on the identification of standardized Adverse Drug Events (ADEs) in tweets. We introduce a novel Retrieval-Augmented Generation (RAG) method, leveraging the capabilities of Llama 3, GPT-4, and the SFR-embedding-mistral model, along with few-shot prompting techniques, to map colloquial tweet language to MedDRA Preferred Terms (PTs) without relying on extensive training datasets. Our method achieved competitive performance, with an F1 score of 0.359 in the normalization task and 0.392 in the named entity recognition (NER) task. Notably, our model demonstrated robustness in identifying previously unseen MedDRA PTs (F1=0.363) greatly surpassing the median task score of 0.141 for such terms.

## introduction social media is a potential wealth of information for contemporary public health monitoring, offering real-time insights into the effects of medications as reported by patients @xcite . x (formerly twitter), with its continuous flow of user-generated content, is a rich but challenging source for identifying adverse drug events (ades), largely due to the informal and diverse language present in tweets @xcite . the task of extracting standardized ades from such unstructured text is complex, as conventional methods typically depend on large training datasets. these not only require heavy computational resources, but also may fail to adapt to the nature of social media language @xcite . here, we present a retrieval-augmented generation (rag) methodology to incorporate only relevant examples, side-stepping the need for large-scale data annotation and training processes @xcite . ## results we evaluated our approach using standard metrics of precision (p), recall (r), and f1 score across three different tasks: named entity recognition (ner), normalization (norm), and normalization on unseen data (norm-unseen). in ## discussion the results of our study highlight the effectiveness of our rag approach in the context of the smm4h 2024 task 1. our approach, which integrates the capabilities of llama 3, gpt-4, and the sfr-embedding-mistral model, has demonstrated a particularly strong performance in handling unseen meddra preferred terms, an important as-pect of real-world applicability given the evolving nature of language used in social media. this capability can enhance the timeliness and reliability of ade detection from social media sources, contributing to faster and more informed public health responses. for future use, there are certain areas of our pipeline that would benefit from experimental validation. for example, we arbitrarily selected the number 10 for both the ade extraction and text normalization to provide our models with an adequate amount of information without overloading them. in addition to refining our pipeline, future work may aim to explore additional few-shot learning techniques and retrieval methods to enhance the model's ability to adapt to the continuously evolving language on social media. if willing to trade cost for a potential performance boost, it would be worthwhile to assess gpt-4 for the ner task. or, with a large quantity of training data, the system may perform better with a fine-tuned local llm (such as llama 3) for both the ner and normalization steps. a.1.2 example of text normalization role content system given a tweet, a highlighted span, the medical terminology, and a reference list of meddra pts, identify the most relevant meddra pt for the medical terminology. return only the integer id of the meddra pt. user tweet: @user_________ i found the humira to fix all my crohn's issues, but cause other issues. i went off it due to issues w nerves/muscle spasms, span: issues w nerves, term: neuralgia, reference list: ['neuralgia: 10029223', 'facial neuralgia: 10061594', 'morton's neuralgia: 10052288', 'trigeminal neuralgia: 10044652', 'trigeminal neuritis: 10074054', 'post-traumatic neuralgia: 10076781', 'trigeminal nerve disorder: 10060890', 'occipital neuralgia: 10068106', 'glossopharyngeal neuralgia: 10018391', 'trigeminal nerve injection: 10044651']