HBUT at # SMM 4 H 2024 Task1: Extraction and Normalization of Adverse Drug Events with a Large Language Model (2024)

Abstract: In this paper, we describe our proposed systems for the Social Media Mining for Health 2024 shared task 1. We built our system on the basis of GLM, a pre-trained large language model with few-shot Learning capabilities, using a two-step prompting strategy to extract adverse drug event (ADE) and an ensemble method for normalization. In first step of extraction phase, we extract all the potential ADEs with in-context few-shot learning. In the second step for extraction, we let GLM to filer out false positive outputs in the first step by a tailored prompt. Then we normalize each ADE to its MedDRA preferred term ID (ptID) by an ensemble method using Reciprocal Rank Fusion (RRF). Our method achieved excellent recall rate. It obtained 41.1%, 42.8%, and 40.6% recall rate for ADE normalization, ADE recognition, and normalization for unseen ADEs, respectively. Compared to the performance of the average and median among all the participants in terms of recall rate, our recall rate scores are generally 10%-20% higher than the other participants’ systems.

## introduction extracting medical entities from social media is a challenging task. bert @xcite is one of the most popular models used for named entity recognition (ner). among its family, biobert @xcite and clinical-bert @xcite are especially reported useful for medical named entity recognition tasks. but without sufficient resources and data for finetuning, bert tends to underperform. some researches @xcite have demonstrated that tailored prompts can drive large language models (llms) to perform various downstream tasks well with only a small amount of data. this paper describes our work in the social media mining for health 2024 (smm4h2024). smm4h2024 task 1 was a mission for mining adverse drug events (ade) from twitter (x) and normalizing to their corresponding preferred terms within the medical dictionary for regulatory activities (meddra) terminology. inspired by the works on few-shot @xcite and zeroshot @xcite learning with llms, we introduce a system for ade identification and normalization. the identification phase of our system contains two steps: a potential ade extraction step and a self-improvement step: in the normalization phase, we use an ensemble method, utilizing the reciprocal rank fusion (rrf) @xcite ) method to integrate the relevance scores based on five pre-trained embedding models and levenshtein distance @xcite . the resulted fused score serves as the final measure of relevance. in this phase, each extracted ade text in the previous phase is labeled with the preferred term id (ptid) of the most related ade in meddra measured in this way. our system scored above mean and median on multiple metrics among all the participants and had the advantage of not requiring fine-tuning on downstream tasks. in ade text extraction, the green rectangle are glm models with different prompt, and blue are the texts. in normalization to meddra ptids, the red rectangle is rrf algorithm, including five language models and levenshtein distance. ## methodology the entire framework is illustrated in figure 1 . we have two main steps in our system: a potential ade text extraction phase and a normalization phase to meddra ptids. in this section, we will introduce the details of these phases. ## experiments besides our method described in section 2, there are several prompting methods that used in similar tasks, including few-shot prompting @xcite , zero-shot prompting and least-to-most prompting @xcite as follows, 1. few-shot: this method provides several examples along in the prompts. our system uses this method in the first-step in the ade extraction phase. we compared these methods in our local experiments. moreover, glm-4 are reported as an improved version of glm-3-turbo. thus, we also evaluated it for this task. in details, we compared five methods that used different prompts and llm models: fewshot prompt + glm-3-turbo wsi(without selfimprovement), few-shot prompt + glm-3-turbo, few-shot prompt + glm-4, zero-shot prompt + glm-3-turbo, and least-to-most prompt + glm-3-turbo. we evaluated these five methods in validation dataset in order to find the best method. the result is shown in appendix a.1. the few-shot + glm-4 achieved leading results across all metrics, so we ultimately decided to use the few-shot prompt + glm-4 method on the test set. ## conclusion our method surpasses the median and average results of the smm4h 2024 task 1 in some metrics, with improvements of 10%-20% over the median in r-norm, r-ner, and r-norm-unseen. this proves the feasibility of using large language models for social media text mining tasks with a welldesigned prompting strategy, especially in the cases that high recall rate is required. however, the decline in the precision of ade mining is still an issue that needs to be addressed. we consider the reason for the low accuracy score in ades mining is that the large language model we used generalize based on their pre-trained data and prompt engineering, while the text of ades in social media comments can be very different from the standard ade text. this gap leads to the failure of the large language model in identifying ades when it mines ades from social media. prompts for coarse-grained ade text extraction in english you are a super medical chemistry expert, please identify any chemical substances or drugs and their adverse drug event (ade) in the following english sentences. the sentences may or may not contain ades. if the sentence does not contain any ade, you don't need output anything. if there are ades, please find them and provide only the result of corresponding side effects without showing the reasoning process. side effects may be multiple. i will give you some examples. example：avelox has hurt your liver, avoid tylenol always, it further damages liver, eat grapefruit unless taking cardiac drugs ade：hurt your liver prompts for self-improvement in english i will give you a sentence and multiple words, and you will need to use your reasoning ability to determine whether these ades are caused by the medication rather than pre-existing conditions. example：avelox has hurt your liver, avoid tylenol always, it further damages liver, eat grapefruit unless taking cardiac drugs ade：hurt your liver reasoning：because of the sentence "avelox has hurt your liver," it can be inferred that "hurt your liver" is caused by avelox. there is no need to output the reason, just yes or no. please use your logical abilities to answer: in the [tweet], is the [ade] caused by the medication, or is it due to the person's own issues or other bad habits? just output "yes" or "no".