HBUT at # SMM 4 H 2024 Task2: Cross-lingual Few-shot Medical Entity Extraction using a Large Language Model (2024)

Abstract: Named entity recognition (NER) of drug and disorder/body function mentions in web text is challenging in the face of multilingualism, limited data, and poor data quality. Traditional small-scale models struggle to cope with the task. Large language models with conventional prompts also yield poor results. In this paper, we introduce our system, which employs a large language model (LLM) with a novel two-step prompting strategy. Instead of directly extracting the target medical entities, our system firstly extract all entities and then prompt the LLM to extract drug and disorder entities given the all-entity list and original input text as the context. The experimental and test results indicate that this strategy successfully enhanced our system performance, especially for German language.

## introduction discussions on drugs and their adverse drug reactions shared by users in social media, including the efficacy, side effects, and personal treatment journeys serve as valuable references for pharmacovigilance. we participated in the 9th social media mining for health research and applications workshop and shared tasks @xcite . the task targets both the extraction of drug and disorder/body function mentions @xcite ) and the extraction of relations between those entities (joint named entity detection and relation extraction, subtask 2b). the paper is about the task 2a. our task was to identify drug and disorder mentions in german, french, and japanese datasets from x(twitter) and a german patient forum. in previous works on similar tasks, people commonly use bert models @xcite . for multilingual tasks, the m-bert (multilingual bert) model is often employed @xcite . however, in this particular open task, the provided dataset is relatively small, especially in the training set. therefore, it is challenging to perform pre-training and fine-tuning on the traditional bert model due to the limited amount of data available. thus, we opted to use a large language model (llm) for the task. because the training data is also not enough to fine-tune a llm well, we opted for the promptbased approach. our system utilizes a two-step prompting strategy. a first step to get a list of all entities, and then a second to further extract drug and disorder entities from the list. ## methodology due to the small size of the provided dataset, we placed our focus primarily on using large language models (llms). we build our system based on a glm @xcite through its online api. it is trained on multilingual dataset and performs almost on a par with chatgpt and worked well for multilingual tasks in our preliminary experiments. we utilize few-shot prompt engineering technologies to adapt glm to the drug and disorder mention extraction task as shown in figure 1 . ## final results our final system employed glm-3-turbo model with isbs prompt. the overall results are shown in especially, our method demonstrated excellent performance for the german part. the results for the german part are shown in ## conclusion due to the small size of our dataset, we found that the performance with conventional small language models (e.g. berts) is not able to give satisfying precision. consequently, we build our system based on glm-3-turbo large language model. besides, we found that conventional straight-forward prompting strategy encountered low precision for this task. to address this issue, we proposed a step-by-step prompting strategy. we firstly prompt to extract a list of all entities. then we prompt to guide the model to choose drugs and disorder mentions from the list within the context of the corresponding original input text. this prompting approach significantly improved the effectiveness of our system.