P olyu CBS at SMM 4 H 2024: LLM -based Medical Disorder and Adverse Drug Event Detection with Low-rank Adaptation (2024)

Abstract: This is the demonstration of systems and results of our team’s participation in the Social Medical Mining for Health (SMM4H) 2024 Shared Task. Our team participated in two tasks: Task 1 and Task 5. Task 5 requires the detection of tweet sentences that claim children’s medical disorders from certain users. Task 1 needs teams to extract and normalize Adverse Drug Event terms in the tweet sentence. The team selected several Pre-trained Language Models and generative Large Language Models to meet the requirements. Strategies to improve the performance include cloze test, prompt engineering, Low Rank Adaptation etc. The test result of our system has an F1 score of 0.935, Precision of 0.954 and Recall of 0.917 in Task 5 and an overall F1 score of 0.08 in Task 1.

## introduction the rise in people using social media for health information has resulted in a significant increase in health-related data, which allows researchers to harness the information, along with nlp and machine learning methods, to contribute to public health @xcite . the 9th social media mining for health applications (smm4h) shared tasks, aiming to advance methods that utilise social media data for health research, have a special focus on large language models (llms) and generalizability for social media nlp. there are 7 tasks given in the 9th smm4h workshop @xcite . our team focused on task 1 and task 5. task 1 has two sub-tasks, which are (1) detecting adverse drug event terms in tweets and (2) normalizing these colloquial mentions to their standard concept to preferred term ids according to meddra. task 5 focuses on classifying tweets reporting children's medical disorders. the challenge of this task is differentiating between tweets from pregnancy users who declared that their child has specific conditions with attention-deficit/hyperactivity disorder (adhd), autism spectrum disorders (asd), delayed speech, or asthma, and tweets that just mention the disorder. the main challenges of both tasks are as follows: ## conclusion in conclusion, this paper presents plm and llmbased methods for task 1 and task 5 in smm4h 2024. our team balanced efficiency and efficacy by employing strategies like cloze test shifting, instruction tuning, and low-rank adaptation. through empirical evaluations, this paper proved that the llm did surpass certain plms in the two tasks, though it suffered from the hallucination issue. more methods need to be explored to ensure solid and reliable results from llm in tasks of the public health domain.