Leveraging C hat GPT in Pharmacovigilance Event Extraction: An Empirical Study (2024)

Abstract: With the advent of large language models (LLMs), there has been growing interest in exploring their potential for medical applications. This research aims to investigate the ability of LLMs, specifically ChatGPT, in the context of pharmacovigilance event extraction, of which the main goal is to identify and extract adverse events or potential therapeutic events from textual medical sources. We conduct extensive experiments to assess the performance of ChatGPT in the pharmacovigilance event extraction task, employing various prompts and demonstration selection strategies. The findings demonstrate that while ChatGPT demonstrates reasonable performance with appropriate demonstration selection strategies, it still falls short compared to fully fine-tuned small models. Additionally, we explore the potential of leveraging ChatGPT for data augmentation. However, our investigation reveals that the inclusion of synthesized data into fine-tuning may lead to a decrease in performance, possibly attributed to noise in the ChatGPT-generated labels. To mitigate this, we explore different filtering strategies and find that, with the proper approach, more stable performance can be achieved, although constant improvement remains elusive.

## introduction pharmacovigilance stands as a pivotal discipline in healthcare that encompasses a range of processes: identifying, evaluating, understanding, and preventing adverse effects and other medicine-related issues @xcite . within this domain, pharmacovigilance event extraction emerges as a crucial practice aimed at extracting structured medication-related event data from medical text sources, serving as valuable inputs for automatic drug safety signal detection. with the 1 related code for this paper is available at github.com/zhaoyuesun/phee-with-chatgpt. rapid expansion of electronic health records (ehr), medical case reports, and other textual resources, the need for efficient and accurate pharmacovigilance event extraction has become increasingly pressing. studies have been conducted to extract pharmacovigilance-related information from text data. however, previous research mainly focused on simple tasks such as entity extraction @xcite or binary relation extraction @xcite . recently, @xcite introduced a novel dataset for pharmacovigilance event extraction, which includes hierarchical annotations of adverse events and potential therapeutic events, capturing information about the subject, treatment, and effect. additionally, they investigate the performance of various models, including sequence labelling and qabased approaches, for this task, providing a foundation for further advancements in extracting structured event data for pharmacovigilance research. the rise of large language models (llms), especially chatgpt @xcite , has sparked considerable interest in their potential applications in the medical field @xcite @xcite . in this study, our focus is on exploring different ways to incorporate chatgpt into the pharmacovigilance event extraction task. figure 1(a ) presents an example of this task. we first explore various strategies for prompting and demonstration selection to assess chat-gpt's performance in zero-shot and few-shot scenarios, comparing it with smaller fine-tuned models. our findings indicate that, with suitable demonstration selecting strategies, chatgpt performs reasonably well but still falls short of the performance achieved by fully fine-tuned smaller models, as demonstrated in figure 1(b) . furthermore, we delve into the utilization of llms for data augmentation, which is suggested to be beneficial in improving small model's performance in recent work @xcite @xcite @xcite . we employ chatgpt to generate sentences structurally resembling demonstration samples, as illustrated in figure 1(c ). however, our experiments show that simply combining these generated samples with the training set leads to an overall performance decrease. considering the possible influence of synthesized data noises, we further introduce a filtering strategy for augmented data quality control, which, though still does not outperform the fully finetuned model, reduces the performance drop and brings it closer to the levels achieved with the original training data, while reducing the variance. this indicates enhanced stability when working with ample high-quality data. in summary, we compare various regimes of leveraging chatgpt to assist in pharmacovigilance event extraction, providing practitioners with meaningful references for choosing suitable strategies. additionally, we conduct a fine-grained qualitative analysis of chatgpt synthesized instances and data augmentation and explore reasons for their lack of positive effect, laying the groundwork for improvements in subsequent work. ## chatgpt as data synthesizer we explore the potential of leveraging chatgpt for data augmentation purposes. to achieve this, we incorporate an example from the training set, along with its annotated events, as input to chatgpt. we then prompt chatgpt to generate a sentence that exhibits a similar event structure to the given sentence and extract the events from the generated sentence. however, based on our initial study, we observed that chatgpt tends to miss specific mentions of drugs or excessively use certain drugs, such as 'ibuprofen'. we address this issue by restricting the inclusion of drug names and their corresponding effects sampled from the training data in generated sentences. details of the prompt for data synthesizing are shown in appendix g. recognizing that directly incorporating generated samples into the training data can lead to performance decline, possibly due to issues related to data quality, we have introduced filtering strategies. the main rationale behind the filtering is to retain annotations for which the model exhibits a certain level of confidence, based on the assumption that a finetuned model possesses some discriminatory ability regarding the quality of annotations, and incorrect annotations may result in lower confidence scores from the model for the annotation sequence. specifically, we introduce: a) train filter: filtering the training set with s @xmath0, where s gold is the average token probability given by the fine-tuned model on the ground-truth event label sequence. this means we filter out training instances for which the annotation sequence has less model certainty than the average level; b) augment filter: filtering augmented data with z(s gold ) < 0 or z(s gold ) < z(s pred ), where s pred is the average token probability for predicted event label sequences. z(s) = (s -mean(s v ))/std(s v ), and s v represents the values of s in the validation set. in this case, we filter out samples generated by chatgpt if their generated annotation sequence probability, as calculated by a fine-tuned model, falls below the average level or is less certain than the sequence predicted by the fine-tuned model itself. considering models potentially assigning high scores to the sequences they predict, we use z-score instead of direct predictive probabilities for comparison. with these filtering rules, we compare the model's performance on several data settings, including: training data (tr.), training data combined with augmented data (tr.+aug.), filtered training data (tr. fil.), training data with filtered augmented data (tr.+aug. fil.) and filtered training data with filtered augmented data (tr. fil.+aug. fil.). ## conclusion this paper provides empirical practice in various approaches to leveraging chatgpt for the pharmacovigilance event extraction task. overall, chat-gpt exhibits impressive few-shot learning capabilities in pharmacovigilance event extraction. nevertheless, considering the sensitivity of the medical field, fine-tuned models retain a clear edge in the presence of abundant data. in our experiments, the introduction of chatgpt-synthesized instances for data augmentation does not improve the performance of small model fine-tuning. however, appropriate quality control may increase the stability of performance. qualitative analysis indicates that errors may arise in chatgpt-synthesized data when distinguishing semantically complex arguments, and the coverage of rare arguments is insufficient. we emphasize the structural complexity and fine granularity of arguments in event extraction, which may pose challenges in generating synthetic data. future work can conduct more in-depth data augmentation research addressing these aspects.