Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development (2024)

Abstract: The mining of adverse drug events (ADEs) is pivotal in pharmacovigilance, enhancing patient safety by identifying potential risks associated with medications, facilitating early detection of adverse events, and guiding regulatory decision-making. Traditional ADE detection methods are reliable but slow, not easily adaptable to large-scale operations, and offer limited information. With the exponential increase in data sources like social media content, biomedical literature, and Electronic Medical Records (EMR), extracting relevant ADE-related information from these unstructured texts is imperative. Previous ADE mining studies have focused on text-based methodologies, overlooking visual cues, limiting contextual comprehension, and hindering accurate interpretation. To address this gap, we present a MultiModal Adverse Drug Event (MMADE) detection dataset, merging ADE-related textual information with visual aids. Additionally, we introduce a framework that leverages the capabilities of LLMs and VLMs for ADE detection by generating detailed descriptions of medical images depicting ADEs, aiding healthcare professionals in visually identifying adverse events. Using our MMADE dataset, we showcase the significance of integrating visual cues from images to enhance overall performance. This approach holds promise for patient safety, ADE awareness, and healthcare accessibility, paving the way for further exploration in personalized healthcare.

## introduction an adverse drug event (ade) encompasses any harm resulting from medication use, whether it's unintended, off-label, or due to medication errors. adverse drug reactions (adrs) are a specific type of ade, denoting unexpected harm arising from the proper use of medication at the prescribed dosage. injuries from inappropriate or off-label use are not classified as adrs @xcite . ades pose significant public health concerns, contributing to numerous fatalities, serious injuries, millions of hospitalizations, and prolonged hospital stays. consequently, they impose substantial financial burdens, costing healthcare systems billions of dollars globally. despite advancements in healthcare, ade detection remains a significant challenge. implementing effective detection and monitoring strategies can substantially mitigate the adverse impacts on patients and healthcare systems @xcite , @xcite , @xcite . most of the previous ade detection works are based on text data only @xcite , @xcite , @xcite , @xcite , @xcite , @xcite , which presents a significant disadvantage due to its subjective nature and lack of specific details of visual cues, leading to potential inaccuracies and incomplete categorization of ade detection. despite extensive research, the potential of integrating textual data with visual information, such as images, has been largely overlooked. visual aids are essential in ade detection for numerous reasons. a substantial proportion of the population lacks proficiency in medical jargon, hindering accurate symptom descriptions. moreover, certain symptoms are inherently challenging to express through text alone. patients may struggle to differentiate between similar symptoms, like skin rash, eczema, peeling, and blister. as depicted in fig. 1 , sample images may present confusion to individuals lacking adequate medical expertise. integrating both text and images in these scenarios can enhance the accuracy and effectiveness of ade detection, offering a comprehensive understanding of the patient's current medical condition. to the best of our knowledge, ade detection using both image and text data has not been explored previously, and we take this opportunity to introduce a multimodal adverse drug event (mmade) dataset comprising adr images paired with corresponding textual descriptions. large language models (llms) and vision language models (vlms) have exhibited remarkable skills in generating human-like text, prompting their integration into various medical applications, including tasks such as chest radiography report generation, summarization, and medical question answering @xcite , @xcite , @xcite , @xcite . however, their potential in ade detection, which involves both text and images, has yet to be explored. leveraging llms and vlms for this task presents inherent limitations as they are predominantly trained on generic natural images sourced from databases like imagenet, wikipedia, and the internet. generic models may not possess the specialized medical knowledge required for comprehensive caption generation, potentially leading to oversimplified descriptions that overlook essential details like symptoms and medical intricacies. furthermore, while vlms have excelled in traditional visual-linguistic tasks, their application to medical imaging presents unique challenges that may hinder the accurate interpretation and description of complex medical images @xcite . specialized models such as xraygpt @xcite and skingpt4 @xcite , which are trained on chest x-ray and skin disease images, exemplify the domain specificity required for accurate medical image analysis. this has led us to explore adr detection within a multimodal framework. to support this exploration, we introduce mmade, a carefully curated dataset crafted for this specific purpose. mmade consists of 1500 instances of patient-reported concerns regarding drugs and associated side effects, each paired with both textual descriptions and corresponding images. in our study, we have employed instruct-blip @xcite , which builds upon the strong foundation of blip-2 @xcite , a pretrained model with high-quality visual representation and strong language generation capabilities. the meticulous fine-tuning process enables it to bridge the disparity between general-purpose models and the specialized demands of ade-specific tasks. moreover, our exploration of blip @xcite and git @xcite reveals that these models exhibit insufficient performance before fine-tuning. nevertheless, upon fine-tuning with domain-specific data, their performance experiences notable improvement. our key contributions are as follows: ## related works this section details the related ade detection tasks based on the data sources. works based on biomedical text and electronic medical record: various techniques have been developed for extracting ades from electronic medical records (emrs) @xcite , @xcite , as well as from medical case reports (mcrs) @xcite . gurulingappa et al. (2012a) utilized machine learning methods to identify and extract potential ade relations from medline case reports. unlike random data sources such as social media, both emrs and mcrs offer significant advantages by providing comprehensive records of a patient's medical history, treatment, conditions, and potential risk factors. moreover, these records are not limited to patients who have experienced adrs (harpaz et al., 2013). sarker and gonzalez (2015) conducted a study by taking data from medline case reports and twitter and reported how combining different datasets increases the performance of identifying adrs. huynh et al. (2016) explored various neural network frameworks for ade classification, utilizing datasets from both mcrs and twitter. disae is another corpus (gurulingappa et al., 2010), which consists of 400 med-line articles with annotations for disease and adverse effect names without drug-related information. d'oosterlinck et al. (2023) introduced the biodex dataset for biomedical ade extraction in real-world pharmacovigilance. this dataset comprises 65,000 abstracts, 19,000 full-text biomedical papers, and 256,000 document-level safety reports crafted by medical professionals. however, to the best of our knowledge, there is currently no publicly accessible annotated multimodal (image and text) corpus suitable for identifying drug-related adverse effects. works based on social media datasets: social media has become vital for accessing vast amounts of real-time information, making it valuable for identifying potential ades. @xcite conducted a pioneering study that analyzed user comments from social media posts, comprising a dataset of 6890 comments. the research demonstrated the significant value of user comments in identifying ades, highlighting their crucial role in this context. several other authors @xcite , @xcite , @xcite employed lexiconbased approaches to extract ades. however, these methods are limited to a specific set of target ades. @xcite em- keywords: ade, adr, adverse drug reaction, adverse drug event, adverse reaction, adverse drug event reporting, side effects, drug reactions, drug side effects, type of infection and reaction, medicine, drugs, skin rashes, red patches, eczema, ulcer, acne, skin irritation, edema, rosacea, alopecia, lip swelling. ployed a rule-based technique instead of a naive lexicon-based approach on the same dataset, enabling the detection of ades not covered by lexicons. several authors utilized supervised machine learning techniques like support vector machines (svm) @xcite , conditional random fields (crf) @xcite , and random forests @xcite for ade detection. @xcite introduced one corpus by collecting data from social media, focusing on adverse drug reactions. tasks included automatically classifying user posts, extracting specific mentions, and normalizing mentions to standardized concepts. with the availability of annotated data, in recent times, the rise of deep learning techniques has significantly influenced research methodologies, leading to the adoption of deep learning models for predicting ades. @xcite explored the synergy between crf and recurrent neural networks (rnn), demonstrating that crf enhances the rnn model's ability to capture contextual information effectively. @xcite developed a multi-task architecture that simultaneously tackled binary classification, adr labeling, and indication labeling, using the psb 2016 social media dataset @xcite . ## corpus development our study began with a thorough literature review to identify existing ade-related datasets. we discovered four text-only datasets: psb 2016 social media shared task @xcite comprising 572 tweets, medline ade corpus @xcite with 4,272 sentences, cadec @xcite ) containing 1,248 sentences, and recently released biodex dataset @xcite . this revealed a notable gap in multimodal ade datasets, where images complement textual data. we take this opportunity to introduce a multimodal corpus consist-image text chlorhexidine-induced tongue discoloration and tongue discoloration was predominantly reported in the drug categories anti-infectives for systemic use and dermatologicals a 37-year-old woman presented to our department with fever and an extensive papulopustular eruption on the face. the skin lesions had started about 2 weeks ago and had worsened with topical metronidazole gel. ing of 1,500 ade images with corresponding english sentences to facilitate further research. while preparing this corpus, we carry out the following steps. ## problem formulation each data point in the dataset encompasses a patient's textual description t along with a corresponding image i illustrating their medical issue or concern visually. the textual representation comprises a sequence of words {t 1 , t 2 , . . . , t n }, while the visual elements are represented by @xmath0, where w and h denote the width and height of the image data, respectively. the objective is to process both the text t and image i for each patient and generate a natural language sequence y that seamlessly integrates both modalities, expressed as @xmath1, i}. ## methodology recent advancements in vlms, such as blip, in-structblip, and git, have showcased remarkable advancements in encoding both textual and visual inputs. these models outperform traditional approaches that rely solely on individual image or text encoders, followed by fusion. by integrating sophisticated mechanisms for joint representation learning, vlms excel at capturing intricate relationships between textual and visual modalities, thereby enhancing their ability to generate more contextually relevant and coherent outputs @xcite . in the proposed work, we have leveraged instructblip @xcite , known for its exceptional performance across a range of vision-language (vl) tasks including visual question answering (vqa), image captioning, and image retrieval. each patient's inquiry or concern is articulated as a textual sentence along with a visual image where they elaborate on their medical queries or concerns on social media platforms or healthcare blogs to obtain pertinent feedback or advice. instructblip integrates two distinct encoders specialized for separate modalities. this architecture includes an image encoder, which utilizes a vision transformer (vit) to extract visual features, alongside an llm and a query transformer (q-former). the q-former interacts with the image encoder's output through cross-attention, resulting in k-encoded visual vectors, which are then linearly projected and fed into the frozen llm for further processing. this representation serves as input for a proficient language model, which generates high-quality text. during instruction tuning, only the q-former undergoes fine-tuning, while the image encoder and llm remain unchanged. figure 6 illustrates the detailed architecture. we fine- please refer to section 12 of the appendix for the fine-tuning details. ## experimental results and analysis we have utilized four commonly employed evaluation metrics to assess the performance of the models, including bleu score (bilingual evaluation understudy) @xcite , rouge score (recall-oriented understudy for gisting evaluation) @xcite , bertscore @xcite , and moverscore @xcite . detailed explanations of experimental settings and metrics are given in section 11 and 13 of the appendix. ## risk analysis while our multimodal model demonstrates promise, it is essential to have medical experts and pharmacovigilance teams validate the findings, considering other critical factors. our model and dataset are intended to support medical professionals rather than replace them. ## conclusion and future work in this paper, we present the task of ade detection within pharmacovigilance mining, leveraging multimodal datasets. in order to solve this task, we have created a multimodal ade dataset, mmade, containing images and corresponding descriptions, enhancing decision-making with the inclusion of visual cues. we have employed instructblip, finetuned with the proposed dataset, and compared it with other models. our findings suggest that domain-specific fine-tuning significantly enhances overall performance, emphasizing the importance of multimodal visual cues. we envision mmade as a pivotal resource for advancing research in multimodal ade detection. moreover, our finetuned architecture holds promise as a valuable tool for pharmacovigilance teams, clinicians, and researchers, facilitating more effective ade monitoring and ultimately improving patient safety and outcomes. in addition to expanding the dataset, future investigations could explore the potential of this multimodal dataset in tasks such as ade severity classification and summarization.