PHEE : A Dataset for Pharmacovigilance Event Extraction from Text (2022)

Abstract: The primary goal of drug safety researchers and regulators is to promptly identify adverse drug reactions. Doing so may in turn prevent or reduce the harm to patients and ultimately improve public health. Evaluating and monitoring drug safety (i.e., pharmacovigilance) involves analyzing an ever growing collection of spontaneous reports from health professionals, physicians, and pharmacists, and information voluntarily submitted by patients. In this scenario, facilitating analysis of such reports via automation has the potential to rapidly identify safety signals. Unfortunately, public resources for developing natural language models for this task are scant. We present PHEE, a novel dataset for pharmacovigilance comprising over 5000 annotated events from medical case reports and biomedical literature, making it the largest such public dataset to date. We describe the hierarchical event schema designed to provide coarse and fine-grained information about patients’ demographics, treatments and (side) effects. Along with the discussion of the dataset, we present a thorough experimental evaluation of current state-of-the-art approaches for biomedical event extraction, point out their limitations, and highlight open challenges to foster future research in this area.

## introduction pharmacovigilance is the pharmaceutical science that entails monitoring and evaluating the safety and efficiency of medicine use, which is vital for improving public health @xcite . unexpected adverse drug effects (ades) could lead to considerable morbidity and mortality @xcite . it has been reported that more than half of ades are preven collecting and maintaining the clinical evidence for pharmacovigilance can be difficult because it requires time-consuming manual curation to capture emerging data about drugs @xcite . much of this information can be found in unstructured textual data including medical literature, notes in electronic health records (ehr), and social media posts. using nlp methods to discover and extract adverse drug events from unstructured text may permit efficient monitoring of such sources @xcite @xcite . past work has introduced pharmacovigilance corpora to support training and evaluation of nlp approaches for ade extraction. however, most of these datasets (e.g., the ade corpus; @xcite ) contain annotations only on entities (such as drugs and side effects) and their binary relations as shown in figure 1(a ). this ignores contextual information relating to human subjects, treatments administered, and more complex situations such as multi-drug concomitant use. to address this problem, @xcite developed the phaedra corpus, which includes annotations of not only drugs and side effects, but also subjects (human, specific species, bacteria, and so on) and events encoding descriptions of drug effects, which involve multiple arguments, and event attributes -see figure 1(b) . despite these refinements, however, phaedra does not provide detailed, nested annotations such as dosages, conditions, and patient demographic details. this granular information may provide critical context to clinical studies. furthermore, phae-dra consists of only 600 annotated abstracts of medical case reports, making it challenging to train nlp models for pharmacovigilance events extraction since its annotations are in the document level and the actual annotated events are sparse. in this work we introduce a new annotated corpus, phee, for adverse and potential therapeutic effect event extraction for pharmacovigilance study. the dataset consists of nearly 5,000 sentences extracted from medline case reports, and each sentence features two levels of annotations. with respect to coarse-grained annotations, each sentence is annotated with the event trigger word/phrase, event type and text spans indicating the event's associated subject, treatment, and effect. in a fine-grained annotation pass, further details are marked, such as patient demographic information, the context information about the treatments including drug dosage levels, administration routes, frequency, and attributes relating to events. an example annotation is shown in figure 1(c ). using phee as the benchmark, we conduct thorough experiments to assess the state-of-theart nlp technologies for the pharmacovigilancerelated event extraction task. we use sequence labelling and (both extractive and generative) qabased methods as baselines and evaluate event trigger extraction and argument extraction. the extractive qa method performs best for trigger extraction with the exact match f1 score of 70.09%, while the generative qa method achieves the best exact match f1 score of 68.60% and 76.16% for the main argument and sub-argument extraction, respectively. further analysis shows that current models perform well on average cases but often fail on more complex examples. our contributions can be summarised as follows: 1) we introduce phee, a new pharmacovigilance dataset containing over 5,000 finely annotated events from public medical case reports. to the best of our knowledge, this is the largest and most comprehensively annotated dataset of this type to date. 2) we collect hierarchical annotations to provide granular information about patients and conditions in addition to coarse-grained event information. 3) we conduct thorough experiments to compare current state-of-the-art approaches for biomedical event extraction, demonstrating the strength and weaknesses of current technologies and use this to highlight challenges for future research in this area. ## related work pharmacovigilance related corpora prior pharmacovigilance-related corpora mainly has focused on annotation of entities (e.g., drugs, diseases, medications) and binary relations between them, namely, drug-ade relations @xcite @xcite , disorder-treatment relations @xcite @xcite , and drug-drug interactions @xcite @xcite . more recent open challenges, including the 2018 n2c2 shared task @xcite and made1.0 challenge @xcite , have considered annotating additional relation types, such as drug-attribute and drug-reason relations, but they are still binary relationships. @xcite introduced the phae-dra corpus, extending the drug-ade annotations to pharmacovigilance events. compared to corpora that only annotate simple drug-ade relationsreferred to as ae events in phaedra-they further annotate three additional relations, namely the potential therapeutic effect (pte) event which refers to the potential beneficial effects of drugs, the combination and the drug-drug interaction event which indicates multiple drug use and interactions between administered drugs, respectively. in addition, phaedra includes the subject as a type of named entities (nes) and annotates three types event attributes, i.e., negated, speculated and manner. however, some key informative details are still missing in phaedra. as the ne annotation of phaedra is usually a single noun or a short noun phrase, detailed information about the subject (such as age and gender), and of the medication (e.g., dosage and frequency) is not captured. we set out to annotate a larger corpus with more detailed information to facilitate training of pharmacovigilance event extraction models. we build on existing corpora (phaedra and ade). the ade corpus comprises ∼3,000 medline case reports and annotations on ∼4,000 sentences indicating adverse effects, but their annotations only involve drugs, dosages and adverse effects, and lack sufficient event details of interest. the phae-dra corpus reuses 227 abstracts from ade and integrates an additional 370 abstracts (from other corpora and some novel entries). however, the phaedra corpus is annotated at the document a 52-year-old black woman on phenytoin therapy for post-traumatic epilepsy developed transient hemiparesis contralateral to the injury. subject pharmacological_substance potential_therapeutic_effect disorder adverse_effect disorder has_agent affects affects has_subject has_agent subject_disorder has_subject a 52-year-old black woman on phenytoin therapy for post-traumatic epilepsy developed transient hemiparesis contralateral to the injury. drug adverse_effect has a 52-year-old black woman on phenytoin therapy for post-traumatic epilepsy developed transient hemiparesis contralateral to the injury. gender race age subject drug treatment treat-disorder adverse_event severity_cue effect level, the actual annotated events are very sparse. we collected sentences in ade and those in phae-dra with ae or pte event annotations and enriched these using our proposed annotation scheme. ## experiments we conduct evaluation of sequence labelling and qa-based methods (both extractive and generative) on our phee dataset. we describe our experimental design, evaluation metrics, and main results in this section. reproduction details are in appendix d. ## challenges and future directions our analysis of experimental results, suggests the following open challenges for the extraction of pharmacovigilance events. firstly, the models perform poorly on arguments with similar entity mentions but different argument roles. for example, a disease mentioned in text could be annotated as treatment.disorder if it is the target of the treatment or subject.disorder if it refers to someone's disease but not targeted for treatment. a similar problem can be observed for arguments of temporal expressions such as time elapsed and frequency. the poor performance on such arguments seems to indicate that existing models are not able to perform deep semantic analysis. additional constraints encoding linguistic constructs between entity mentions and main argument types could be explored to guide the event extraction model through, for example, posterior regularisation. secondly, the models' performance on argument types with limited annotated training instances deteriorates drastically. one path forward is therefore to explore efficient few-shot learning strategies to improve models' generalisability on rare argument types. also, there might exist corpora annotated with similar argument roles but for different purposes, for example, the corpora for medication extraction @xcite where drug dosage and frequency are annotated. it is possible to leverage external drug or disease knowledge through knowledge distillation. finally, none of the existing models cope well with the presence of multiple events in a sentence. this is mainly because existing annotations rely heavily on event triggers to differentiate events and require explicit linking between arguments and their respective event triggers. however, trigger identification itself is ambiguous and difficult even for human annotators. in some cases, multiple events could share the same trigger. for pipelinebased models, i.e., the qa models in our work, detection of multiple triggers is prone to error, thus making it hard for subsequent argument extraction due to error propagation. for the sequence labelling model, it is difficult to flatten the annotations of multiple events into a single label sequence. we thus duplicate the multi-event cases during training, and only provide a single event annotation for each case at one time. however, it becomes impossible to obtain full extract results for multiple events during the inference stage. in the future, rather than sequence labelling or qa-based extraction approaches, it is worth exploring graph-based approaches for multi-event extraction in which entity mentions are nodes in the graph while event extraction can be framed as soft clustering of entity mentions. ## conclusion in this paper, we present the development of a novel corpus, phee, composed of sentences from the medical case reports annotated with pharmacovigilance-related events. events in phee are hierarchically annotated with coarse and finegrained information about patient demographics, treatments, and (side) effects. we use it to evaluate state-of-the-art nlp models for pharmacovigilance event extraction. experimental results show that current models could capture reasonable information in common cases but face challenges for complex situations such as distinguishing semanticallysimilar arguments, dealing with the low resource setting, and extracting multi-events from text.