NADE : A Benchmark for Robust Adverse Drug Events Extraction in Face of Negations (2021)

Abstract: Adverse Drug Event (ADE) extraction models can rapidly examine large collections of social media texts, detecting mentions of drug-related adverse reactions and trigger medical investigations. However, despite the recent advances in NLP, it is currently unknown if such models are robust in face of negation, which is pervasive across language varieties. In this paper we evaluate three state-of-the-art systems, showing their fragility against negation, and then we introduce two possible strategies to increase the robustness of these models: a pipeline approach, relying on a specific component for negation detection; an augmentation of an ADE extraction dataset to artificially create negated samples and further train the models. We show that both strategies bring significant increases in performance, lowering the number of spurious entities predicted by the models. Our dataset and code will be publicly released to encourage research on the topic.

## introduction exploring social media texts is becoming more and more important in the field of pharmacovigilance @xcite , since it is common for internet users to report their personal experiences with drugs on forums and microblogging platforms. given the inherent noisiness of social media texts (colloquial language, slang and metaphors, non-standard syntactic constructions etc.), the natural language processing (nlp) community dedicated a consistent effort in developing robust methods for mining biomedical information from social media outlets. this also led to the creation of several dedicated shared tasks series on ade detection (smm4h -social media mining for health) @xcite @xcite @xcite . although these models have seen great advancements in the last years, also thanks to the introduction of pre-trained transformers-based architectures @xcite , it is still unknown how robust they are in face of some pervasive linguistic phenomena such as negation. however, general investigations on machine comprehension and question answering tasks confirmed that such phenomena often pose a serious challenge @xcite . managing to efficiently handle the scope of negations and speculations in clinical notes is a key problem in biomedical nlp @xcite , and similarly, for digital pharmacovigilance it is essential to recognize whether the association between a drug and an ade is actually being stated or negated because the consequences of extracting misleading information about the possible side effects of drugs can be extremely serious. in this paper, we analyze the performance of some of the latest state-of-the-art ade detection systems on nade: a new dataset derived from smm4h data, which contains a relevant amount of annotated samples with negated ades. we also introduce and analyze two strategies to increase the robustness of the models: adding a negation detection module in a pipeline fashion to exclude the negated ades predicted by the models; augmenting the training set with artificially negated samples. as a further contribution, our dataset and scripts will be made publicly available for researchers to test the robustness of their ade extraction systems against negation. foot_0 ## related work detecting negation scopes is a traditional topic of nlp research. an early, popular system was introduced by @xcite , whose negex algorithm exploited regular expressions to identify negations in clinical documents in english. later, machine learning approaches became more popular after the publication of a common gold standard, i.e. the bioscope corpus @xcite . several proposals consisted of a two-steps methodology: a first classifier to detect which token in a sentence is a negation/speculation cue, and a second classifier to determine which tokens in the sentence are within the scope of the cue word @xcite @xcite . more recently, approaches based on neural networks @xcite @xcite have been introduced in the literature, showing some crosslinguistic and crossdomain transferability. moreover, bert-based models have been proposed to handle this phenomenon @xcite , also with the aid of multitask learning architectures @xcite . to our knowledge, the research in biomedical nlp mostly focused on scope detection per se and on more formal types of texts (e.g. clinical notes, articles). in our research we focus instead on the specific task of ade detection and on the impact of negation on the performance of ade systems for noisy social media texts (e.g. tweets, blog posts), with the goal of making them able to distinguish between factual and non-factual information. ## nade dataset while there are several datasets for ade detection on social media texts @xcite , the biggest collection of tweets tagged for ade mentions is the one released yearly for the smm4h workshop and shared task. we used the following resources: a) smm4h19ext. the training set for the ade extraction task of smm4h19 @xcite , consisting of 2276 tweets that mention at least one drug name. 1300 of them contain ades, and annotations of their position in the text (ade class). the other 976 are control samples with no ade mentions (noade). as the blind test set is not publicly available, we rely on the training set only and use the splits by @xcite , which balance positive and negative samples; b) smm4h19 cls and smm4h20 cls @xcite . the training sets for smm4h classification tasks, containing tweets labeled as ade or noade (1:9 ratio). the community focused on the ade extraction task, so most datasets are made of samples that either do or do not contain an ade. because of this, they include a small number of negated ades by construction: no particular attention is given to these samples when curating the data and when they are present they are treated as noade samples and not explicitly labelled. this leads to their class being misrepresented and makes it harder to study this phenomenon. ## experiments all the reported results are the average over 5 runs. for the transformer models we used the same hyperparameters reported by @xcite . as metrics, we consider the number of false positive predictions (fp) and the relaxed precision (p), recall (r) and f1 score as defined in the smm4h shared tasks @xcite : the scores take into account "partial" matches, in which it is sufficient for a prediction to partially overlap with the gold annotation. as a preliminary step, the two negation detection models are trained and used to predict the negation scopes for all the test samples once. this allows us to compute the predictions of any pipeline model. exp 1: to provide a measure of the initial robustness of the base models and their general performance, we train them on the ade and noade samples only (842+785 samples). we then test the efficacy of the pipeline negation detection method, applying negex and bertneg to the base models. exp 2: to test the effect of augmenting the training data with artificial samples (i.e., the second exp 3: to investigate whether the two methods are complementary in their action, we combine the two strategies, applying the pipeline architecture to the models trained on the augmented dataset. ## conclusions in this paper, we evaluate the impact of negations on state-of-the-art ade detection models. we introduced nade, a new dataset specifically aimed at studying this phenomenon. the dataset proves to be a challenging setting and the experiments show that current methods lack mechanisms to deal with negations. we introduce and compare two strategies to tackle the problem: using a negation detection module and adding negade_g samples in the training set. both of them bring significant increases in performance. both the dataset and the code are made publicly available for the community to test the robustness of their systems against negation. future work should focus on more refined techniques to accurately model the semantic properties of the samples, also by jointly handling negation and speculation phenomena. this might be an essential requirement for dealing with the noisiness and variety of social media texts. the main short term directions are increasing the quality and quantity of real negade samples (possibly via crowdsourcing), and creating a model that is able to discard negade (keeping an high precision level), without sacrificing recall.