ADEQA : A Question Answer based approach for joint ADE -Suspect Extraction using Sequence-To-Sequence Transformers (2023)

Abstract: Early identification of Adverse Drug Events (ADE) is critical for taking prompt actions while introducing new drugs into the market. These ADEs information are available through various unstructured data sources like clinical study reports, patient health records, social media posts, etc. Extracting ADEs and the related suspect drugs using machine learning is a challenging task due to the complex linguistic relations between drug ADE pairs in textual data and unavailability of large corpus of labelled datasets. This paper introduces ADEQA, a question- answer(QA) based approach using quasi supervised labelled data and sequence-to-sequence transformers to extract ADEs, drug suspects and the relationships between them. Unlike traditional QA models, natural language generation (NLG) based models don’t require extensive token level labelling and thereby reduces the adoption barrier significantly. On a public ADE corpus, we were able to achieve state-of-the-art results with an F1 score of 94% on establishing the relationships between ADEs and the respective suspects.

## introduction everyday hundreds of drugs are being introduced to the market. however, every drug has contraindications. a study conducted by @xcite showed that 7000 deaths are being caused by adverse drug events (ade) annually. organizations like the world health organization (who), the food and drug administration (fda), the european medicines agency (emea), and the medicines and healthcare products regulatory agency (mhra) maintain a reporting system that enables individuals to spontaneously report the experienced adverse effects related to the use of medicines or healthcare products @xcite . although these systems store the adverse event information in a structured format, a vast amount of information still remains in the unstructured textual data like clinical trial reports, patient health records, medical transcripts, social media posts, etc. it's a tedious process to have humans go through each of these documents and record the mentioned adverse events and the related suspect drugs. with the advancements in machine learning, specifically in the field of natural language processing (nlp), information extraction models are being widely used to extract useful information from unconstrained texts. such models can learn contextual patterns to identify and extract specific entities, after being trained using large corpus of annotated data. similar approaches have been applied to extract ades and suspect drugs using named entity recognition models @xcite . however, since the ades are semantically similar to any other unrelated symptoms, most often such models predict false positives. hence, improving precision in this task depends on the ability to contextually relate the ades to the relevant suspect drug(s), instead of extracting the ades independently. unfortunately, generalized extraction of relationships among entities @xmath0, predicate, @xmath1. in this work, we wanted to address this shortcoming by modelling the drug-ade relationship extraction task as question answering tasks. deep neural network based, supervised nlp models require tens of thousands of annotated data to learn the contextual information and identify hidden patterns. for tasks like ner (wikipedia contributors, 2023), annotation of text data is a critical prerequisite needing manual effort, coupled with domain knowledge. the classical approach of annotating entities with b-i-o offsets @xcite increases the efforts further. considering the ongoing exponential growth of data, annotating huge corpus of new data to train or retrain the models in future would be very expensive, if not entirely impossible. this is where models which require light weight labelling come into the rescue. sequence to sequence transformer @xcite models like t5 @xcite ) can be trained by transforming the entities and relationships to be extracted into text sequences, so that these models can learn the contextual patterns to identify and generate the required entities, without any explicit token offset information. in this paper, we intend to elaborate two approaches of modelling drug -ade relation extraction as question-answering solution using natural language generation (nlg) technique via sequence-to-sequence modelling. first approach is a two-step solution that first extracts drugs and ades and subsequently confirms associations between them, while the second one directly discovers the potential drug -ade pairs from a given text. one of our approach achieved state-of-the-art f1 scores of 94% in establishing the relationships between ades and the respective suspects on the public ade benchmark corpus (gurulingappa h, 2012). ## related works several works have already been tried out for adesuspect identification task. earlier approaches focused mainly on pipeline design with a ner model to extract entities and their offsets, followed by a relation classification model which takes two pair of entities and identify the relation between them (gurulingappa h, 2012) @xcite . with the advancements in deep learning, rnn based sequence models like lstms (hochreiter and schmidhuber, 1997), grus @xcite started being applied on all nlp use cases. @xcite used a feed forward neural network to jointly extract drug-disease entity mentions and their relations. (li et al., 2017) explored bidirectional lstms for learning entity representations from text sequences. they used shortest dependency paths (sdp) between probable entities to identify related ades and suspects. attention based models like transformers @xcite which can learn contextual patterns efficiently have more or less replaced lstm based models lately. @xcite (alimova and tutubalina, 2020) applied pretrained bert @xcite models for the ade extraction task. @xcite eration to classify the relationships. multi-turn qa @xcite also casts the ner-re problem as a multi-turn question answering task. mrc4ere @xcite improves on this question answering approach by leveraging a diverse set of questions. however, both the approaches consider deterministic methods for extracting the answers and uses bert @xcite for modeling. sequence-to-sequence transformer models like bart @xcite , t5 @xcite , etc. are being studied for ade-suspect extraction task recently and shown positive results. our work is heavily inspired from some of the latest researches which adopted nlg models for the relation extraction tasks. rebel @xcite which achieved state of the art results in multiple re benchmark datasets transformed the entity relationships as text sequence of triplets and used bart @xcite to generate these triplets. similarly, tanl by @xcite frame this as a translation task by generating an augmented text with entity and relation information marked. @xcite explored t5 model for medical product safety monitoring in social media. ## dataset we have used the ade dataset (gurulingappa h, 2012), which is an annotated data of adverse events and drugs identified from biomedical texts. the original corpus is distributed in three files i.e., drugade relation data, drug-dosage relation data and ade-negative data, out of which we used the drugade data for our experiments. although there are begin and end offsets annotated for the ades and suspects, our approaches do not require that information. there are 6,821 texts available in the corpus with only 20% of them including more than one ade or suspect, as seen in fig. 1 and 2. altogether, there are 2984 unique ades and 1050 unique suspects in the whole corpus. a sample row from the dataset looks like this "10030778|intravenous azithromycin-induced ototoxicity.|ototoxicity|43|54|azithromycin|22|34". columns 2, 3, and 6 provide the text, ade, and suspect information, respectively. ## approaches sequence-to-sequence transformer models like t5 are capable of handling several nlp tasks concurrently. as explained in @xcite , every nlp task we consider including translation, question answering, classification, etc. is cast as feeding the model text as input and training it to generate some target text @xcite . this allows us to use the same model, loss function, hyperparameters, etc. across diverse set of tasks. in order to train a single model on the diverse set of tasks described above, t5 cast all of the tasks we consider into a "text-to-text" format that is, a task where the model is fed some text for context or conditioning and is then asked to produce some output text @xcite . t5 framework provides a consistent training objective both for pre-training and fine-tuning. specifically, the model is trained with a maximum likelihood objective (using "teacher forcing" @xcite ) regardless of the task @xcite . to specify which task the model should perform, we add a task-specific (text) prefix to the original input sequence before feeding it to the model @xcite . we use questions as the prefix in our experiments. algorithm 2 joint ade-suspect relation extraction as single task 1: input: 2: @xmath2: @xmath3: output: 5: @xmath4: start: 7: @xmath5: for every text do 9: @xmath6: @xmath7: @xmath8, c) 12: example @xmath9: ## experiments and setup for the first approach of ner-re pipeline, as shown in fig. 3 , we used a single model to execute multi-task learning in order to identify suspects, ades, and examine the relationships between the two. this is equivalent to mapping an input sequence of n words to an output sequence of m ades or supects, conditioned over a question and a context as shown in ( 1 ) and ( 2 ). we employed questions such as "what are the ades?" and "what are the suspects?" for ades and suspects extraction, respectively. since there can be multiple ades and suspects within the same text, the model should be able to generate all available entities. we used a special @xmath10. @xmath11, we removed the repetition penalty in the t5 model. additional post processing was used to eliminate duplicate results. in order to execute our second approach of endto-end extraction, we used a single question like "what are the ades and suspects?" and tuple generation method similar to @xcite . we used additional tokens @xmath12, <sus-@xmath13. 4 . it was also observed that, unlike @xcite we didn't have to perform any entity sorting, based on the positions in the text. model was able to perform the extractions of tuples accurately without sorting. this is equivalent to mapping an input sequence of n words to an output sequence of m pairs of ades and supects, conditioned over a question and a context as shown in (4). p(y ## results comparative performance of both the approaches are shown in individual metrics for ade, suspect and re from the approach 2 are evaluated by splitting the result into ades and suspects and by checking if a pair of ade-suspects are related correctly or not. it's also observed that the approach 2 suffers when there are more than 3 pairs of ades and suspects with in a text. as seen in fig. 6 , identification of ades and suspects is most effective when there are 3-4 or less of these entities per text. ratio of correct to wrong prediction goes up as the numfigure 6: performance of the model on ade/suspect extraction when the number of ades/suspects per text varies. ## conclusion in this paper, we propose a question answer based approach for solving ade-suspect extraction problem by using a sequence-to-sequence transformer architecture, t5 @xcite . we also detail our performance relative to the current baselines and present several experiments carried out utilizing various qa methodologies. we found that qa based re approach outperforms existing baselines on the benchmark dataset. for industry usecases, it is recommended to use state-of-the-art ner followed by our qa based re modeling for best results. this approach can be extended to extract ades and suspects from social media posts, clinical trial docs, medical transcripts, etc. we think that this work will be helpful when introducing a specific drug to the market or researching the negative effects of an existing drug since it will enable quick decisions to be made with little delay, preventing future causalities.