T p T - ADE : Transformer Based Two-Phase ADE Extraction (2024)

Abstract: Extracting adverse reactions to medications or treatments is a crucial activity in the biomedical domain. The task involves identifying mentions of drugs and their adverse effects/events in raw text, which is challenging due to the unstructured nature of clinical narratives. In this paper, we propose TpT-ADE, a novel joint two-phase transformer model combined with natural language processing (NLP) techniques, to identify adverse events (AEs) caused by drugs. In the first phase of TpT-ADE, entities are extracted and are grounded with their standard terms using the Unified Medical Language System (UMLS) knowledge base. In the second phase, entity and relation classification is performed to determine the presence of a relationship between the drug and AE pairs. TpT-ADE also identifies the intensity of AE entities by constructing a parts-of-speech (POS) embedding model. Unlike previous approaches that use complex classifiers, TpT-ADE employs a shallow neural network and yet outperforms the state-of-the-art methods on the standard ADE corpus.

## introduction adverse drug event (ade) is a negative or harmful patient outcome that seems to be associated with a medication or drug. analyzing the adverse events (aes) helps a practitioner to identify susceptible patients who may be at risk due to a particular drug. ade extraction has several uses: pharmaceutical companies can identify the sections of the population that were adversely impacted by the drug. for governments and regulatory authorities, ade information is the key to monitoring the performance of drugs already in the market and identifying any adverse effects that have not appeared during clinical trials. generally, ades are reported in an unstructured manner and are to be extracted from various sources like clinical narratives, medical journals, formal systems that report ades, etc. in some cases, the patients may report adverse events in social media posts, like "i got rashes on my back today after taking two tablets of amoxicillin yesterday". in this post, "rashes" is the adverse effect (ae) that could be caused by the drug "amoxicillin". identifying the drugs and adverse events and finding relations between them from such unstructured text is quite challenging due to the complex nature of the text containing multiple drugs and adverse events. we illustrate with an example below to understand the complexity of such texts. the text in red color is the ae and the text in blue is the drug name. example -"atypical ventricular tachycardia ae torsade pointes ae induced by amiodarone drug : arrhythmia ae previously induced by quinidine drug and disopyramide drug ." following are the drug, ae relations that could be extracted from the above example: disopyramide drug ,quinidine @xmath0, torsade de pointes ae ade extraction is a two-step process. the first step is identifying the mentions of the drugs and the aes from raw text. this is similar to the task of named entity recognition. in the second step, @xmath1, @xmath2, which can be cast as a classification problem. some methods @xcite train separate models for the two steps of ade extraction. in contrast to these works, @xcite @xcite @xcite proposed joint methods for ade extraction that perform better in both recognizing the entities and ade extraction tasks. a major drawback of the former approach is that if the first step of identification of drugs and aes entities is incorrect, then the ade extraction will also be incorrect, thereby resulting in poor performance due to the error propagation. also, the joint models have been proven to be effective in performing many related tasks such as partof-speech tagging and parsing @xcite , keyword extraction using joint modeling of local and global context @xcite , entity extraction and classification @xcite , entity and coreference extraction @xcite , and many more. in this paper, we introduce tpt-ade, a joint two-phase model for ade extraction from clinical texts by fine-tuning bert @xcite . in the first phase, tpt-ade identifies and standardizes mentions of entities such as drugs and adverse effects against the unified medical language system (umls) 1 . this ensures uniformity in naming across different mentions. the second phase uses this processed text to jointly extract entities and classify relations. our model employs a robust span-based extraction method, which can extract entities consisting of multiple successive tokens. that is, tpt-ade is able to extract overlapping entities. our approach can also detect the intensity of an adverse event, distinguishing between terms like "fever", "severe fever", and "mild fever". unlike previous works that rely on complex relational classifiers, tpt-ade uses a shallow neural network and yet achieves higher f1-score on the standard ade corpus @xcite . ## related work in this section, we discuss the related works in ade extraction. we first discuss the pipeline based approaches that extract ades by training separate models for entity extraction and relation extraction tasks. then, we discuss the joint methods that follow an end-to-end approach to extract ades. under the joint models, we discuss the related works that are bilstms based, graph convolutional networks based and span-based models. the pipeline based approaches @xcite @xcite are designed to complete one subtask and then go ahead with the next subtask. in the case of ade extraction, the output from the entity extraction model is passed as the input for the relation extraction task. both the models are trained separately with different loss 1 https://www.nlm.nih.gov/research/umls/index.html functions. @xcite @xcite use bilstm based models for ade extraction. @xcite employs the same bilstm based classifiers for both entity and relation extraction. other works train two different classifiers for the two tasks. @xcite employ a hybrid approach by combining feature based machine learning classifiers and neural networks. identifying negative entities is a crucial step for extracting ades. negative entities are those that are not drugs or adverse effects. towards this, @xcite proposed an attention based bi-lstm model that reduced the number of negative instances, helping to overcome the imbalance class problem. their method could also handle the discontinous entitiess. more recently, @xcite proposed an lstm based adaptive knowledge distillation model. the authors used bert to adaptively distill the knowledge to the lstm model. other recent works proposed in this regard are @xcite @xcite . joint entity and relation extraction methods @xcite @xcite have been recently proposed to capture the dependency between the two tasks in ade extraction. @xcite utilize character and word2vec embeddings to represent their input. then, they use bil-stm model combined with conditional random field (crf) model to jointly extract entities and their relations. @xcite @xcite cast the ade extraction problem as a table-filling problem. these methods construct a table that jointly represents the entities and relations and each element in the table depicts the presence of a relation between entities. then, the relation triples are extracted from the filled table. @xcite constructed a partition filter network to learn the feature representations that can classify entities and the relations. then, the relation triples extracted by following a table-filling approach. similarly, @xcite proposed a table-filling method that learns contextualized representations to compute entity mentions and capture long-range dependencies. for relation extraction, a tensor dot product is used to predict the relation labels. however, these tablefilling methods are computationally expensive due to building and deconding these tables for relation triples @xcite . span-based methods @xcite @xcite have shown remarkable performance in obtaining contextualized representations. in contrast to works that follow the bio (beginning, inside, outside)/bilou (beginning, inside, last, outside, unit)/bies(begin, inside, end, single) @xcite , span-based approach can identify the overlapping entities. in our work, we follow a span-based approach and combine bert @xcite with pos embedding model. in contrast to the previous works, we follow a two phase joint modelling approach that standardizes the entity mentions in the input text with their representative terms. in addition, unlike the above works that use complex classfiers, we use a shallow neural network for ade extraction. ## methodology in this section, we detail the two phases of tpt-ade model and training it. in the first phase phase i, we extract the entities and represent them with their standard medical terms. in section 4, we show the effectiveness of this step. the second phase, phase ii utilizes this processed text for ade extraction. ## evaluation in this section, we present the evaluation of tpt-ade and compare it with the state-of-the-art (sota) methods. we first start by describing our dataset. next, we present the evaluation results of our model against the sota methods. lastly, we perform ablation studies with various variants of our model and show the effectiveness of various components of our model.