Efficient Joint Learning for Clinical Named Entity Recognition and Relation Extraction Using F ourier Networks:A Use Case in Adverse Drug Events (2022)

Abstract: Current approaches for clinical information extraction are inefficient in terms of computational costs and memory consumption, hindering their application to process large-scale electronic health records (EHRs). We propose an efficient end-to-end model, the Joint-NER-RE-Fourier (JNRF), to jointly learn the tasks of named entity recognition and relation extraction for documents of variable length. The architecture uses positional encoding and unitary batch sizes to process variable length documents and uses a weight-shared Fourier network layer for low-complexity token mixing. Finally, we reach the theoretical computational complexity lower bound for relation extraction using a selective pooling strategy and distance-aware attention weights with trainable polynomial distance functions. We evaluated the JNRF architecture using the 2018 N2C2 ADE benchmark to jointly extract medication-related entities and relations in variable-length EHR summaries. JNRF outperforms rolling window BERT with selective pooling by 0.42%, while being twice as fast to train. Compared to state-of-the-art BiLSTM-CRF architectures on the N2C2 ADE benchmark, results show that the proposed approach trains 22 times faster and reduces GPU memory consumption by 1.75 folds, with a reasonable performance tradeoff of 90%, without the use of external tools, hand-crafted rules or post-processing. Given the significant carbon footprint of deep learning models and the current energy crises, these methods could support efficient and cleaner information extraction in EHRs and other types of large-scale document databases.

## introduction adverse drug events (ades) are defined as any injury resulting from medication use and comprise the largest category of adverse events @xcite . serious ades have been estimated to cost from @xmath0137 billion in ambulatory settings in the us @xcite , and their costs have been doubling since then @xcite . due to safety concerns, between 21% to 27% of marketed drugs in the us have received black-box warnings or have been withdrawn by the food and drug administration (fda) within the first 16 years of marketing @xcite . clinical notes stored in electronic health record (ehrs) systems are a valuable source of information for pharmacovigilance @xcite . however, only 1% of ades recorded in ehrs are reported to ade registries, such as the fda adverse event reporting system (faers), while coded diagnoses have low sensitivity for ades @xcite . recognizing medication-related entities in clinical notes, extracting relations among them, and structuring this information can help identify ades in early stages of the drug marketing process, thus improving patient safety @xcite . the state-of-the-art for biomedical named entity recognition (ner) and relation extraction (re) is dominated by bidirectional lstm @xcite or bert @xcite architectures, combined with a crf @xcite layer and often hand-crafted rules @xcite @xcite @xcite . despite the high performance of end-to-end (e2e) ner+re models, they have some important limitations imposed by the model complexity, e.g., quadratic in terms of entity types in the crf layer or in terms of tokens in the dot-product attention mechanisms @xcite , which hinders their effective application in the biomedical domain due to its large number of entities and large size of free text databases. a particularity of ner and re for pharmacovigilance is that efficient recall of entities and relations is of utmost importance, as we would like to avoid missing a serious ade. nevertheless, cur-rent approaches tend to automatically discard long distance (or inter-passage) relations @xcite . moreover, ehr documents varies significantly in length, containing from a few hundred tokens for simpler patient records up to several thousand tokens for more complex patients (e.g., chronic diseases) @xcite . due to their computational complexity, these methods cannot process ehrs in their integrity without resorting to impractical and/or inefficient techniques such as windowing strategies @xcite @xcite . ongoing research is predominantly performancedriven, leading to a resurgence of resourceintensive models, neglecting the carbon footprint of deep learning models in favor of often marginal improvement in effectiveness @xcite @xcite @xcite . as a consequence of the technical constraints induced by highly complex models, these methods are currently being associated to a significant excess on carbon emissions @xcite . the most direct impact of training and deploying a machine learning model is the emission of greenhouse gases due to the increased hardware energy consumption @xcite . therefore, a direct way to reduce the ecological impact of training and deploying machine learning models is to reduce the training and inference time, i.e., providing the community with low memory and computational cost models. to tackle these limitations and issues, we propose the joint-ner-re-fourier (jnrf) model with a reduced algorithmic complexity for information extraction. we combine positional encoding with unitary batch size training so that the model processes automatically variable size ehrs with consistent performance. we use a fourier network to contextualize tokens with fair time and space complexity, allowing to process long documents with low-resource hardware and avoid rolling window strategies. finally, we reach the theoretical computational complexity lower bound for relation extraction using a selective pooling strategy and distance-aware attention weights with trainable polynomial distance functions. the main contributions of this paper are as follows: the main methods to produce e2e information extraction systems are the so called pipeline @xcite @xcite and joint modeling @xcite @xcite @xcite . the pipeline method consists of training two independent modules, one for ner and one for re. these models naturally suffer from cascading errors, as the error signal from one module is not back-propagated to the other. joint modeling aims to overcome this shortcoming by learning a unique model on a combination of ner and re losses. joint modeling tends to outperform pipeline methods, consistently achieving state-of-the-art performance @xcite @xcite @xcite . in addition, joint modeling techniques have some major advances as they allow to train two models at the same time, saving time and computation, and minimizing engineering efforts. in both cases, the e2e approach has been dominated by lstm-crf architectures @xcite @xcite . however, they suffer from two main limitations: i) the computational complexity of the crf layer @xcite ; and ii) the auto-regressive nature of the lstm model, which prevents full parallel training @xcite . ## approach in this section, we provide a step-by-step formal description of the proposed architecture using the forward pass representations and operations, as illustrated in figure 1 . first, we describe i) the vectorial token representation strategy, then ii) the language/contextualization layer, next iii) how the ner and re task is jointly modelled, and finally iv) the cost functions used. lastly, we conduct a computational complexity analysis of the proposed model. ## benchmark dataset we used the 2018 n2c2 ade dataset 1 to evaluate our model. the data consists of 505 annotated discharge summaries from mimic-iii @xcite . the passages contains annotations for strength, form, dosage, frequency, route, duration, reason, and ade entities, each associated with a drug entity. we used the official splits to train and evaluate our model, with 303 records for training and 202 for testing. data summary statistics are presented in the appendix a.1. duration and ade entities and their respective relations are not as well represented in the dataset (see ## experiments we trained our models in three different data representation scenarios, where we use whole documents, sentences only, and a mixed configuration where we use both documents and sentences as training instances. performance was then evaluated at both document and sentence levels for these different training scenarios. our models were compared to baseline models based on mlp with selective pooling and a sliding window bioclinicalbert-base model (wbert) @xcite with selective pooling, both trained and evaluated using the whole documents. we implemented our models using pytorch and a single tesla v100 gpu. we used adam (kingma and ba, 2014), mini-batches of size 1 and 64 for documents and sentences, respectively. models were trained using gradient accumulation to avoid using padding tokens. the final model was selected based on the best dev f1-score obtained during training. in the following, we present the results of our experiments using micro-lenient precision, recall, and f1-score using the challenge's official evaluation tool. ## comparison with sota in the n2c2 ade challenge in this section, for a reference we show our results against state-of-the-art e2e ner+re models described in the n2c2 ade challenge @xcite . nevertheless, due to their different modelling strategy (e.g., multiple models, external tools, post-processing techniques and hand-crafted rules specifically designed for this dataset), they are not directly comparable. uth @xcite used a joint learning model consisting of a lstm-crf layer for ner and a cnn-rnn layer for re. clamp @xcite was employed for text pre-processing, including sentence boundary detection and pos labeling, and to create a set of hand-crafted features that fed the crf layer. entities without a relation were associated to the closest drug in the post-processing step. nact @xcite used a majority voting ensemble of feature-based crf, including ade dictionary, and stacked bilstm-crf for ner. for re, they used an ensemble of lstm for intra-sentence relations and a transformer network for inter-sentence relations. bch @xcite used svm to detect entities, and pair these detected entities for a second svm relation classifier. they used ctakes @xcite to pre-process data and cleartk @xcite api to extract features. ra @xcite used dictionary-based features, crfs and logistic regression for ner. for re, they used a tree-based boosting classifier @xcite . ## conclusion in this paper, we proposed an end-to-end, generalizable, lightweight, and efficient model to jointly detect entities and multiple relations at the intraand inter-passage levels. we combined a fourier network with a pooled attention layer to significantly reduce time and space complexity, thus providing the community with a low carbon footprint solution for end-to-end relation extraction. we demonstrated that our model outperformed the sliding window bert with selective pooling by 0.42% in f1-score, while being 2 times faster to train. furthermore, we showed that our model trains 22 times faster and consumes 1.75 times less gpu memory than state-of-the-art bilstm-crf architectures, with a reasonable performance tradeoff of 90% on the n2c2 ade benchmark, without using external tools or hand-crafted rules. furthermore, we showed that this approach achieves consistent performance regardless of the length of the input sequence, eliminating the need for sliding window techniques and easing the overall data processing pipeline and engineering effort.