Joint Learning with Pre-trained Transformer on Named Entity Recognition and Relation Extraction Tasks for Clinical Analytics (2020)

Abstract: In drug development, protocols define how clinical trials are conducted, and are therefore of paramount importance. They contain key patient-, investigator-, medication-, and study-related information, often elaborated in different sections in the protocol texts. Granular-level parsing on large quantity of existing protocols can accelerate clinical trial design and provide actionable insights into trial optimization. Here, we report our progresses in using deep learning NLP algorithms to enable automated protocol analytics. In particular, we combined a pre-trained BERT transformer model with joint-learning strategies to simultaneously identify clinically relevant entities (i.e. Named Entity Recognition) and extract the syntactic relations between these entities (i.e. Relation Extraction) from the eligibility criteria section in protocol texts. When comparing to standalone NER and RE models, our joint-learning strategy can effectively improve the performance of RE task while retaining similarly high NER performance, likely due to the synergy of optimizing toward both tasks’ objectives via shared parameters. The derived NLP model provides an end-to-end solution to convert unstructured protocol texts into structured data source, which will be embedded into a comprehensive clinical analytics workflow for downstream trial design missions such like patient population extraction, patient enrollment rate estimation, and protocol amendment prediction.

## introduction clinical trial protocols, often called "study protocols" or just "protocols", are the foundational documents that specify the detailed plans of conducting clinical trials to validate the safety and/or efficacy of drugs. they contain key information about the targeted disease indications, the eligible patients, the investigated medication, the visit schedules, and the treatment endpoints etc. across the entire lifecycle of clinical trials starting from study design & planning to data analysis & publication, it is always critical to comprehend this information accurately and unambiguously. however, since protocols are mainly unstructured or semi-structured texts (i.e. natural languages), application of computer-aided information extraction is challenging and thus limited. current protocol analytic practices are labourand time-intensive, involving numerous manual resource checking and cross referencing activities. the pressing needs of reducing the costs and boosting the speed of drug development have created an industry-wide demand in developing a more efficient, effective, and scalable mechanism to process text-based protocols. to address the above demand, we present in this paper our efforts and progresses in developing a deep learning natural language processing (nlp) approach to extract clinically relevant information from protocols. in particular, we targeted two tasks: named entity recognition (ner) and relation extraction (re), and transferred the bidirectional encoder representations from transformers model (bert, a pre-trained transformer nlp model) via a joint-learning strategy to extract clinically relevant entities and their syntactic relationships simultaneously by training on our in-house clinical trial protocol corpus. in alignment with the industry's patient-centric business emphasis, we focused this work on extracting the patient eligibility information from the "eligibility criteria" section in the protocols, which unambiguously determines whether a patient could be included in or excluded from the clinical trial. this is particularly important because patient recruitment is an essential and currently rate-limiting step in clinical trials. accurate parsing of this part of protocols can facilitate quick identification of eligible patients as well as other clinical analytics missions. clinical trial protocols are a type of professional documents with rigorous and highly domainspecific terms associated via complex yet precise relations. like other professionally developed documents, protocols have to pass multiple qualitycontrol checkpoints, and thus require less preprocessing (e.g. text correcting/cleaning) than many other types of documents such as social media posts before submitting to nlp models. on the other hand, the domain-specific nature of protocols requires extra attention when transferring models trained from generic or other professional domains. to elaborate, a protocol contains many clinical and medical terms (e.g. medications and diseases etc.) that are not commonly seen in other domains, but those are exactly the entities that our model needs to recognize; furthermore, it is also challenging that the entities may be connected in dramatically different ways under different domain-specific contexts. for example, in the clinical domain, the word "trial" refers to "clinical trial" that is associated with "patients", "diseases", and "medicines" etc.; whereas in the legal or even generic domains, "trial" commonly means "legal trial" that is frequently connected to "jury", "prosecutor" and "defendant" etc. therefore, the success of the transfer-learning largely relies on maximizing domain-specific "gradients" for fine-tuning the model parameters. this presented work is continued from our recent study on clinical protocols, in which we developed standalone bert-based nlp models for ner and re tasks for processing the "eligibility criteria" section in protocols @xcite . based on the observation that different clinically relevant entities are not equally involved in all relations, we hypothesized that by combining the ner and re tasks in the same bert network via a jointlearning strategy, the textures of clinical trials may become more visible for training and thus improve the performance of both tasks. as will be shown in the later sections of this paper, our results validated this hypothesis and showed that the joint-learning model can provide significant improvement over standalone models. this improved model is being embedded into an automated pipeline that aims to accelerate the current manual process of identifying similar clinical trials from the historical protocols, and to streamline the querying process of identifying potentially eligible patients for clinical trials. with regard to re, it is usually treated as a text classification between the interested entity pairs. many classification algorithms, such as support vector machine, logistic regression, perceptron etc., have been applied to this problem @xcite . similar to ner, the latest developments in solving re tasks have also employed deep learning algorithms using neural network models to emulate entity relations with components such as attention, biaffine, and bidirectional tree-structured lstm-rnns (nguyen and verspoor, 2019; @xcite . pre-trained models were also used to provide contextualized encoding information to the neural network layers for the re task @xcite . although they can be tackled independently, ner and re tasks are in fact synergistically connected: if we knew two entities and their types in a sentence, it would be easier to classify their relations; similarly, if we knew the relation between two phrases, then it would be less challenging to label their entity types. this has naturally motivated efforts in joint or multi-task learning for ner and re, hoping to achieve better performances in both tasks by simultaneously training the same neural network towards combined objectives. despite the differences in their details, the practices in ner and re joint learning usually share a general highlevel architecture: they sequentially stack together the word and sequence embedding layers, the ner prediction layer, the ner entity embedding layer, and the relation representation/handling layers. for word and sequence contextualized embedding layers, where many network variations be present, researchers have investigated using bilstm, rnn, and bert pre-trained transformers. @xcite @xcite @xcite . these studies usually emphasized more on evaluating different joint models, leaving the comparison between joint and standalone models to be investigated. pre-trained transformer models, e.g. bert, xl-net, and gpt, have achieved state-of-the-art performance across a great number of benchmark nlp tasks @xcite @xcite . they provide the benefits of representing bidirectional context and encoding text sequence by a series of attention layers. from the transfer learning standpoint, various nlp tasks can be treated as downstream tasks appended to the pre-trained models, and the pre-trained parameters (usually from large scale corpora in a generic domain) together with the nlp task specific parameters are fine-tuned via continued training on a relatively smaller and task-specific training data set. to enhance domain specificity, bert has also been customized and retrained on specific domains such as the biomedical domain against relevant corpus, examples including biobert, clinicalbert, and scibert @xcite @xcite . also, there has been a surge in studies applying bert in specific nlp contexts for fine-tuning tasks such as predicting hospital re-admission, extracting bacteria-biotope relations, biomedical named entity normalization, etc. @xcite @xcite . we have previously investigated the applications of fine-tuning pre-trained bert models on a protocol corpus for ner and re tasks separately. encouraged by many successful studies on pretrained transformer and joint models, we continued to explore joint-learning strategies combined with bert to co-train ner and re tasks against our in-house clinical protocol corpus. we abstracted a neural architecture from two popular joint models and experimented with a number of variations @xcite . we believe these continued efforts not only help selecting the best-performing model for our applications, but also provide a comprehensive understanding of various transfer learning strategies' performance under a real-world setup, shedding light on developing business-oriented ai applications for the healthcare and clinical trial industry. the sentences were manually annotated by biomedical experts. clinically relevant entities and the associated entity relations are labelled based on our annotation guideline. we used the bio tag format to denote the beginning, inside, and outside of the entities (ramshaw and marcus, 1999). we focused on 15 types of entities and 7 types of syntactic relations (as shown in we used bert pre-trained transformer as the embedding/encoding layer. the ner layer occurs after the bert layer and uses softmax for ner classification. more specifically, it takes the bert output vectors as its input and first passes through a fully connected layer and then the output layer where ner labels are classified using softmax function @xcite . the ner classification loss function based on cross-entropy is: following the ner layer, we appended an ner label embedding layer, which is concatenated with the outputs from the previous bert layer to serve as the input for the subsequent re task. because an entity could be paired with other entities before or after it in a sentence, it should be mapped differently in these 2 cases. in our model, the entity vectors are processed in the relation pair handling layer, by 1) mapping them using a fully connected layer to head vectors for representing entities as heads in a pair, and 2) mapping them using a parallel fully connected layer to tail vectors for representing tails in a pair. then an entity pair, composed of a head and a tail vector, employs a classification function, being either softmax or biaffine function, to produce the re classification result, i.e., the relation type between the two entities. more details about the re model variations can be found in section 4.2.1. the re loss function is also cross-entropy based: ## conclusion and future work in this reported work, we employed joint-learning models to identify entities and relations in clinical protocols by using pre-trained transformer nlp deep learning models. to the best of our knowledge, this is the first attempt to tackle the ner and re tasks in a joint and pre-trained deep learning fashion on real-world protocols, which is inherently a corpus of high complexity. our contribution is three fold: 1) we abstracted a neural network architecture from literature combining pre-trained transformer model with joint learning for ner & re tasks, 2) we experimented with different model options based on the joint learning network architecture, 3) we examined performance on a complex clinical corpus, which is a less studied but highly impactful domain for such tasks. our results demonstrated that joint-learning models can greatly improve re performance over the standalone models despite of a minor decrease in the ner performance. among all the evaluated joint-learning strategies, the biaffine re model, gold-standard based negative sampling, together with the bert pre-trained model, led to generally better performance than other strategies. these results provide evidence on the effectiveness of using joint and deep learning on parsing clinical protocol text, and thus for future work, we will continue exploring more sophisticated joint and multi-task learning network architectures to further enhance the ner and re parsing performance.