Extracting Semantic Aspects for Structured Representation of Clinical Trial Eligibility Criteria (2020)

Abstract: Eligibility criteria in the clinical trials specify the characteristics that a patient must or must not possess in order to be treated according to a standard clinical care guideline. As the process of manual eligibility determination is time-consuming, automatic structuring of the eligibility criteria into various semantic categories or aspects is the need of the hour. Existing methods use hand-crafted rules and feature-based statistical machine learning methods to dynamically induce semantic aspects. However, in order to deal with paucity of aspect-annotated clinical trials data, we propose a novel weakly-supervised co-training based method which can exploit a large pool of unlabeled criteria sentences to augment the limited supervised training data, and consequently enhance the performance. Experiments with 0.2M criteria sentences show that the proposed approach outperforms the competitive supervised baselines by 12% in terms of micro-averaged F1 score for all the aspects. Probing deeper into analysis, we observe domain-specific information boosts up the performance by a significant margin.

## introduction clinical trials (cts) are research studies that are aimed at evaluating a medical, surgical, or behavioral intervention @xcite , @xcite . through such trials, researchers aim to find out whether a new treatment, like a new drug or diet or medical device is more effective than the existing treatments for a particular ailment. from an organization's perspective, a successful completion of a trial depends on achieving a significant sample size of patients enrolled for the trial within a limited time period. ## problem formulation given an eligibility criteria sentence in the form of a word sequence @xmath0...., x n ) , where n is the maximum length of the sequence, the task is to predict an output sequence @xmath1, ...., y n ) in which each y i is encoded using standard sequence labeling encoding scheme. each y i might take one of the following aspects : ## data annotation to induce semantic categorization of aspects in the eligibility criteria, we generate a small pool of annotated data by manually examining some of the most frequently used n-gram patterns such as history of, upper limit of normal, treated by, allergy to as specified in @xcite in the initial phase. during pre-processing, we filter out the most frequently occurring n-grams (@xmath2, @xmath3, @xmath4, @xmath5. secondly, the criteria sentences are also tagged with cliner tagger @xcite for extracting out the diseases and drugs. further details of data are provided in the supplementary material foot_0 . after these two steps, finally, the false positives are being removed during manual supervision by four independent domain-expert annotators. these include annotations for each of the different categories. the mean cohen's kappa (mchugh, 2012) was 0.82, which indicate good inter-annotator agree-algorithm 1 aspect extraction using co-training algorithm input u : large amount of unlabelled criteria sentences, τ : co-training threshold, v 1 , v 2 : two views of labelled aspect annotated criteria sentences output model parameters : θ bilstm-crf , θ bigru-crf t 1 , t 2 ← v 1 , v 2 initialize the model parameters θ bilstm-crf , θ bigru-crf randomly. while (stopping criteria is not met) do c 1 ← train bilstm-crf on t 1 (minimize the aspect loss) c 2 ← train bigru-crf on t 2 (minimize the aspect loss) ## methodology in this work, we experiment with two different methods of aspect extraction. one of the following being the traditional supervised setup of using bilstm-crf/bi-gru crf with input representation optimized using categorical cross-entropy loss @xcite . the second one being the co-training @xcite ) method to extract the semantic aspects which has been outlined in algorithm 1. the later method uses two conditionally independent feature views of the same dataset illustrated below: at each step of co-training, the classifiers c 1 and c 2 are trained on respective views of training sets v 1 and v 2 , thereby minimizing the loss function. each instance from the unlabeled samples (u) is scored using a scoring function computed as follows. first, the current classifier is used to decode the output label distribution for each word in the unlabeled instances. for each word in the output, we choose the output label which has the maximum probability. we compute the score for the sample as the multiplication of the probabilities of each label type for all labeled words in sequence normalized by the total number of words in the sentence. if this confidence score of the sample is greater than some pre-defined threshold τ , the sample has been added to the training set of the other classifier along with its output labels as generated by the classifier. this is the process of generation of weak labels for each sequence. due to interchange of training data, both classifiers can learn from mistakes of each other and work in synergy. ## experimental details we implement the model using pytorch 0.3.0. the two classifiers considered for co-training are c 1 : bi-lstm-crf and c 2 : bi-gru-crf. for both supervised and co-training methods, the training data is divided according to 70-30% trainvalidation split. the two different views of cotraining setup are explained as follows: hyper-parameters for two independent views: we run two experiments based on co-training, one using contextual embeddings (c-ctr) and the other using context-independent embeddings (nc-ctr). the hyper-parameter settings for the two views as required by the co-training method are as follows: view 1: for the first view (v 1 ), we use bi-lstm-crf @xcite with domain-independent word embeddings. we experiment with both a) (nc-ctr) word2vec embeddings trained on googlenews corpus with dimension 300 b) (c-ctr) pre-trained bert-base (12 layers, 12 attention heads, and 110 million parameters). view 2: ## results and analysis in this section, we have provided a detailed analysis of the various results and findings that we have observed during experimentation. there are various criteria on which we have tried to evaluate our semi-supervised approach. comparison with the baselines: the results of the baseline methods are enumerated in feature ablation on model architecture: for the purpose of fair comparison, we experiment with different ablations of feature extractor and types of input representation (in the supervised setup) and present the results of macro-averaged f1-score in ## conclusion in this paper, we have proposed a semi-supervised co-training method to tackle the scarcity of annotated data for the semantic clinical aspect extraction. this method augments a limited pool of annotated data with a large number of unlabeled clinical eligibility criteria outperforming pure supervised approaches. to the best of our knowledge, we are the first to provide an effective semi-supervised approach to detect the semantic aspects from clinical eligibility criteria which is a promising direction for further research on automatic linking of the patient electronic health records (ehr) to clinical eligibility criteria with promising performance. as a future work, we aim to propose an end-to-end automatic matching system for patient-based clinical trial eligibility with low-cost data annotation.