Noise Pollution in Hospital Readmission Prediction: Long Document Classification with Reinforcement Learning (2020)

Abstract: This paper presents a reinforcement learning approach to extract noise in long clinical documents for the task of readmission prediction after kidney transplant. We face the challenges of developing robust models on a small dataset where each document may consist of over 10K tokens with full of noise including tabular text and task-irrelevant sentences. We first experiment four types of encoders to empirically decide the best document representation, and then apply reinforcement learning to remove noisy text from the long documents, which models the noise extraction process as a sequential decision problem. Our results show that the old bag-of-words encoder outperforms deep learning-based encoders on this task, and reinforcement learning is able to improve upon baseline while pruning out 25% text segments. Our analysis depicts that reinforcement learning is able to identify both typical noisy tokens and task-specific noisy text.

## introduction prediction of hospital readmission has always been recognized as an important topic in surgery. previous studies have shown that the post-discharge readmission takes up tremendous social resources, while at least a half of the cases are preventable @xcite . clinical notes, as part of the patients' electronic health records (ehrs), contain valuable information but are often too time-consuming for medical experts to manually evaluate. thus, it is of significance to develop prediction models utilizing various sources of unstructured clinical documents. the task addressed in this paper is to predict 30day hospital readmission after kidney transplant, which we treat it as a long document classification problem without using specific domain knowledge. the data we use is the unstructured clinical documents of each patient up to the date of discharge. in particular, we face three types of challenges in this task. first, the document size can be very long; documents associated with these patients can have tens of thousands of tokens. second, the dataset is relatively small with fewer than 2,000 patients available, as kidney transplant is a non-trivial medical surgery. third, the documents are noisy, and there are many target-irrelevant sentences and tabular data in various text forms (section 2). the lengthy documents together with the small dataset impose a great challenge on representation learning. in this work, we experiment four types of encoders: bag-of-words (bow), averaged word embedding, and two deep learning-based encoders that are clinicalbert @xcite and lstm with weight-dropped regularization @xcite . to overcome the long sequence issue, documents are split into multiple segments for both clinicalbert and lstm (section 4). after we observe the best performed encoders, we further propose to combine reinforcement learning (rl) to automatically extract out task-specific noisy text from the long documents, as we observe that many text segments do not contain predictive information such that removing these noise can potentially improve the performance. we model the noise extraction process as a sequential decision problem, which also aligns with the fact that clinical documents are received in time-sequential order. at each step, a policy network with strong entropy regularization @xcite decides whether to prune the current segment given the context, and the reward comes from a downstream classifier after all decisions have been made (section 5). empirical results show that the best performed encoder is bow, and deep learning approaches suffer from severe overfitting under huge feature space in contrast of the limited training data. rl is experimented on this bow encoder, and able to improve upon baseline while pruning out around 25% text segments (section 6). further analysis shows that rl is able to identify traditional noisy tokens with few document frequencies (df), as well as task-irrelevant tokens with high df but of little information (section 7). ## data this work is based on the emory kidney transplant dataset (ektd) that contains structured chart data as well as unstructured clinical notes associated with 2,060 patients. the structured data comprises 80 features that are lab results before the discharge as well as the binary labels of whether each patient is readmitted within 30 days after kidney transplant or not where 30.7% patients are labeled as positive. the unstructured data includes 8 types of notes such that all patients have zero to many documents for each note type. it is possible to develop a more accurate prediction model by co-training the structured and unstructured data; however, this work focuses on investigating the potentials of unstructured data only, which is more challenging. ## reinforcement learning reinforcement learning is applied to the best performing encoder in section 4 to prune noisy text, which can lead to comparable or even better performance, as many text segments in these clinical notes are found to be irrelevant to this task. figure 1 describes the overview of our reinforcement learning approach. the pruning process is modeled as a sequential decision problem, for the fact that these notes are received in time-order. it consists of two separate components: a policy network, and a downstream classifier. to avoid having too many time steps, the policy is performed on the segment level instead of token level. for each patient, documents are split into short segments g 1:@xmath0, g 2 , • • • , g t }, and the policy network conducts a sequence of decisions a 1:@xmath1, a 2 , • • • , a t } over segments. the downstream classifier is re-sponsible for the reward, and the reinforce algorithm is used to train the policy @xcite . state at each time step, the state s t is the concatenation of two parts: the representation of previously selected text, and the current segment representation g i . the previously selected text serves as the context and provides a prior importance. both parts are represented by an effective encoder, e.g. the best performing encoder from section 4. ## experiments before experiments, we perform the preprocessing described in section 2.1, and then randomly split patients in every note type by 5 folds to perform cross-validation as suggested by @xcite . to evaluate each fold f i , 12.5% of the training set, that is the combined data of the other 4 folds, are held out as the development set and the best configuration from this development set is used to decode f i . the same split is used across all experiments for fair comparison. following @xcite , the averaged area under the curve (auc) across these 5 folds is used as the evaluation metric. ## noise analysis to investigate the noise extracted by rl, we analyze the pruned segments on the validation sets of the consultations type (co), and compare the results with other basic noise removal techniques. qualitative analysis the social worker met with this pleasant year old caucasian male on this date for kidney transplant evaluation . the patient was alert , oriented and easily engaged in conversation with the social worker today . he resides in atlanta with his spouse of years , who he describes as very supportive . he reports occasional alcohol drinks per month but denies any illicit drug use . he has a grade education . he has been married for years . he is working full -time while on peritoneal dialysis as a business asset manager . he has medicare and an aarp prescriptions supplement . family history : mother deceased at age with complications of obesity , high blood pressure and heart disease . in addition, we also notice that the policy is able to remove certain duplicate segments. medical professionals sometimes repeat certain description from previous notes to a new document, causing duplicate content. the policy learns to make use of the already selected context, and assigns negative coefficients to certain tokens. duplicate segments are only selected once if the segment contains many tokens that have opposite feature importance in the context and segment vectors. ## conclusion in this paper, we address the task of 30-day readmission prediction after kidney transplant, and propose to improve the performance by applying reinforcement learning with noise extraction capability. to overcome the challenge of long document representation with a small dataset, four different encoders are experimented. empirical results show that bagof-words is the most suitable encoder, surpassing overfitted deep learning models, and reinforcement learning is able to improve the performance, while being able to identify both traditional noisy tokens that appear in few documents, and task-specific noisy text that commonly appear.