Incorporating Risk Factor Embeddings in Pre-trained Transformers Improves Sentiment Prediction in Psychiatric Discharge Summaries (2020)

Abstract: Reducing rates of early hospital readmission has been recognized and identified as a key to improve quality of care and reduce costs. There are a number of risk factors that have been hypothesized to be important for understanding re-admission risk, including such factors as problems with substance abuse, ability to maintain work, relations with family. In this work, we develop Roberta-based models to predict the sentiment of sentences describing readmission risk factors in discharge summaries of patients with psychosis. We improve substantially on previous results by a scheme that shares information across risk factors while also allowing the model to learn risk factor-specific information.

## introduction about 1 in 5 medicare patients discharged from the hospital is rehospitalized within 30 days @xcite . four out of the top ten conditions with the largest number of readmissions among the medicaid enrollees were mental health conditions or substance use disorders @xcite . readmissions are harmful both in being disruptive to patients and families, and as a major driver of health-care costs in psychiatry @xcite . also, premature discharge of patients contributes not only to rehospitalization but to increased risk of homelessness and the possibility of violent behavior or suicide. thus, reducing rates of early hospital readmission has been recognized and identified as a key to improve quality of care and reduce costs. there are a number of risk factors that previous work compiled from the literature @xcite to be important for understanding readmission risk, including such factors as problems with substance abuse, ability to maintain work, relations with family. studying readmission through the use of explicit risk factors may go against prevailing trends in machine learning (black box models that take in raw signals), but has at least two potential benefits: 1) allowing for descriptive study of, and thus better understanding of, the factors that are important for readmission; and 2) the possibility that a risk classifier that uses explicit risk factors will be more interpretable and trustworthy to providers, and thus more likely to be put into practical use. in this work, we make use of a publicly available data set of sentences from discharge summary that is annotated for seven risk factors, along with the "sentiment," marked as positive, negative, or neutral (more details in background). previous work on this dataset showed that the task was approachable with neural methods, but performance suffered because of small dataset size @xcite . here, we address this issue with two advances. first, we show that transfer learning helps dramatically. while the previous work used smaller neural models trained from scratch, we start from pre-trained transformer models (roberta @xcite ), and improve them for the task with architectural and data augmentation strategies. second, we show that sharing data between the different risk factor domains is better for performance than training separately, at least with pretrained models. previous work trained separate classifiers for each of the risk factor domains, due to the (legitimate) belief that there are significant differences in predicting sentiment in different risk factor domains @xcite . here, we demonstrate that a method that allows for sharing of information between different risk factor domains can improve performance. because of a mismatch in the way training and test data were annotated, we introduce a data augmentation method for the training set that allows our method to improve over the vanilla roberta baseline. the improvements from this latter technique point in a direction that could allow for further improvements even without additional training data. ## background the dataset we use was created in previous work @xcite and made publicly available. it consists of sentences extracted from discharge summaries of patients with psychosis at boston-area hospitals. previous work had examined two tasks on this dataset, the identification of the risk domain represented by the sentence (appearance, mood, interpersonal, substance use, occupation, thought content and thought process), and the sentiment of the sentence given the risk domain. in the training data, each sentence has only one risk factor, and thus one sentiment, but the test data allows for sentences to have multiple risk factors, and thus multiple different sentiments for a single sentence, conditioned on the risk factor domain. both the previous work for risk factor classification @xcite and sentiment classification @xcite used multilayer neural networks in their experiments and found them to be the best-performing. despite the promise of the above work, the tasks are still unsolved. recent work in contextualized embeddings has shown great success for sentence classification tasks. specifically, bert-based models @xcite , based on the transformer architecture @xcite , showed that pretraining deep transformer encoders on massive text datasets with a language modeling objective could lead to improvements in a variety of tasks. the best performance is typically obtained by fine tuning, where a classifier head is attached to a special sentence token, and new tasks are learned via standard supervised learning, in which the weights of the classifier head are trained from scratch while the weights of the transformer encoder are allowed to update. in this work, we make use of the roberta updates to bert @xcite , which use the same architecture but pre-trained on a larger dataset and for a longer time. ## data the training dataset contains 3500 sentence-length texts, 500 from each of the seven readmission risk factors mentioned above. the test dataset contains 1650 texts which can involve multiple risk factors and are more variable in length compared with the training data, as described in the previous study @xcite . we divided the training instances into training and development set with an 80%/20% split, foot_0 leaving us with 2800 training instances and 700 development instances. since we are focusing on sentiment prediction in this work, we take the risk factor domains as given, and for test sentences with multiple risk factor domains, we create multiple instances where the input pairs the sentence text with each domain, and the target is the gold sentiment label for that domain. this results in 2103 test instances. 750 of these are labeled as the other domain, which does not have training instances nor reported results, so we discard these, leaving 1353 test instances. ## methods we developed several variations of a risk factor sentiment classifier based on the roberta architecture. ## evaluation we evaluated the four different architectures on this task to explore the importance of sharing information between different risk factor domains as well as learning the domain specific information. we first evaluate at the instance level, computing precision and recall for each sentiment label, and combining to get an f1 score for each sentiment label, as well as overall accuracy (proportion of instances correctly predicted regardless of sentiment or risk factor domain). the improvement obtained by fine-tuning one single roberta model instead of seven independent models suggests that the benefits of sharing information between risk factor domains outweigh the potential risks that the model will learn conflicting information. however, simply using domain as the second sentence during roberta fine-tuning does not lead to improvement in the overall model performance. augmenting the training data to look more like the test data was necessary in order for the domain embedding input to show benefits. we tested the significance of the improvements in accuracy and macro-f1 between roberta and roberta+d+aug by fine-tuning with 20 randomly selected seeds, and performing a one-tail t-test, and results were found to be significant (@xmath0.05, @xmath1.005 for accuracy and macro-f1). ## discussion and conclusion we selected instances from the test set where the system trained on augmented data (roberta+d+aug) did well, and others where it did not (see overall, our new approach shows major gains in performance over the existing state of the art for this problem. the biggest gains come from simply using large pre-trained models. however, the modified architecture and data augmentation technique lead to further gains, and have the ability to separate out multiple sentiments for a single sentence on new data. future work may see larger benefit with methods for creating augmented training data that create more natural-looking sentence pairs. the source code used to fine-tune the model will be made publicly available 2 . and jyoti aggarwal. 2005. the economic burden of schizophrenia in the united states in 2002. the journal of clinical psychiatry, 66(9):1122-1129. place: united states.