P \text{M}^2\text{F}^2 N : Patient Multi-view Multi-modal Feature Fusion Networks for Clinical Outcome Prediction (2022)

Abstract: Clinical outcome prediction is critical to the condition prediction of patients and management of hospital capacities. There are two kinds of medical data, including time series signals recorded by various devices and clinical notes in electronic health records (EHR), which are used for two common prediction targets: mortality and length of stay. Traditional methods focused on utilizing time series data but ignored clinical notes. With the development of deep learning, natural language processing (NLP) and multi-modal learning methods are exploited to jointly model the time series and clinical notes with different modals. However, the existing methods failed to fuse the multi-modal features of patients from different views. Therefore, we propose the patient multi-view multi-modal feature fusion networks for clinical outcome prediction. Firstly, from patient inner view, we propose to utilize the co-attention module to enhance the fine-grained feature interaction between time series and clinical notes from each patient. Secondly, the patient outer view is the correlation between patients, which can be reflected by the structural knowledge in clinical notes. We exploit the structural information extracted from clinical notes to construct the patient correlation graph, and fuse patients’ multi-modal features by graph neural networks (GNN). The experimental results on MIMIC-III benchmark demonstrate the superiority of our method.

## introduction with the development of information technology in medical area, an increasing number of devices are used for monitoring patients. and a large number of data is stored as electronic health records (ehr), which contain numerical results of physical examination in time series and clinical notes in text for patients' relevant information. the multi-type data can be utilized to predict the condition of patients, which can help in managing the resources in hospitals. most previous works focused on modeling the problem using the time series data recorded by medical instruments @xcite . however, the time series data gathered from medical devices only reflects physical status of patients in a one-sided way. medical professionals need to utilize their expertise to analyze patients' data and make the diagnosis. the important analyses to patients' data are recorded in ehr as clinical notes. more recent work applied natural language processing (nlp) methods to take full advantage of medical information in clinical notes for prediction tasks @xcite @xcite . they utilized pre-trained language models to extract text features of clinical notes and fed them into recurrent or convolution neural networks for clinical outcome prediction. further more, considering to combine time series data with clinical notes for improved prediction on clinical outcome, some recent work proposed multi-modal learning methods to jointly model the two kinds of data @xcite @xcite . they used sequence models to extract features of time series and clinical notes respectively, and concatenated them for predicting clinical outcome. however, the existing methods do not consider that the features of time series data and clinical notes fuse different parts of each other with various weights. besides, the multi-modal features of a single patient is not sufficient for clinical outcome prediction, and the medical correlation between patients has not been exploited for this task. to overcome the above disadvantages of the existing methods, we propose the patient multi-view multi-modal feature fusion networks (pm 2 f 2 n) 1 for clinical outcome prediction. the model enhances the multi-modal feature fusion ability in two views. firstly, from the patient inner view, we use the co-attention @xcite module to enhance the fine-grained feature interaction between time series data and clinical notes. the co-attention module allows our model to attend to important parts of time series data as well as correlated medical information of clinical notes. secondly, from the patient outer view, other patients' information is useful to predict the status of the observed one. we construct the patient correlation graph based on the structural medical information extracted from clinical notes, and fuse patients' multi-modal features by graph neural networks (gnn). with the multi-modal feature fusion from different views, our model can gain better generalization ability to predict clinical outcome. the contributions of this manuscript can be summarized as follows: ## model we introduce the notations about clinical outcome prediction before getting into the details of the proposed model. the training set with n s samples is denoted as {(x (i) , c (i) , y (i) )} ns @xmath0, where x (i) and c (i) are i-th patient's time series data and clinical note respectively, and y (i) is the task-defined label. given a time series with n t time steps and n v observed variables, the patient's vital signals can be formulated as @xmath1, x 2 , . . . , x nt } ∈ r nt×nv . we denote the clinical note with n w words as @xmath2, c 2 , . . . , c nw }. after pre-processing the multi-modal data, we feed them into the proposed model. the pm 2 f 2 n model is shown in figure 2 . for time series data, we utilize the bidirectional gru to extract hidden representations. considering to acquire the multi-grained features of clinical notes, we apply the ner model to extract medical entities as local features and use the term frequency-inverse document frequency (tf-idf) method to extract global features of clinical notes. to combine the entity representations of clinical notes with hidden representations of time series data in a fine-grained way, we exploit the co-attention module to acquire the multi-modal fusion features with various attention weights. based on the medical information of different patients, we build the patient correlation graph and exploit it to aggregate multi-modal features of various neighbors via gnn. the concatenation of global features of clinical notes, last hidden features of time series data and aggregation multi-modal features is fed into ffnn for outcome prediction. ## conclusion in this paper, we analyze the disadvantages of existing multi-modal learning methods for clinical outcome prediction. to enhance the multi-modal feature in different views, we propose the patient multi-view multi-modal feature fusion networks (pm 2 f 2 n) for the task. from the inner view, we extend the co-attention module to fuse the features of the time series data and structural medical knowledge in a fine-grained way. from the outer view, we exploit the correlation between patients to aggregate the multi-modal features between the similar patients. with the multi-view multi-modal feature fusion strategy, the proposed model can learn the general patient representations for clinical outcome prediction. compared with the existing methods, our model can gain the best results on benchmark dataset mimic-iii. and the further discussion including ablation study, effect of pcg and visualization analysis, verifies the effectiveness of the proposed strategy. considering the heterogeneity of patients, we will try to adapt the heterogeneous graphs for modeling the correlation between patients in the future.