Paper:
MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
Deep learning models exhibit state-of-the-art performance for many predictive healthcare tasks using electronic health records (EHR) data, but these models typically require training data volume that exceeds the capacity of most healthcare systems. External resources such as medical ontologies are used to bridge the data volume constraint, but this approach is often not directly applicable or useful because of inconsistencies with terminology. To solve the data insufficiency challenge, we leverage the inherent multilevel structure of EHR data and, in particular, the encoded relationships among medical codes. We propose Multilevel Medical Embedding (MiME) which learns the multilevel embedding of EHR data while jointly performing auxiliary prediction tasks that rely on this inherent EHR structure without the need for external labels. We conducted two prediction tasks, heart failure prediction and sequential disease prediction, where MiME outperformed baseline methods in diverse evaluation settings. In particular, MiME consistently outperformed all baselines when predicting heart failure on datasets of different volumes, especially demonstrating the greatest performance improvement (15% relative gain in PR-AUC over the best baseline) on the smallest dataset, demonstrating its ability to effectively model the multilevel structure of EHR data.

Review:
The authors present a novel methodology designed to take into account some of the structure in the way EHR data is emitted. They observe that a visit consists of a set of diagnosis, which in turn trigger a series of medications and treatments. They leverage this hierarchy to construct multi-level embeddings that allows them to aggregate information across different levels to create embeddings. Overall, I like the idea of using “hidden” structure to improve a model and I like the idea of using the auxiliary loss function.   The authors evaluate all methods to predict heart failure under a variety of scenarios. They mention a sequential disease prediction task in the abstract, but I was unable to find an evaluation for this in the main text, though it is mentioned in the supplement. Without commenting on the results for this task in the main the text, I would recommend removing the reference to it.  On heart failure, they show that their method improves up Med2Vec and GRAM, two existing methods in this area, especially when data are scarce (Figure 3) and when the visit complexity is high (Table 3).  Quality: Over all the paper appears to be sound, though some of the details are missing. I wasn’t clear from the main text or the supplement which version of MiME was the one reported in the main body of the text. Was it the one which included the MLP? If so, as the authors note, this gives MiME and GRAM an advantage of Med2Vec and the fair comparison would add an MLP to Med2Vec. Also, the performance advantage for MiME is less clear when considering the auROC, even though I mostly agree with the authors use of AUC-PR for this purpose.   I also had a question with respect to the data source. The authors state: "We conducted all our experiments using EHR data provided by a large healthcare system" More details about the population in this EHR would help the reader assess the generalizability of this data. Is it US? European? What is the socio-economic status of the population? Ethnicity? etc  I also know that healthcare data can be hard to come by, but 30,000 patients is not large by any standard. Since most methods performed the same when using all of the data, I am unsure if the results seen by the authors would hold when using an EHR of millions of patients, which is common for most major healthcare centers and insurance claims databases.  Clarity: The manuscript is mostly well written, though it is a little hard to follow in spots, mainly in section 2, though I found figure 2 to be very helpful.   Small typos: Page 2: In a typical visit,a physician gives a 59 diagnosis to a patient and then order medications or procedures based on the diagnosis.  Originality: The method presented here is novel and interesting.   Significance: The significance of this work could be improved by using addition datasets (which I know in healthcare is tough) and by considering more conditions. As written, it is of limited significance given the small, elderly population and the single prediction task.