Performance and utility trade-off in interpretable sleep staging
Abstract
Recent advances in deep learning have led to the development of models approaching the human level of accuracy.
However, healthcare remains an area lacking in widespread adoption.
The safety-critical nature of healthcare results in a natural reticence to put these black-box deep learning models into practice.
This paper explores interpretable methods for a clinical decision support system called sleep staging, an essential step in diagnosing sleep disorders. Clinical sleep staging is an arduous process requiring manual annotation for each 30s of sleep using physiological signals such as electroencephalogram (EEG).
Recent work has shown that sleep staging using simple models and an exhaustive set of features can perform nearly as well as deep learning approaches but only for some specific datasets.
Moreover, the utility of those features from a clinical standpoint is ambiguous.
On the other hand, the proposed framework, NormIntSleep demonstrates exceptional performance across different datasets by representing deep learning embeddings using normalized features.
NormIntSleep performs 4.5% better than the exhaustive feature-based approach and 1.5% better than other representation learning approaches.
An empirical comparison between the utility of the interpretations of these models highlights the improved alignment with clinical expectations when performance is traded-off slightly.
NormIntSleep paired with a clinically meaningful set of features can best balance this trade-off by providing reliable, clinically relevant interpretation with robust performance.
Introduction
There has been a steady accumulation of digital records of patient health data due to the increasingly widespread adoption of electronic health records [1, 2, 3].
Breakthroughs in deep learning have leveraged this influx of clinical data to create increasingly complex and capable systems [4, 5, 6].
However, the lack of interpretability and explainability of deep learning models prevents most of them from being used in practice because clinicians need to understand the reasoning behind each classification to avoid noise and bias [7, 8, 9, 10, 11].
Although challenging to design due to the manual effort required, linear models paired with a robust set of features can provide a degree of interpretability [12].
However, linear models paired with features too complex may result in models whose interpretation does not have clinical relevance.
In this paper, using sleep staging as a case study, we take a deeper dive into the trade-off between the clinical relevance of explanations and performance and propose a model attempting to provide the ideal balance.
Around 70 million US adults are affected by sleep disorders [13, 14] such as insomnia, narcolepsy, or sleep apnea.
The most crucial step for the diagnosis of sleep disorders is sleep staging [15].
The gold standard for sleep staging remains the manual annotation of Polysomnogram (PSG) by [Table] clinicians [16].
During this process, clinicians inspect the PSG signals from multiple channels and annotate each 30s segment with one of five sleep stages, i.e., wake, rapid eye movement (REM), and the non-REM stages N1, N2, and N3, by following guidelines stated in the American Academy of Sleep Medicine (AASM) Manual for the Scoring of Sleep and Associated Events [17].
This manual annotation scheme is time-consuming and expensive because a clinician needs several hours to annotate a patient’s recordings from a single night [18].
There has been considerable research in automating sleep staging to overcome this problem.
These approaches have primarily remained in the realm of deep learning that lack interpretability [19], for example, convolutional neural networks (CNN) [20, 21, 22, 23], recurrent neural networks (RNN) [24, 25], recurrent convolutional neural networks (RCNN) [26, 27], graph convolutional networks (GCN) [28, 29], and attention [30, 31, 28].
On the other hand, the AASM sleep scoring manual guidelines [17] are interpretable for clinicians but need more explicit definitions to design a robust computational model [32].
A recent study proposed a feature-based linear model [12] that performed as well as deep neural networks.
However, the features used in this study were not designed considering clinical guidelines.
To generate clinically relevant explanations, we propose NormIntSleep , a representation learning framework that projects deep neural network embeddings into an interpretable normalized feature space.
Thus, by choosing an appropriate feature space, NormIntSleep can unite clinically relevant explanations with the high accuracy of a deep learning model.
Data
Two publicly available datasets are used for evaluation.
These datasets are summarized in Table 1. In
PhysioNet-EDFX [33, 34, 35], sleep stages N3 and N4 annotated using the R&K schema [36, 37] were combined into a single N3 class to align with the AASM standards [17, 38, 39] used in the
ISRUC dataset [40].
After this alignment procedure, both datasets provide a sleep stage annotation from wake (W), rapid eye movement (REM), and the non-REM stages (N1, N2, N3) to each 30second epoch.
Interpretable sleep staging aims to predict these sleep stages using multi-channel physiological signals while providing clinically meaningful interpretations for each classified sleep stage.
NormIntSleep uses the PSG recordings, X , to generate an interpretable representation for deep neural network embeddings using the following steps:
A CNN-LSTM network [42] is trained end-to-end on sleep staging using the multi-channel EEG, EOG, and EMG signals as input.
The CNN is composed of 3 convolutional layers where each layer is followed by batch normalization, ReLU activation, and max pooling.
The kernel sizes of the three layers are 201, 11, and 11, and the output channels are 256, 128, and 64.
The CNN output is used as input for a layer of bi-directional Long Short-Term Memory (LSTM) cells with 256 hidden states.
The resulting 512 hidden states represent the embedding space , E .
During model training using cross-entropy loss, the LSTM output, E ( X ) is connected to a fully-connected layer with 5 outputs for the 5 sleep stages.
Features F ( X ) , defined in the feature space F , are extracted from the dataset.
There are two possible sets of features: • FeatLong : an exhaustive list of features inspired by the recent work of Van Der Donckt et al.
The features are not designed considering clinical guidelines in the AASM
Manual for the Scoring of Sleep and Associated Events [17].
It results in 2488 features for the ISRUC dataset and 1048 features for the Physionet dataset.
10% of the most significant features are retained for the next steps using ANOVA, resulting infeatures for the ISRUC dataset and 105 for the Physionet dataset. • FeatShort : a smaller set of clinically interpretable features inspired by the recent work of Al-Hussaini et al.
The features are designed according to the AASM manual [17].
87 features are extracted for the ISRUC dataset and 38 for the Physionet dataset.
90% of the most significant features are retained for the next steps using ANOVA, resulting in 78 features for the ISRUC dataset and 34 for the Physionet dataset.
A linear transformation, T , is learned from the embedding space to the feature space, E T −→ F , using linear least squares regression with L 2 regularization.
R = E ( X ) · T defines the interpretable representations of embeddings after projecting the embedding, E ( X ) , to the interpretable feature space, F .
These representations of the embeddings, R , are normalized before being used as input to classifiers.
[Equation]
where µ R = (cid:80) RN and σ R = (cid:113) 1 N (cid:80) Ni =
( R i − µ R ) 2 .
These normalized interpretable representations of the embeddings, R ′ , are used as inputs to simple classifiers such as logistic regression and decision tree
Experiments 4.1 Experimental setup
The CNN-LSTM network was trained using PyTorch 1.0 [43] and a batch size of 1000 samples from 1 PSG.
The training was continued for 20 epochs with a learning rate of 10 − 4 using ADAM [44] as the optimization method.
The simple models were trained using scikit-learn [45, 46], XGBoost [47], and CatBoost [48, 49].
The features were extracted using scikit-learn [45, 46], yasa [50], and tsflex [51].
The data is randomly split by subjects into a training and a test set in a 9:1 ratio using the same seed for all experiments.
The training set is used for each dataset to fix model parameters, and the test set is used to obtain performance metrics.
The same model hyperparameters and feature extraction schema are used to prevent overfitting and ensure consistent performance across different datasets.
4.2 Baselines
• CNN-LSTM: the model used to generate the embeddings • FeatLong and FeatShort: defined in Section 3.
They are used as inputs to simple classifiersCatBoost [48, 49], XGBoost [47], Logistic Regression, and Gradient Boosted Trees • SLEEPER [32]: a prototype based interpretable sleep staging algorithm • SERF [42]: an interpretable sleep staging algorithm based on embeddings, rules, and features • U-Time [21]: state-of-the-art deep learning model for sleep staging
4.3 Results
Accuracy, Macro F1-Score, and Cohen’s κ are used to evaluate the models.
The results in Tablewere produced in the same experimental setup expanded upon in Section 4.1.
Since the performance of models varies a lot between the two datasets, an aggregated metric, Average Performance , is calculated based on the average value of each model across the two datasets and three metrics.
The best interpretable methods using FeatShort and FeatLong are underlined, and the best overall approach is highlighted in bold.
The results show that NormIntSleep surpasses all other interpretable methods, even when it uses the smaller set of interpretable features, FeatShort.
The benefits of using this over the exhaustive set of features, FeatLong, are discussed in Section 4.4. [Table]
4.4 Interpretation and Clinical Relevance
Figure 1 highlights the top 10 important dimensions of the interpretable representation of the
NormIntSleep embeddings, R ′ , according to SHAP values and their influence on the classification of each of the 5 sleep stages.
N1 is a challenging sleep stage to classify
[Table]
with around 50% agreement among human annotators, so its classification is not discussed in detail.
The differences between Figure 1a and Figure 1b show the importance of having clinically relevant representations.
A deeper analysis is performed on the 3 most critical interpretable representations of embeddings in NormIntSleep -XGBoost-FeatShort (Figure 1a): Beta waves [42] have frequencies between 8 Hz and 20 Hz.
This frequency range is present in Wake [52], REM [52], and N2 (through the overlapping sigma band in Spindles [53]).
Consequently, it is the perfect attribute to differentiate N3 from the other stages and aligns with the high SHAP value assigned to N3.
Slow waves are essential for restorative sleep [54].
As a result, slow waves play a critical role in maintaining sleep and daytime function in patients with insomnia or nonrestorative sleep [54]. During sleep staging, these slow waves are used by clinicians to annotate N [52, 17]. Thus, clinical evidence supports the importance of slow waves in N3 classification by NormIntSleep -XGBoost-FeatShort. 3. EOG has been used to detect a person’s wakefulness [55] and movement of the eyes [56, 57].
Torsvall et al. [58] show that Delta activity in EOG is highest in the wake stage before sleep induction. Thus, the high SHAP value attributed to Delta waves in ROC-A1 (an EOG channel) in classifying wake using NormIntSleep -XGBoost-FeatShort aligns with clinical expectations. The alignments with clinical guidelines highlight the utility of the explanations provided by
NormIntSleep -XGBoost-FeatShort. On the other hand, 6 of the top 10 representations in NormIntSleep -XGBoost-FeatLong (Figure 1b) are not defined using clinical guidelines such as the AASM manual for the scoring of sleep [17], and the rest are derived from the same frequency band, delta.
Thus, the clinical relevance of NormIntSleep -XGBoost-FeatLong’s interpretations is ineffective.
The sleep staging interpretations such as that provided by NormIntSleep -XGBoost-FeatShort can give physicians confidence in the automated labels and facilitate validation, thus paving the way for faster adoption.
The feature space of the two datasets using FeatLong and FeatShort are shown in Appendix A. It highlights the efficacy of FeatLong in segregating the Physionet dataset into the 5 classes through the well-partitioned clusters for each sleep stage.
The importance of features using XGBoost paired with FeatShort and FeatLong are shown in Appendix B. [Figure]
Conclusion
Interpretability is crucial for the adoption of clinical decision support systems.
Complex features paired with a simple model can provide interpretation.
However, those explanations can only be helpful if the features are clinically meaningful.
NormIntSleep offers a generalizable framework to leverage the most clinically significant features for classification with accuracy higher than complex feature-based models and similar to black-box deep learning.
As a result, the interpretations are clinically relevant.
Thus NormIntSleep takes forward strides towards adoption. Acknowledgment
This research was funded by the National Science Foundation CAREER grant 1944247 to C.M, the
National Institute of Health grant U19-AG056169 sub-award to C.M., and the McCamish Parkinson’s
Disease Innovation Program at Georgia Institute of Technology and Emory University to C.M.
Appendix A: Feature Space Visualization
[Figure]
[Figure]
B Appendix B. SHAP Feature Importance
[Figure]