Towards a holistic assessment of health data representations under realistic dataset shifts
Anonymous Author(s)
AffiliationAddress email
Abstract
The recent availability of large datasets in bio-medicine has inspired the development of representation learning methods for multiple healthcare applications.
Despite advances in predictive performance, the clinical utility of such methodsis limited when exposed to real-world data.
This study attempts to address thisgap through development of model diagnostic measures to detect potential pitfallsduring deployment without assuming access to external data.
Specifically, we focuson modeling realistic data shifts in electrophysiological signals (EEGs) via datatransforms, and extend the conventional task-based evaluations with analyses ofa) model’s latent space and b) predictive uncertainty, under these transforms.
Weconduct experiments on multiple EEG feature encoders and two downstream tasksusing publicly available large-scale clinical EEGs.
Our results suggest that measures of latent space integrity and model uncertainty under the proposed realisticdata shifts may help anticipate performance degradation during deployment.
Introduction
In recent years, the availability of large datasets in bio-medicine has inspired the development of deeprepresentation learning methods for diagnostic applications in healthcare domains such as cardiology,neurology and radiology, to name a few.
However, there are increasing examples of failures ofsuch systems during real-world deployment, predominantly due to dataset/distribution shifts in thedeployment setting (1; 2; 3; 4; 5; 6).
Currently, the most common approach undertaken to evaluatehealthcare machine learning (ML) models is analyzing task performance in a held-out test dataset.
The recent advances in covariate-shift detection and adaptation approaches (7; 8; 9; 10; 11) havenot been widely adopted in the healthcare ML community because they do not reflect the uniquehealth-data-related challenges.
As such, the practical value of many healthcare ML models remainsunclear at present without additional assessments of model resilience in deployment settings (12).
This gap reflects a major hurdle in translating healthcare ML models as they are required to performconsistently under varying noise characteristics, diverse data acquisition protocols and multiple sitesand populations, during deployment.
Hence, there is a critical need to evaluate the robustness ofhealthcare ML models under realistic dataset shifts prior to deployment.
In this paper, we focus on a particular area of healthcare ML pertaining to learning representations ofphysiological signals such as electroencephalograms (EEGs).
EEGs are commonly used to diagnosevarious neurological, psychiatric, and sleep disorders, as well as in applications involving brainmachine interfaces.
Deep representation learning of raw EEG signals has recently gained popularitybecause of the availability of large-scale EEG datasets (13) and has shown promise in improving thelabor-intensive and error-prone manual process undertaken in clinical EEG reviews.
A variety ofapproaches have been proposed including, fully-supervised deep feature encoders (14; 15; 16), selfsupervised encoders (17; 18; 19), and traditional feature-based encoders using power spectral features(20).
However, many of those approaches struggle to generalize to previously unseen real-world data,i.e., data acquired from different populations, institutions, or devices (21; 22).
We believe that a multi-pronged approach is necessary to holistically evaluate the robustness ofhealthcare ML models prior to deployment.
First, models of realistic data shifts are needed to assessrobustness during model development itself.
For instance, realistic EEG data shifts may includedifferences in behavioral states during EEG acquisition (awake or asleep), the number of EEG sensorsused, the EEG reference used, noise during acquisition, analog-to-digital conversion, hardware filtersettings, and impedance resulting from electrodes.
Second, we argue that robustness evaluationsmay also benefit from analyzing the properties of the latent embedding space (7).
Specifically,quantifying the topological changes in the latent space under perturbations can, 1) provide early cluesinto test set performance when ground truth labels are not available; and 2) indicate failure modesof model interpretation techniques that rely on analyzing similarity properties in the latent space.
Third, we highlight the need for estimating the uncertainty in model predictions at the output layer(23).
Uncertainty modeling is critical in ensuring the safe and reliable use of healthcare models sinceaccurate communication of uncertainty allows human experts to selectively ignore the model.
In this study, we evaluated the robustness of several previously proposed EEG feature encoders usingthe above multi-pronged approach.
Specifically, we introduced four EEG signal transformations thatmodel the effects of instrumentation-related real-world EEG variability, namely: hardware band-passfilters, quantization precision, narrow-band impedance noise, and broad-band Gaussian noise.
Next,we developed a topological measure of latent space integrity based on Delaunay neighborhood graphs(adapted from (24)) and a Monte Carlo dropout-based approach to estimate predictive uncertainty.
Inthis setting, we then evaluated fully-supervised, self-supervised, and traditional feature-based EEGencoders in both classification and regression tasks.
Our major contributions are the following:
We developed four domain-guided EEG data shifts (EEG-DS) that reflect the real-world
variability observable during deployment.
Using EEG-DS, we devised a multi-pronged approach to evaluate the robustness of MLmodels including assessments of
latent space integrity and predictive uncertainty Through evaluation of multiple
EEG feature encoders
using
large -scale TUH- EEG data we show that our approach can help anticipate potential pitfalls in encoder performanceduring deployment and that popular EEG encoders have variable robustness profiles.
Related Work
Generalization to unseen data : The importance of generalization in healthcare ML is widelyacknowledged (1; 5; 6; 25).
Prior work in general ML has proposed, a) methods to detect covariateshift or out-of-distribution samples (7; 26; 27); and b) novel learning approaches that generalize wellor adapt to new domains (28; 29; 30; 8; 9; 10).
However, the majority of the adaptive approachesrequire examples from target distributions and therefore have limited utility.
Some methods to detectcovariate-shift focus on specific properties such as non-stationarity (26; 27), which are simplisticand do not reflect the complexity of healthcare data and shifts.
In this paper, we focus on detectingfailure modes prior to deployment via topological analysis of the latent space and evaluation ofmodel uncertainty under realistic data shifts.
In this context, the closest to our work is (7), whereauthors used measures such as Wasserstein’s distance for detecting data shifts in the latent spacefor vision and language tasks.
Here we develop a more comprehensive topological evaluation ofthe latent space using Delaunay neighborhood graphs and use it to analyze the robustness of scalp
EEG representations.
In addition, some recent studies have highlighted the benefits of uncertaintyquantification in out-of-distribution settings, in knowing when not to trust a model’s predictions (31).
Some studies have also indicated that certain types of representation learning approaches exhibitbetter robustness in terms of predictive uncertainty than others (32).
Building on these insights, weutilized a Monte Carlo dropout (MCD)-based uncertainty quantification to analyze the predictiveuncertainty of different EEG feature encoders under specific data shifts.
Data shifts in EEG data: Several previous studies have analyzed the performance of EEG MLmodels using datasets from multiple institutions (21; 22; 33).
Apart from reporting differences inmodel performance, these studies did not investigate approaches to identify failure modes prior todeployment, model data shifts, or generate insights for future studies.
Some studies have developeddata augmentations for EEG signals to improve the diversity of training data and have demonstratedsome success in achieving better task performance (18; 34; 35).
While augmentations such asamplitude scaling, time shift, DC shift, and band-stop filtering are useful to increase training data diversity (18), they do not meet the complexity of real-world EEG signal variability observed indeployment settings.
Hence, in this paper, we focus on developing domain-guided data shiftsobservable in deployment settings as opposed to simpler training time data augmentations.
EEG feature encoders: A variety of EEG feature encoders have been proposed including, fullysupervised deep encoders (14; 15; 16), self-supervised encoders (17; 18; 19), and traditional domainderived feature encoders using power spectral features (20; 36).
While the generalization capabilitiesof the majority of these encoders are unclear, an encoder derived based on self-supervised learninghas shown promising trends in generalization when specific domain-derived self-supervision taskswere utilized (17).
However, a comprehensive robustness evaluation of the different types of EEGfeature encoders under realistic EEG data shifts is currently lacking.
Background & Methods
In this section, we describe the formulations of multichannel EEG data, feature encoders, and theproposed robustness evaluation approach including realistic EEG data shifts (EEG-DS), latent spaceanalysis, and uncertainty quantification.
The proposed approach does not require additional data orlabels obtained in deployment settings and therefore is applicable during model development.
3.1 Learning EEG data representations
We use x i ∈ R M × N to denote the EEG data of a single epoch recorded using M sensors with N samples (i.e., N = time duration × sampling frequency).
Typically, a full EEG recording consists of multiple such non-overlapping epochs.
Consider a feature encoder f θ : R M × N → R d that maps a multi-channel EEG epoch
x i into a d -dimensional latent representation z i ∈ R d , where z i =
f θ ( x i ) .
In traditional supervised learning, f θ is trained jointly with a classification or regression head g ϕ ( . ) to predict the desired target
y i with the following learning objective.
[Equation]
where L is a loss function.
However, in representation learning (e.g., self-supervised learning), f θ is first pretrained using different objective to obtain a feature extractor f θ ∗ , which produces a latent embedding for a given input.
Classification or regression heads g ϕ ( . ) can then be trained using the latent embeddings keeping f θ ∗ fixed, where the learning objective takes the following form.
[Equation]
3.2 Realistic data shifts in EEG data
In this study, we focus on modeling data shifts in EEG datasets caused by real-world variability.
Consider the training (i.e., source) data domain p s ( x ) and the real-world data domain (i.e., target)p t ( x ) , where x is input data.
Covariate shifts are defined when the concept mapping g ϕ ◦ f θ : x → yremains unchanged, resulting in the same conditional distribution of labels i.e., p s ( y | x ) =
p t ( y | x ) .
However, the marginal distributions of data are assumed to change i.e., p s ( x ) ̸ = p t ( x ) .
In thisstudy, we assume no access to external labels or additional data from the real-world.
Instead, wesynthetically construct the target domain p t ( x ) based on an expert understanding of the EEG signalacquisition process via a set of EEG data transformations T .
Therefore, within the scope of thecurrent study, p t ( x ) = p s ( T ( x )) , and the proposed robustness assessment approach can be viewedas quantifying the differences between p s ( x ) and p s ( T ( x )) through a set of metrics M .
The effects of real-world variability on EEG segments can be modeled using a data transformationfunction t : R M × N → R M × N , where ˜ x = t ( x ) represents a shifted EEG sample.
Real-world EEGdata shifts may occur due to differences in populations, behavioral states (e.g., awake, asleep), thenumber of EEG sensors used, the EEG reference used, noise during acquisition, analog-to-digitalconversion, hardware filter settings, and impedance resulting from electrodes.
However, here we focuson data shifts induced by acquisition conditions and assume that other variables are controlled.
Inthat context, we develop four EEG transformations: 1) band-pass filter t BP , 2) quantization precisiont QP , 3) narrowband impedance noise t IN , and 4) broadband noise t BN (summarized in Table 1).
[Table]
Band-pass filter: EEG device manufacturers commonly implement hardware-level band-pass filters
[Table]
in order to restrict the spectral content of the acquired signal.
However, those filter settings may varybetween different manufacturers and devices.
Hence, we define a data shift t BP to reflect the effectsof different band-pass filter settings such as, 0.5–30 Hz, 1–30 Hz, and 1–25 Hz.
Quantization precision: In commercial EEG devices, resource constraints limit the precision of therecorded EEG data and the analog signal is digitized with variable number of bits.
We implementthis effect as a truncation of decimal digits to either 6, 8, or 12 points and denote it using t QP .
Signal noise: The acquisition of EEG signals is highly susceptible to multiple sources of noiseand artifacts: subject movement-related sources in the form of eye blinks, head movement, jawclenching, etc., environmental sources such as AC power line interference, and instrumentationrelated sources such as electrical impedance due to poor sensor contact with the scalp.
We use t INto model electrical impedance noise due to poor sensor contact or choice of dry/wet electrodes thatmanifests as narrow-band low-frequency (0–1 Hz) random noise in the EEG signal (37) with differentstrengths ( σ ∈ { 0 . 001 , 0 . 01 , 0 . 1 } ).
Next, we use t BN to capture random physiological noise that maymanifest as broadband (0–45 Hz) noise with different strengths ( σ ∈ { 0 . 001 , 0 . 01 , 0 . 1 } ).
3.3 Topological analysis of the latent space
Here we aim to quantify the distortion in latent space caused by input data transforms, i.e., differencesbetween z
= f θ ( x ) and ˜ z
= f θ ( T ( x )) .
Some previous studies have utilized distance metrics suchas Wasserstein’s distance to quantify the differences (7).
Here we employ a more comprehensiveapproach adapted from (38) and (24), where we estimate a manifold that contains point cloudscomprising the latent representations { z i } and { ˜ z i } and conduct a geometric/topological analysis onthis estimated manifold.
We estimate the manifold based on Delaunay neighborhood graphs, whichconnect spatial neighbors in a way that captures local point density as well as global outliers (24).
Suppose we use Z ∈ R d and ˜ Z ∈ R d to denote the sets including all embeddings of raw andtransformed input data, respectively, and define a set of vertices V =
{ Z ∪ ˜ Z } .
Then, the Voronoicell of v ∈ V is the set of points in R d for which v is the closest among V , i.e., Cell ( v ) = 153 (cid:8) x ∈ R d | ∥ x − v ∥ ≤ ∥ x − v i ∥ ∀ v i ∈ V (cid:9) .
Then, we define edges E by connecting points v
i andv j whose Voronoi cells intersect, i.e., E = { ( v i , v
j ) ∈ V × V | Cell ( v i ) ∩ Cell ( v j ) ̸
= ∅ , v i ̸
= v j } .
This set of vertices V and edges E form the neighborhoods in the Delaunay graph G =
( V , E ) .
Intuitively, the points in V that share a Voronoi cell boundary are considered natural spatial neighborsirrespective of the distance between them.
This property of Delaunay neighborhoods enables robustmanifold estimation even when point density varies locally and outlier points exist.
Notably, thesespatial neighbors or edges can emerge in a way that connects points in Z to points in ˜ Z
(denotedby R and E respectively in (24)).
Such edges can be considered ‘heterogeneous’, as opposed to‘homogeneous’ edges which connect two points belonging to the same set.
The degree of edgeheterogeneity of G can be measured by a simple quality measure (38) q ( G ) defined in Equation 1.
In Equation 1, (cid:12)(cid:12) G Z (cid:12)(cid:12) E and (cid:12)(cid:12)(cid:12) G ˜ Z (cid:12)(cid:12)(cid:12) E denote the number of homogeneous edges within G belonging to
Z and ˜ Z , respectively, and |G| E denotes the total edges in G .
Therefore, if G exclusively contains homogeneous edges, q ( G ) = 0 , and if all edges are heterogeneous, then q ( G )
= 1 .
In this work, weuse the degree of edge heterogeneity, q ( G ) , as a measure of latent space integrity between un-shiftedand shifted samples under perturbations defined above ( T ).
[Equation]
3.4 Quantifying model uncertainty using Monte Carlo dropout
In this work, we use a Monte Carlo dropout (MCD) strategy to examine per-sample variability in predictions, as a measure of model ‘uncertainty’ for a new sample (23).
Specifically, we directly comparethe changes in model uncertainty (measured using the standard deviation) under the aforementioned
EEG-DS.
Here we briefly summarize the theoretical background and our implementation.
In the Bayesian setting, the predictive distribution of a model f with parameters θ for an input x
andoutput ˆ y is expressed as p (ˆ y | x, D ) =
(cid:82) p (ˆ y | x, θ ) p ( θ | D ) dθ , where D is training data.
Becauseexact computation of this expression is intractable, we perform approximate inference using a variational distribution.
A previous study showed that training neural networks with Bernoulli dropoutvariables is equivalent to approximate inference of the posterior of model parameters (23).
With T setsof Bernoulli realizations of θ { θ t } Tt =1 , the Monte Carlo estimate of the mean prediction and its variance are computed as E [ˆ y | x ] ≈ 1 T (cid:80)
Tt =1 f ( x, θ t ) and V ar (ˆ y | x ) ≈ 1 T (cid:80)
Tt =
( f ( x, θ t )
− E
[ˆ y | x ]) 2 .
In our experiments, we used a dropout probability of c = 0 .
5 and repeated forward passes T =
20times to estimate the mean and variance of predictions ˆ y .
We then compare the standard deviation andthe agreement ratio of the predictions to make conclusions about predictive uncertainty of regressionand classification tasks, respectively.
The agreement index between multiple Monte Carlo dropoutsfor a threshold τ t is defined as, ϕ = 1 T (cid:80)
Tt =1 1 { f ( x, θ t ) ≥ τ t } .
Experimental Setup
Our overall experiment setup is illustrated in Figure 1.
Using raw EEG signals and their shiftedversions under transforms EEG-DS introduced in Section 3.2, we evaluate the robustness of EEGencoders in both the latent space and the output predictions.
Through our experiments, we hope toanswer the following questions: a) do latent space analysis and predictive uncertainty help anticipatemodel behavior in deployment settings?
, b) what types of real-world EEG data shifts impact modelrobustness the most?, and c) do some encoders exhibit better robustness than others?
In the following,we briefly describe the components of our evaluation setup: the EEG dataset, preprocessing steps, EEG feature encoders, learning tasks, data splits, and evaluation metrics.
[Figure]
EEG dataset & preprocessing: We utilize data from a public large-scale database including scalp EEG recordings of patients who were evaluated at the Temple University Hospital (TUH) (13).
Asubset of recordings have been carefully reviewed and broadly annotated by experts as either “normal" or “abnormal", and have been released as a derived dataset called the TUH EEG Abnormal Corpus(TUAB).
For our experiments, we only utilize the TUAB recordings, containing a total of {+
 EEGs from 2329 distinct patients.+}
All EEGs were recorded according to the standard 10-20 montage(39).
We extracted age labels from text reports accompanying the EEG recordings using a previouslypublished tool (40).
Next, we preprocessed the EEGs as follows: a) we ordered the EEG channels as‘Fp1’, ‘Fp2’, ‘F3’, ‘F4’, ‘C3’, ‘C4’, ‘P3’, ‘P4’, ‘O1’, ‘O2’, ‘F7’, ‘F8’, ‘T7’, ‘T8’, ‘P7’, ‘P8’, ‘Fz’,‘Cz’, and ‘Pz’, b) resampled the recordings to 128Hz, c) applied a bandpass filter between 0.5Hz and45.0Hz, d) divided the recordings into contiguous non-overlapping epochs of 10-seconds each, e)identified and removed bad epochs when the total power in their ‘Cz’ channel exceeded 2 standarddeviations as calculated from statistics of each recording, f) clipped amplitude values to not exceed± 800 µ V, and finally g) we normalized the recording per-channel using the identified valid epochs.
Note that transforms T were applied directly on the raw data before preprocessing.
EEG encoders: We train three diverse types of raw EEG encoders: fully-supervised, self-supervised,and traditional feature-based.
The fully supervised encoders (FSE) were based on the widely used‘ShallowNet’ architecture (14) with additional classification and regression heads described below.
The self-supervised encoder (SSE) also utilized the ‘ShallowNet’ architecture with pre-trainingperformed using a proxy measure of the patient’s behavioral state (17).
Finally, we utilized a powerspectral-density-based encoder (PSDE) with EEG spectral power calculated within 7 frequency bands:δ (2-4Hz), θ (4-8Hz), low α (8-10Hz), high α (10-13Hz), low β (13-16Hz), high β (16-25Hz), andγ (25-40Hz) for each of the 19 channels.
Both FSE and SSE took raw EEG traces as inputs andproduced 128-dimensional embeddings, while PSDE generated 19 × 7 = 133 features.
We utilized additional classification and regression heads to develop task-specific predictors using theabove encoders.
Note that we trained FSE entirely using the task labels, whereas we fine-tuned SSEby training the classification/regression layer while keeping the encoder network frozen.
In addition,we employed dropout with p = 0 .
5 in all heads and used a binary cross-entropy loss and a smoothvariant of the mean absolute error loss for training the classification and regression heads, respectively.
We trained the models up to a maximum of 500 epochs and while monitoring convergence using avalidation set of 373 EEGs.
We used either Adam or SGD with a cyclic learning rate scheduler (41)optimizer.
The full set of hyper-parameter values are provided in the supplementary material and arealso documented within code released as a URL in the supplement.
Classification and regression tasks:
We utilized meaningful classification and regression tasksto evaluate the robustness of above encoders.
EEG grade classification as ‘normal’ or ‘abnormal’is a common task performed by clinical experts via visual review.
Performing such classificationusing ML approaches in an automated fashion is of great interest since it has the potential to reducethe workload of clinical experts.
Next, the potential to predict an individual’s age using EEGs, aregression task commonly known as brain age prediction, is useful because it can indicate deviationsfrom healthy brain aging and therefore help with early diagnosis of neurodegenerative diseases.
Hence, we utilized two tasks: a binary classification task classifying EEG grades as ‘normal’ or‘abnormal’ and a regression task predicting brain age from EEGs, to conduct our experiments.
Training, validation, and testing: We split TUAB into fixed training (1,490 EEGs; 258,631 epochs),validation (373 EEGs; 68,493 epochs), and held-out test (466 EEGs; 82,331 epochs) sets.
Furthermore,we performed all model training at the epoch level.
Since both EEG grade and age are recording-levelmeasures, during evaluation, we aggregate epoch-level predictions by averaging the predictions acrossall epochs in a recording, an approach commonly used in EEG-ML studies (17) (20).
Evaluation metrics: We perform all evaluations on the held-out test set.
The key metrics utilized toperform the proposed encoder assessments are described in Section 3.
Based on the graph constructionoutlined in Section 3.3, we interpret the measure of edge heterogeneity obtained using Equation1 as a score of integrity of the latent space under perturbation.
For each encoder, we compare therepresentations of raw EEG samples against their shifted versions in the latent space.
Next, we usepredictive uncertainty to analyze per-sample variability of encoders under dataset shifts.
Specifically,we analyze agreement index ( ϕ ) and standard deviation in model predictions obtained via MCD asmeasures of uncertainty.
Finally, we quantify task performance using the ground truth labels.
Toanalyze task performance, we use the area under the receiver operating characteristic curve (AUC)and the mean absolute error (MAE) metrics for EEG grade and brain age tasks, respectively.
Notethat we performed the analyses of the latent space using embeddings of all the epochs in the test setand analyzed task performance and uncertainty using aggregated EEG-recording level metrics.
Software & hardware:
The preprocessing, baseline features, and model training were implementedusing a combination of the following Python libraries: 1) MNE-python (42), 3) PyTorch (43), and 3)
NumPy (44).
Delaunay graph construction and analysis were performed using code released by (24).
The experiments were performed using 2 Nvidia RTX 3090 GPUs.
Results
Our results are summarized in Figures 3 and 2 and Table 2.
Note that we performed our analyses ofthe latent space and uncertainty quantification without knowledge of the ground truth labels. However,we subsequently compared the classification and regression performances using the ground truthlabels to make general conclusions about the different encoders and data shifts.
In the following,we use the following acronyms to refer to the data shifts and encoders: BP - bandpass filter, QPquantization precision, IN - impedance noise, BN - broadband noise, SSE - self-supervised encoder, FSE-grade - fully-supervised encoder fine-tuned on grade classification, FSE-age - fully-supervisedencoder fine-tuned on age prediction, PSDE - traditional power-spectral density-based encoder.
[Figure]
Impact of data-shifts on predictions : Figure 2 shows an example of how model predictions areaffected when the proposed transformations are applied to the input data.
In the top row, we show acomparison of the predicted probability of EEG grade by the FSE encoder on held-out data underadditive broadband noise t BN of varying strength σ .
The probabilities are grouped along the x-axis byexpert labels.
We observe that the predicted probability distributions for the two ground truth classesoverlap more under stronger shifts.
In the bottom row, we show scatter plots of brain age predictedby FSE against the ground truth under similar comparison settings as above.
With stronger shifts, weobserve a collapse in the range of FSE predictions.
Evaluation of the latent space : Figure 3 shows the latent space integrity scores ( ↑ ) computedbetween unmodified and realistically shifted versions of EEG data as defined in Table 1 for multiple EEG encoders.
The same results are provided in a tabular form in Table 4 of the appendix includingstandard deviations of the scores.
First, we established an empirical baseline for ideal integrity scoresby comparing un-modified data with itself for each encoder (labeled as ‘No shift’ in Figure 3).
Thenwe compared the integrity scores between the encoders by increasing the strength of each data shift(left → right).
We find that all proposed data shifts affect the integrity scores, albeit to varyingdegrees, with BN and QP being the most and least effective, respectively.
Interestingly, we also findthat PSDE and SSE encoders produce better overall integrity in the latent space compared to others.
Task performance & uncertainty: Table 2 lists the task performance measures and the uncertaintymeasures for EEG grade classification and brain age regression tasks evaluated on the held-out setunder different data shifts. Note that all encoders were trained on un-shifted data and shifts were
N o s h
i f t f = [ 0 .
5 ,] f
= [ 1 ,] f = [ 1 ,] 0.
BP
N o s h
i f t D =
D =
D = 6 0.
QP
N o S
h
i f t = 0 [-.= 
-].
001 = 0 .= 0 .
IN
N o S
h
i f t = 0 [-.= 
-].
001 = 0 .= 0 .
BN
[Figure]
applied only at test-time (see Figure 1).
The task performance metrics reported are the means of therecording-level AUCs and MAEs computed based on the 20 MCD repeats (standard deviations areprovided in Table 3 of the appendix).
Regardless of shift, we find that FSE outperforms both SSE and
PSDE in both tasks significantly.
However, SSE and PSDE exhibit consistent performance underperturbations in the age prediction task although their absolute performance is worse compared to FSE.
Data shifts IN and BN affect all encoders in EEG grade classification, and BP, IN, and BN affect FSE encoder in age prediction.
As measures of uncertainty, we report the average agreement index in EEG grade prediction and the standard deviation of the predicted age values across [-
-]20 MCD repeats.
We find that both SSE and PSDE exhibit overconfidence in model predictions despite achievingworse task performance than FSE in both tasks.
Next, all EEG-DS cause only marginal changes onuncertainty measures and only the strongest BN ( σ = 0 . 1 ) causes a noticeable change.
age regression (Age) evaluated on a held-out set under different data shifts.
We measure classification performance using AUC ( ↑ ), regression accuracy using MAE ( ↓ ), classification predictive uncertainty using the agreement index - ϕ ( ↑ ), and regression predictive uncertainty using the standard deviation of the predicted values - SD ( ↓ ).
Up/down arrows indicate direction of desirable trends.
[Table]
Discussion
The aim of this study was to assess robustness of EEG encoders under various realistic datasetshifts.
We modeled real-world variability in EEG signals by designing four data shifts, i.e., EEGDS (summarized in Table 1), that capture the effects of variable hardware-level band-pass filters,analog-to-digital quantization precision, and manifestations of noise in scalp EEG signals.
Then,we trained diverse types of EEG encoders (self-supervised, fully-supervised, and conventional) onun-shifted data for two clinically important tasks (Section 4), and assessed their robustness underthese shifts during model development.
Notably, we analyzed the encoders’ latent space (Figure3) and predictive uncertainty (Table 2) in a label-agnostic fashion, in addition to analyzing taskperformance (Table 2).
The significance of our study is that we propose evaluation metrics beyondtraditional task performance that can be computed during model development to evaluate modelrobustness and anticipate failures in practical deployment settings.
The value of latent space analysis : If we consider task performance as gold-standard metric indeployment settings, the latent space analysis allows us to see ‘early’ signs in the ML pipelinesuch as distortions in the latent space that predict degraded task performance (Figure 3 and Table
Intuitively, a strong distortion in the latent space is likely to push samples from one side of thedecision boundary learned by the model to the other, leading to worse task performance.
We believethat such integrity/distortion analyses can be used to early detect failures during deployment.
The value of uncertainty quantification : Our uncertainty studies identify a major pitfall that ispossible to occur during deployment.
We found that both SSE and PSDE encoders were highlyconfident in their predictions although their task performances were much worse compared to FSE.
This might indicate that the SSE and PSDE encoders were not sufficiently trained to achieve desirabletask performances (i.e., classifier bias) and hence showed less variability in predictions.
On the otherhand, the uncertainty measures of the FSE encoder provided meaningful interpretations although thechanges in uncertainty under the proposed data shifts were marginal.
Data shifts : Trends in Figure 3 indicate that a (1-25Hz) band-pass filter and strong additive broadbandor narrowband noise ( σ = 0 .
1 ) impact latent space the most.
Table 2 shows the same trend in taskperformance, although the effects are more pronounced in age prediction compared to EEG gradeclassification.
From an EEG perspective, sensitivity to band-pass filtering and noise is expectedsince they affect the spectral content of the signal, which is important for the tasks of EEG gradeclassification and brain age regression.
We found that the data shifts did not impact uncertaintymeasures all that much except for the strongest broadband noise ( σ = 0 . 1 ).
EEG feature encoders : A general trend seen in Table 2 across all shifts is that although FSE providesbetter absolute performance, SSE and PSDE encoder performance, although poorer, is more stable.
This apparent trade-off between absolute performance and robustness can be explained by consideringthe effective model sizes (i.e. number of learnable parameters) in each setting.
Recall from Section 4that only the last linear layer g ϕ ( . ) is trainable for SSE and PSDE while the entire network g ϕ ( f θ ( . ))is trained in case of FSE.
Due to disproportionately higher number of parameters, FSE can apriori beexpected to perform better than SSE and FSE.
However, this also implies that dropout in FSE wouldinactivate a disproportionately higher number of units, leading to less stable performance.
While ourresults are inconclusive about the superiority of any single type of EEG encoder, they suggest thatmodel parsimony may be a good design principle/goal to ensure robustness in deployment settings. This observation is reflected in the latent space integrity analysis where SSE and PSDE showed betteroverall robustness profiles compared to FSE (Figure 3). Limitations and future directions: This study focuses exclusively on instrumentation-related shiftsin scalp EEG signals.
However, shifts originating from inherently spontaneous brain activity mayalso play a significant role in real-world robustness.
Secondly, our design of the quantizer precisionshift could be made more rigorous by re-mapping the discrete signal amplitude to a restricted set ofquantized numerical levels instead of manipulating decimal precision.
Directions for future workinclude, implementing additional data shifts representing the physiological variability in brain activity,investigate whether training with data shifts would improve the proposed assessment metrics, andevaluating the real-world value of the proposed framework using data from external sites.
Conclusion
In this paper, we address the problem of generalization under data shifts in healthcare machinelearning.
Our approach included implementation of realistic data shifts and evaluation of modelrobustness using a topological analysis of the latent space and quantification of predictive uncertainty.
Focusing on the task of scalp EEG representation learning, we developed four domain-guided datashifts for EEG signals that reflect the real-world variability observable during test time.
We thenperformed experiments to evaluate differences in latent space integrity, predictive uncertainty, and taskperformance, under the above data shifts.
Through evaluation of multiple EEG feature encoders usinglarge-scale EEG data, we empirically showed that the proposed approach can help in anticipatingfailure scenarios during deployment and that self-supervised and traditional feature-based EEGencoders exhibit better robustness in the latent space, although such robustness may fade with tasktraining.
These insights can be helpful in deciding when not to trust a model’s predictions in highstakes fields such as healthcare.
In future work, we will consider other plausible data shifts for EEGdata, training models explicitly on realistic data shifts, and evaluation on datasets from external sites.