Contrastive Learning of Electrodermal Activity Representations for Stress Detection
Abstract
Electrodermal activity (EDA), usually measured as skin conductance, is a biosignal that contains valuable information for health monitoring.
However, building machine learning models utilizing EDA data is challenging because EDA measurements tend to be noisy and sparsely labelled.
To address this problem, we investigate applying contrastive learning to EDA.
The EDA signal presents different challenges than the domains to which contrastive learning is usually applied (e.g., text and images).
In particular, EDA is non-stationary and subject to specific kinds of noise.
In this study, we focus on designing contrastive learning methods that are tailored to EDA data.
We propose novel transformations of EDA signals to produce sets of positive examples within a contrastive learning framework.
We evaluate our proposed approach on the downstream task of stress detection.
We find that the embeddings learned with our contrastive pre-training approach outperform baselines, including fully supervised methods.
Electrodermal activity (EDA) refers to electrical changes that arise in the skin and it is most commonly recorded using a measure of electrical conductance on the surface of the skin (1).
Changes in EDA are related to sympathetic nervous system activity and are representative of changes in psychological or physiological arousal (1; 2).
EDA can be measured through wrist-worn smartwatches which makes it a valuable signal for remote health monitoring. A leading application of EDA is stress estimation and numerous lab studies have validated its association with stress (3; 2; 4; 5). Stress is thought to be a significant factor in many health conditions (6; 7).
As such, several recent longitudinal studies have incorporated wrist-worn EDA measurements to deepen our understanding of how stress mediates conditions such as suicidal thinking (8), substance-use disorder and post-traumatic stress disorder (10). Beyond stress, EDA is an important signal for remote monitoring of other conditions and constructs such as seizures, sleep and pain (11; 12; 13).
There are two significant limitations to the analysis of EDA data collected in ambulatory settings.
First, modeling most EDA-related outcomes relies on self-reported patient labels (e.g., psychological stress level) or clinical labels (e.g., a diagnosis).
As such, EDA datasets are sparsely labelled; there is a natural upper bound on the number of labels one can collect (at most a few per day from a patient or clinician) relative to the number of EDA measurements one can record (ambulatory EDA runs at sample rates ≥ 4Hz).
Second, wrist-worn EDA measurements are noisy (14).
This noise often results from physical disruptions to the positioning of the watch, e.g., when the electrodes do not make consistent contact with the skin, or environmental factors, e.g., temperature and humidity.
One approach that can help to overcome the challenges of working with noisy and sparsely labelled data is unsupervised contrastive learning .
This is a self-supervised method for learning useful data representations from unlabelled data.
It works by (1) generating transformed versions of the input examples, and (2) encouraging the representations of transformations of the same (i.e., positive) examples to be close together, and transformations of different (i.e., negative) examples to be farther apart.
The efficacy of unsupervised contrastive learning methods is contingent on choosing good transformations: they should be both challenging (i.e., it is non-trivial to distinguish between positive and negative examples) and label-preserving.
In domains such as computer vision and speech processing, there is considerable literature surrounding the selection of data transformations and the use of unsupervised contrastive learning (15; 16; 17; 18).
More recently, contrastive methods have also been applied to biosignal time-series in the health domain, for example to electrocardiogram (ECG) data for cardiac arrhythmia detection and stress detection (19; 20; 21), and to electroencephalogram (EEG) data for sleep stage scoring and eye state classification (22; 23; 20).
In contrast, there is almost no existing work that examines unsupervised contrastive learning with electrodermal activity data.
The EDA signal has unique properties that make it considerably different from many of these other modalities – it is a non-periodic / non-stationary time-series that is subject to specific sources of noise when measured at the wrist.
Therefore, we do not expect the findings from other domains to generalize directly to EDA data.
In particular, it remains unclear what types of transformations work best for contrastive learning applied to EDA.
In this study, we take steps towards closing this gap.
We present ongoing work to develop an unsupervised contrastive learning framework that incorporates transformations that are specific to the nature of the EDA signal.
We evaluate these transformations on their ability to generate embeddings that have predictive utility in the downstream task of stress estimation.
We first outline the methodology of our approach.
We then present interim results on an EDA dataset with stress labels (WESAD, 24).
Methods 1.1 Contrastive Learning Formulation and Model Architecture
In line with Chen et al.(16), we formulate a self-supervised contrastive learning set up where we train a model to distinguish positive examples of the input signal from negative examples .
We segment the EDA signal into windows of a fixed length ( | x i | =
M samples).
We generate transformed versions of each segment ˜ x
i =
t ( x i ) by applying a transform t that is randomly sampled from a set of transforms T (cf.
Section 1.2).
We define positive examples as those that have the same base segment (
e.g., ˜ x i =
t ( x i ) and ˜ x ′ i =
t ′ ( x i ) where t, t ′ ∼ T .), and negative examples as those that have different base segments ( x j , ∀ j : j ̸ = i ).
Our model architecture consists of an encoder, f ( · ) , followed by a linear projection head, g ( · ) .
The encoder produces lower-dimensional embeddings of the input signal h
i = f
(˜ x i ) , which can be transferred to downstream prediction tasks.
We implement it using a 1-D convolutional neural network (CNN).
The projection head g further reduces the dimensionality of the input, producing z i
= g ( h i ) .
We implement this as a single linear layer.
We use the InfoNCE loss (15) to optimize model parameters during pre-training: [Equation] where τ is a temperature parameter, z i , z ′
i are the embeddings of a positive pair, z i , z ′ j are the embeddings of a negative pair, and sim ( z i , z ′ j ) is the cosine similarity between embeddings. Further details on the model architecture and hyperparameters can be found in Appendix A.3. 1.2 Domain-Specific Transformations of the EDA Signal
We collaborated with several EDA experts to develop EDA-specific transformations.
Details and visualizations are in Figure 1 and Appendix A.2.
Transformations that alter the high-frequency components of the signal.
Existing work has found that nearly all of the information related to sympathetic nervous system activity is restricted to the low frequency components of EDA ( < = 0 .
24 Hz) (25).
Motivated by this finding, we design several transformations that primarily alter the parts of the signal where relevant information is not expected:
• Low-Pass Filter:
We apply a filter that passes only signals with a lower frequency than a chosen cutoff frequency.
We examine cutoff frequencies: f = { 0 . 1 , 0 .
• Band-Stop Filter:
We apply a filter that rejects / attenuates a specific band of frequencies, while letting all others pass.
We examine reject frequencies: f = { 0 . 1 , 0 .
• High Frequency Noise: We add noise to the high frequency bands of the signal.
To do this, we map the signal to the frequency domain using an FFT, and add Gaussian noise to the frequency components starting at a chosen cutoff.
We examine cutoff frequencies: { 0 . 5 , 1 . 0 } and σ = { 0 . 1 , 0 . 2 , ..., 0 .
for the noise distribution.
Transformations based on signal decomposition.
EDA signals are characterized by two components: (1)
a slow-varying tonic
component, which represents the skin conductance level (SCL) and (2) a faster-varying phasic component that represents the skin conductance response (SCR) (1; 14).
Because SCRs can reflect responses to stimuli (such as external stressors), the phasic component of the signal is of particular interest for stress detection (26):
• Extract Tonic: Isolates the tonic component of the EDA signal. • Extract Phasic: Isolates the phasic component of the EDA signal.
Transformations that simulate artifacts.
EDA data, even when collected in lab settings, are prone to artifacts, i.e., changes in the signal that are not a result of the electrodermal system (27; 28).
Because artifacts do not reflect sympathetic nervous system activity, we would like our model predictions to be invariant to them.
To help achieve this, we simulate adding two common types of artifacts that stem from the data recording process:
• Motion Artifact: We add simulated motion artifacts to the signal, i.e., artifacts that arise when the placement of wearable sensors is altered due to movement.
Motion artifacts often appear as abrupt peaks in the signal.
• Loose Sensor Artifact: We simulate the addition of artifacts that appear when the sensors lose contact with the skin.
This typically results in the signal dropping to near-zero values.
Standard data transformations.
We include transformations commonly used on other signals:
• Gaussian Noise: We add noise by sampling from a Gaussian distribution with σ = { [Table]
• Temporal Shift: We shift the signal forward or backward in time.
We examine shifts of length l = { 10 , 20 , 30 , ..., 240 } samples.
Experiments 2.1 Experimental Settings
Dataset.
We use the WESAD dataset (24), a multimodal wearable sensor dataset for stress and affect detection.
WESAD includes data collected from 15 subjects in a laboratory setting.
Subjects were exposed to experimental conditions that were designed to elicit different affective states.
We focus on the binary classification task of distinguishing between the stress and baseline (i [Table] e., neutral) conditions.
The WESAD dataset contains multiple physiological measures, but since our study focuses on EDA data, we use only the EDA wrist data in our experiments.
As in prior work (24), we segment the EDA data into 60-second wide windows, overlapping with a window shift of 0.25 seconds.
This produces 103,172 segments with a stressed or non-stressed label (based on the experimental condition).
There are more non-stressed segments (65%) than stressed segments (35%).
Evaluation.
To assess the utility of our proposed contrastive pre-training approach, we freeze the pre-trained encoder and examine the performance of a linear classifier trained on top of it.
As baselines, we consider two other models with the same architecture (i.e., the same encoder and linear classifier): a model trained end-to-end with supervised learning (SL), and a linear classifier trained on top of a frozen, randomly initialized encoder (RE).
During the contrastive learning phase we train with all examples (and assume no access to labels).
During the supervised learning phase, we train on x % of the labelled data (we vary x from 10% to 100% ).
This setup allows us to examine how each method performs when we make different assumptions about the sparsity of labels.
We perform leave-one-subject-out (LOSO) cross-validation to assess how well each method performs on held-out subjects.
The details of our model architecture and hyper-parameter selection are provided in Appendix A.3.
2.2 Results and Discussion
Impact of individual transformations.
To understand the utility of each transform we examine versions of the contrastive pre-training encoder trained with only one transform applied (i.e., for positive pairs of examples, one segment is transformed ˜ x
i =
t ( x i ) , while the other is not ˜ x i = x i ).
Table 1 displays the test accuracy and F1 score for each single-transform experiment, where 10% of the labelled data is used.
The best parameters for each transform are selected based on the validation accuracy (see Appendix A.4).
We see that the Temporal Shift transform achieves the best accuracy (Acc=0.86), followed by Gaussian Noise (0.79) and Low-Pass Filter (0.78).
By contrast, we find that the Extract Tonic and Loose Sensor Artifact transforms perform worst with accuracies of 0.62 and 0.66, respectively.
We hypothesize that the strong performance of the Temporal Shift transform is partially due to the labelling procedure of the WESAD dataset.
The labels are derived from experimental conditions, which are contiguous in time, so the Temporal Shift transform is necessarily label-preserving.
In future work, we will investigate if this result holds in real-world datasets, where stress can fluctuate quickly, producing more frequent changes in the labels.
The poor performance of the Loose Sensor
Artifact transform may be related to the controlled data collection setting, where we expect to see few instances of improper sensor placement.
The limited utility of the Extract Tonic transform aligns with previous findings that the phasic component of EDA is the most useful for stress detection (26).
Impact of amount of labelled data.
We consider the performance of contrastive learning models relative to the baselines of supervised learning (SL) and random encoder (RE).
Figure 2 compares these models over different fractions of the labelled data.
We examine three different versions of the contrastive learning approach: one that uses all transforms, one that uses only the best transform ( Temporal Shift , Top-1), and one that uses the Top-6 transforms ( Temporal Shift , Gaussian Noise ,
Low-Pass Filter , High Frequency Noise , Band-Stop Filter , and Motion Artifact ), where the transforms are ranked on validation performance from the single transform experiments (cf. Appendix A.4).
The baseline model performances are as expected: the accuracy of RE is low and does not increase notably with more labelled data, reflecting its limited capacity to learn, while the accuracy of SL gradually increases with more labelled data.
Regarding the contrastive models, we see that Topconsiderably outperforms all other models at all label fractions.
Adding more transforms to the pre-training procedure worsens performance, with the Top-6 model only exceeding SL performance at label fractions ≤ 20% , and the All Transforms model performing on par with RE.
The performance of the Temporal Shift transform is remarkably strong; it beats the fully supervised model for all fractions of data (even 100% ).
The finding that the other transforms do not add utility beyond Temporal Shift is unexpected. However, most of the other transforms involve adding noise to the signal, and since the WESAD data was collected in a lab, it is relatively clean. In future work, it will be interesting to see if the utility of our proposed transforms is more apparent on noisier, naturalistic data.
Conclusion
This work shows the promise of using contrastive learning to address label sparsity and noise in EDA analysis.
In future work, there are several limitations that we will address.
First, the WESAD dataset was collected in an artificial laboratory setting, so we will need to investigate additional datasets to understand how our results translate to more naturalistic data.
Second, stress is likely best categorized by multiple physiological modalities and therefore we will extend this work to consider multi-modal contrastive learning methods.
A Appendix A.1 Related Work
Self-supervised learning (SSL) using contrastive learning (CL) techniques such as Contrastive
Predictive Coding (CPC), SimCLR, and Wav2Vec has enabled significant progress on learning from noisy and sparsely labelled datasets in domains such as computer vision and speech processing (15; 16; 17; 18).
More recently, contrastive methods have also been applied in health domains, for example to electrocardiogram (ECG) data for cardiac arrhythmia detection (19; 20) and stress detection (21) and to electroencephalogram (EEG) data for e.g., sleep stage scoring and eye state classification (open vs. closed) (22; 23; 20).
Multimodal contrastive learning methods that use the signals from more than one sensor in a wearable have been considered in studies to predict generic health and demographic characteristics such as VO 2 max and age (29).
Regarding CL methods for electrodermal activity (EDA), EDA has been included as a modality in multimodal CL systems that apply standard transforms across all signals generically (i.e., using transforms that are agnostic to the nature of the sensor signals) (30; 31).
However, to the best of our knowledge, no previous work has focused exclusively on EDA in a domain-specific way that carefully considers the nature of the EDA signal – which is non-stationary and subject to specific sources of noise when measured at the wrist.
A.2 Implementation Details of Data Transformations
• Low-pass Filter:
We apply a low-pass Butterworth filter of order 4 using the signal filter function provided in NeuroKit 2 (32). • Band-stop Filter: We apply an IRR notch filter with quality factor Q = 0 .
• High Frequency Noise: We apply a fast Fourier transform (FFT) to map the signal to the frequency space.
We sample noise from a Gaussian distribution with a standard deviation proportional to σ .
We scale σ based on the mean power of the signal, to control the resulting signal-to-noise ratio (SNR) across signals.
We add noise to the FFT of the signal and then map it back to the time domain.
• Extract Tonic/Phasic: For these two transforms, we separate the signal into its two components using the high-pass filtering extraction method provided in NeuroKit2 (32) • Motion Artifact: We sample a number of sensor spikes s (i.e., sharp peaks in the signal due to movement) from a uniform distribution with support { 1 , 2 , 3 } .
For each spike s , we sample a spike location l from all possible indices, and a height factor h from a uniform distribution over 0 .
1 − 0 .
To simulate a spike artifact, we compute the PDF of a Gaussian distribution centered at the spike location, scale this by the spike height factor, and add it to the original signal. • Loose Sensor Artifact: We sample a number of sensor drops d (i.e., occurrences of the sensor losing contact with the skin) from a uniform distribution with support { 1 , 2 , 3 } .
For each drop d , we randomly sample a start location s from all possible start indices, and a drop length l from a uniform distribution spanning 1 − 3 seconds.
We simulate a sensor drop by subtracting the mean power of the signal and zeroing out all resulting negative values.
• Gaussian Noise: We sample noise from a Gaussian distribution with zero-mean and standard deviation proportional to σ .
We scale σ by the mean power of the signal to control the signal-to-noise ratio (SNR).
• Temporal Shift: We randomly sample whether to shift forward or backward in time with equal probability.
We shift the signal by l samples, where l is the the length of the shift.
A.3 Model Architecture and Hyper-parameters
Model architecture.
The following modules are used to create the different models in our experiments:
(a) Encoder, f ( · ) : 1-D convolutional layers followed by 1-D max-pooling are used to transform the M-dimensional input signal, x i , to a k-dimensional embedding, h i .
Batch normalisation and dropout are used in each convolutional block and ReLU is used as the activation function.
For the experiments of this paper, we use 3 convolutional blocks in the encoder, transforming the input from a 240 dimensional input segment (60 seconds at 4Hz) to adimensional embedding.
The input and output channels of these blocks are Conv 1 (in=1, out=4), Conv 2 (in=4, out=16) and Conv 3 (in=16, out=32).
A kernel size of 7 and a stride of 1 are used in the convolutional layers and a kernel size of 2 is used in the max-pooling operations.
A single linear layer with ReLU activation then maps the output to 64 dimensions.
(b) Projection head, g ( · ) : is implemented as a single linear layer that maps the 64-D embedding, h i , to a 32-D embedding, z i .
The loss function 1 is computed on these 32-D embeddings.
(c) Linear classifier for task prediction : a linear classifier is used for supervised learning on the downstream prediction task (binary classification in the case of WESAD).
This is implemented as a single linear layer mapping from the 64-D embeddings, h i , to 1-D for the binary prediction.
This model is trained for this task with binary cross entropy loss.
For the contrastive learning models in the pre-training phase, we compose a trainable version of the encoder module (a) and the projection head module (b).
Then, for the downstream prediction phase, the pre-trained encoder is frozen and transferred to the downstream task, where the linear classifier (c) replaces the projection head and parameters in the linear classifier are optimized using the binary cross entropy loss.
For the supervised learning (SL) and random encoder (RE) models, there is only the downstream prediction phase, and these models comprise modules (a) and (c).
In the case of SL, the model is trainable end-to-end (i.e., all parameters in (a) and (c) can be optimized), whereas for RE only the linear classifier module (c) can be trained.
Hyperparameters.
Different hyperparameters are used for the contrastive pre-training and the supervised learning phases of the modeling
• Pre-training hyperparameters : Adam is used as the optimizer with batch size of 256, learning rate of 0.001, and 400 epochs.
The dropout probability is 0.1.
Early stopping is implemented using the training InfoNCE loss.
• Supervised learning hyperparameters : Adam is used as the optimizer with batch size of 32, learning rate of 0.001, and 200 epochs.
The dropout probability is 0.1.
Early stopping is implemented using the validation error
A.4 Extended Single Transform Results
The validation accuracy and F1 score for the single-transform experiments with our proposed transformations and with transformations that are commonly applied to other signals are shown in tables 2 and 3 respectively.
[Table]
F
[Table]
[Table] [Table]
F
[Table]