Paper: Universal Sleep Decoder: Aligning awake and sleep neural representation across subjects (ICLR-2024)

Reviewer comment (Evaluation):
A core claim of the paper is that the experimental paradigm allows probing memory reactivation during sleep. However, the presented analyses may not actually allow studying memory reactivation, as the trained neural encoders likely picked up on evoked activity related to the audio stimuli presented during sleep.

Author response:
Sorry for the confuse. We supplemented the extra results of Lasso GLM on sleep->awake (including sleep->image and sleep->audio) experiments. We can observe that the accuracy of Lasso GLM, on either awake->sleep experiments (see Fig.3(c)) or sleep->awake experiments (see Fig.8 in Appendix), is significantly higher than random levels.

Besides, all audio stimulus during sleep last for 0.5s. The audio stimulus used during sleep are the same with those used during wakefulness. We have revised the experimental design related to the sleep portion in Appendix A, emphasizing that the duration of the all audios played during sleep is 0.5 second.

Reviewer comment (Experiments):
The EEG recordings were epoched from -0.2 to 0.8 s around the stimulus onset (Section 4.1). The paper does not describe the distribution of audio stimuli duration, but it is likely that the audio clips lasted a few hundreds of milliseconds. In that case, the EEG windows likely contained evoked responses to these auditory stimuli rather than an associated memory. To assess that, an analysis of the evoked response that also takes into account the spatial dimension would be important (see Q1). On a related note, details of how auditory cues impacted sleep would be important to provide (Q2).

Author response:
Thank you for bringing this up, and it's indeed a crucial question. We separately computed the averaged event-related potentials (ERPs) induced by image and audio stimuli during wakefulness and TMR-related audio stimuli during sleep, along with their topographical maps. As you noted, the temporal patterns of stimulus ERPs during sleep are indeed markedly different, spanning longer windows compared to wakefulness. These results have been added to the Appendix Fig.6.

Furthermore, in order to assess the spatial dimensions of TMR-related ERPs and their similarity to auditory or visual ERPs during wakefulness, we initially identified the time point with the maximum variability in the averaged ERPs for image and audio across all participants. Each participants’ spatial distributions of ERPs at these two time points for image and audio were considered as the templates. Subsequently, the spatial correlation between the TMR-related ERPs at each time point and the two templates for each participant was computed. Finally, paired t-tests were conducted on the two sets of correlation coefficients at each time point, and significant differences were annotated. The results indicate that the spatial features (topomaps) of TMR-related ERPs are significantly closer to the wakeful period image stimulus template at 0.56-1.00 seconds and significantly closer to the wakeful period audio stimulus template at 1.29-1.42 seconds. The result have been added to the Appendix Fig.7.

Reviewer comment (Evaluation):
The data collected during sleep could be used to clarify whether memory reactivation is taking place. Specifically, decoding performance when only looking at auditory cues of mismatched pairs should be reported. If semantic category classification performance remains high for cues for which the evoked response should be different from the “memory-evoked” response (i.e. mismatched pairs), this could support the authors’ claim (see Q3).

Author response:
We apologize for not clearly describing the experimental design. In fact, there were no mismatched stimulus pairs in our experiment; we used 15 images and 15 audio as the stimulus, and all stimulus were paired during wakefulness. Additionally, we did not use the classical TMR experimental design. Instead, all the image-audio stimulus pairs used in the experiment were presented with equal probability during sleep. This is why we refer to stimuli during N2/N3 sleep as TMR-related stimuli.

Reviewer comment (Presentation):
A core contribution regarding the relative performance of the various models is not supported as well as it could be; in particular, it should be easier to determine whether performance follows the expected pattern (awake+sleep+contrastive > awake+sleep > [awake or sleep in whatever order]). Although the necessary data is available in Figure 3, it is difficult to tell from eyeballing the plots whether models such as sleep→sleep CNN are better or worse than awake+sleep→sleep CNN. These results should ideally be plotted together for clarity.

Author response:
Yes, the ideal result would be awake+sleep+contrastive > awake+sleep > sleep, as we mainly introduce awake data to improve sleep decoding. However, there is a large gap between neural patterns of awake and sleep data, (as demonstrated in Appendix Fig.6). 

Directly using a balanced batch (comprising half awake and half sleep data), does not exhibit a significant improvement on the sleep decoding performance. In some cases, there was even a slight decrease in model performance (~0.1%), primarily due to the issues mentioned before. We conducted a control experiment with CNN model, shuffling labels for awake data and inputting them alongside sleep data into the model. This significantly reduced the model's performance, see Appendix Fig.11 for more details. Besides, detailed performance (mean/std) of different models are shown in Appendix Table5-6.

Reviewer comment (Evaluation):
Sections 4.2.2 and 4.2.3 report some statistics but do not provide comparisons to directly support the core claim about performance ordering mentioned above.

Author response:
Sorry for the mistake. Yes, it’s $y\in\\{1,...,K\\}$. $y\in\mathbb{R}^{K}$ is just a one-hot vector corresponding to $y\in\\{1,...,K\\}$.

Reviewer comment (Presentation):
There is no results table for the relevant experiments that would make the comparisons clear, unless it was missed by the reviewer.

Author response:
Sorry for the mistake. $\mathcal{X}^{img}$ means all image-evoked EEG recordings of one subject, i.e., $\mathcal{X}^{img}=\\{x_{n}^{img}\\}_{n=1}^{N^{img}}$. $\mathcal{X}^{aud}$ and $\mathcal{X}^{tmr}$ are similarly defined.

Reviewer comment (Writing):
There is a notation concern about whether it is really the case that \( y \in \mathbb{R}^K \); each label as a vector of real numbers of length equal to the number of classes seems unlikely, and it should perhaps instead be \( y \in \{1 \ldots K\} \) or similar.

Author response:
Sorry for the mistake. We use $\\{1,...,|\mathcal{B}|\\}$ to indicate the indices of $\mathcal{B}$. For example:

$\mathcal{A}(i) = \\{1,...,|\mathcal{B}|\\} \setminus \{i\}$

Due to display issues on the web page, you can find the updated formula of the whole contrastive loss in the updated pdf.

Reviewer comment (Writing):
The \(\mathcal{X}\)s are not explicitly defined.

Author response:
Sorry for the mistake. We have made modificatiosn to the relevant terminologies, e.g., dataset with a considerable number of categories, dataset with few categories. We apologize for not being able to provide a more precise definition; we can only delineate the categories of the dataset in a relatively vague manner.

Reviewer comment (Writing):
In the definition \(\mathcal{P}(i) = \{k | k \in \mathcal{A}(i), y_k = y_i\}\) where \(\mathcal{A}(i)\) is a set of instances \(\{x_i, y_i\}\), the term \(k\) refers to an instance, making the meaning of \(y_k\) appear overloaded or poorly defined.

Author response:
We sincerely apologize for the omission of the crucial reference to the BENDR paper. In BENDR, the primary focus is multivariate mask modeling. In DreamDiffusion, a similar approach is employed (with a slight modification when embedding subseries-level patches).

Reviewer comment (Writing):
Another careful proofread of the math might be useful, as the reviewer caught several notation issues quickly.

Author response:
Sorry for the confuse. We have removed those confusing arrows. Updated figure (Fig.2 in main text) is uploaded.

Reviewer comment (Novelty):
Kostas et al. 2021 (doi:10.3389/fnhum.2021.653659) is likely worth mentioning in the discussion of larger-scale SSL pretraining for EEG.

Author response:
We sincerely apologize for the omission of the crucial reference to the BENDR paper. In BENDR, the primary focus is multivariate mask modeling. In DreamDiffusion, a similar approach is employed (with a slight modification when embedding subseries-level patches).

Reviewer comment (Presentation):
In Figure 4, the two plots have the same y axis, but the x axes seem to indicate percentages of different amounts of data, even though each batch is balanced. If this understanding is correct, it might be better for the x axis to reflect the number of instances or hours to facilitate direct comparison.

Author response:
Sorry for the confuse. As every sleep data item is paired with one awake-image data item and one awake-audio data item (randomly selected from the awake dataset from the same subject), the amounts of finetune data used in awake&sleep pretrained model is three times the amounts of finetune data used in sleep pretrained model. But the amounts of sleep data used in finetuning stage are equal. We have modified the figure caption to emphasize the distinction on the x-axis between Fig.4(a) and Fig.4(b). However, the label on the x-axis should remain in percentage form, as the data sizes vary for each finetuning individual, and using percentages provides a standardized representation. The caption of Fig.4 is updated.

Reviewer comment (Reproducibility):
The data is not released. Without the data it's difficult to verify the claims, since one of the main claims of the paper seems to be the unique data collected. Since this appears to be a dataset paper, it is essential that the data is released and verified before this can be accepted.

Author response:
The dataset is still under collection, targeting about 100+ available subjects. Thus, we can provide a large enough benchmark dataset for sleep decoding. And the dataset will be released by the end of this year (2023). **To verify the claims, we provide a demo dataset (preprocessed and anonymized) containing 3 randomly selected subjects. You can download the demo-dataset from [here](https://drive.google.com/drive/folders/1013NKON4U9D8E60OdLbImwRfvoGlPlaE?usp=sharing).** You can directly run the code, after creating the conda environment, see README.md for more details.

Reviewer comment (Experiments):
There is no comparison with other works. Without the dataset, this paper does not contribute much else.

Author response:
As there is no previous study that has collected such a large sleep decoding dataset, previous studies mainly uses conventional machine learning algorithms, e.g., lasso GLM, instead of deep learning models to build a sleep decoder. Not to mention the alignment between awake data and sleep data. To our knowledge, we are the first to collect such a large and high-density dataset, and apply advanced deep learning methods (e.g., supervised pretraining, contrastive learning) to the semantic sleep decoding problem.

Reviewer comment (Presentation):
What are the asterisks in Figure 3?

Author response:
Sorry, we apologize for not specifying the meaning of the asterisks in the figure caption. The asterisks indicate statistical significance, as we conducted Paired T-tests for each individual to assess differences between the models. In commonly used statistical conventions, one asterisk corresponds to a significance level (p-value) below 0.05, two asterisks below 0.01, and three asterisks below 0.001. Updated pdf is uploaded.