Paper: Large-scale Training of Foundation Models for Wearable Biosignals (ICLR-2024)

Reviewer comment (Presentation):
It would be great to include a figure representing the model showing encoder, MLP head and other implementation details.

Author response:
We thank the reviewer for pointing this out. We have now added a high-level visualization of our pre-training framework in the **new Appendix Fig. 3**.

Reviewer comment (Experiments):
What was the reason for choosing the InfoNCE loss vs. e.g., the normalized temperature-scaled cross entropy loss (NT-Xent) from SimCLR?

Author response:
This is a good point and we apologize for the confusion. As far as we understand, the term InfoNCE loss and NT-Xent are often used interchangeably for a similar loss function, for example, the SimCLR paper (“*A Simple Framework for Contrastive Learning of Visual Representations*”, Chen et al,., 2020) says “*This loss has been used in previous work (Sohn, 2016; Wu et al., 2018; Oord et al., 2018); for convenience, we term it NT-Xent (the normalized temperature-scaled cross entropy loss)*” and in the CPC paper (“*Representation Learning with Contrastive Predictive Coding*”, van den Oord et al., 2018), the same loss was termed as InfoNCE. To further clarify this, **we have now added NT-Xent term as well to where we define the loss**, “*We use InfoNCE (also known as NT-Xent) to maximize the mutual information*”, and we would like to clarify that the contrastive component of our loss is similar to SimCLR.

Reviewer comment (Experiments):
Usually large batch sizes and more learning steps are beneficial in SSL, have you experimented with even bigger batch sizes than 256?

Author response:
The reviewer raises a very good point. We agree with the reviewer that larger batch sizes have been shown to be beneficial in SSL, for instance in the original SimCLR paper (Chen et al,., 2020). However, several follow up works have shown that adding momentum training and/or regularization and/or different objective functions could alleviate the need for larger batch sizes: examples are BYOL (“*Bootstrap your own latent: A new approach to self-supervised Learning*”, Grill et al., 2020) and Barlow Twins (“*Barlow Twins: Self-Supervised Learning via Redundancy Reduction*”, Zbontar et al., 2021), to name two. More importantly, a recent work has shown that it is possible to even train SimCLR on ImageNet with smaller batch sizes without an important drop in performance (“*Towards Democratizing Joint-Embedding Self-Supervised Learning*”, Bordes et al., 2023). To address this we have now added this to the **new Appendix A.1**: “*Regarding batch size in our pre-training, while early contrastive self-supervised learning works have shown that larger batch size is necessary to improve performance (Chen et al., 2020; 2021), follow up works removed this dependency by changing the pre-training (Grill et al., 2020; Zbontar et al., 2021), and a recent work has shown that it is possible to train SimCLR on ImageNet with smaller batch sizes without an important drop in performance (Bordes et al., 2023). We also did not see meaningful change in performance with larger batch sizes in our early experiments.*”

Reviewer comment (Experiments):
Could you clarify the reason for selecting this specific encoder and specific embedding sizes, have you experimented with other models?

Author response:
We appreciate the reviewer for their careful comment. We picked the EfficientNet model architecture given its efficacy for model performance w.r.t number of parameters. However, to address the reviewer’s comment, we did a **new Ablation 5.2.4** and a **new Table 4** by comparing the performance of our default 1D-EfficientNet model with our variation of 1D-ResNet and 1D-ViT to enable comparison with alternative model architectures and model sizes: “*For comparisons, we used smooth effective rank as a crude proxy of general downstream performance, and observed that different encoder architectures, with different model sizes, can achieve relatively similar performance, demonstrating that the performance is not unique to the 1D-EfficientNet encoder architecture (Table 4). We observed that 1D-EfficientNet model yielded similar performance to these alternative encoders with significantly smaller number of parameters, which was one of the main reasons we picked 1D-EfficientNet as our default backbone encoder given less memory footprint when potentially running on a wearable device. An interesting future direction is scaling up the model size further, particularly with transformer-based models (e.g., 1D-ViT) given their scalability (Kaplan et al., 2020), and larger datasets to study its effect on downstream performance*". Also regarding the embedding size, in our early experiments, we investigated larger and smaller embedding sizes and 256 offered a reasonable sweet point and we did not tune it further afterwards.

Reviewer comment (Reproducibility):
Are you going to make the dataset public?

Author response:
This is a very important point. Given the sensitivity of health data and that such data is collected under informed consent, we cannot release the data to general public. However, we have added details for how researchers can request for sourcing the dataset and the code, “*the aggregated data that support the findings of this study can be made available on request from the corresponding author. Request for data will be evaluated and responded to in a manner consistent with the specific language in the study protocol and informed consent form. Similarly, code for all data analyses may be available upon request from the corresponding author. Requests for code will be evaluated and responded to in a manner consistent with policies intended to protect participant confidentiality and language in the study protocol and informed consent form*”, which is now added to the **new Reproducibility statement** of the paper.

Reviewer comment (Experiments):
Would you be able to include results for PPG model trained on a smaller dataset (similar number of segments as ECG)? I believe this would be useful for comparing the behavior of the method given similar amount of data.

Author response:
We thank the reviewer for their comment about, we agree that this is an important point, and we have now added a **new Appendix Table 10** containing these results.

Reviewer comment (Experiments):
Have you thought about using both modalities in one model? Do you think that modulating, e.g., ECG with PPG embeddings would improve accuracy? Such approaches are useful when one modality is less descriptive than the other, so I was interested in learning more about your thought of how this would apply to this use case.

Author response:
We thank the reviewer for this comment. We agree that this is an important future direction, and an future work could investigate multi-modal pre-training, similar to CLIP (“*Learning Transferable Visual Models From Natural Language Supervision*”, Radford & Kim et al., 2021) by supervising one modality from the other in pre-training. We think this could improve the downstream performance of weaker modalities by supervising them in the pre-training with a stronger modality, and perhaps let the encoders learn potential relationships between modalities. While this is beyond the scope of our current manuscript, we have now added this point to the **updated Discussion** section as a potential future direction": "*Another future area of investigation of investigation is multi-modal pre-training by: 1) using a multi-modal encoder that takes multiple modalities (e.g., PPG and ECG) as input, 2) or CLIP-style multi-modal pre-training by supervising one modality with another one (Radford et al., 2021), or training multiple encoders for multiple modalities simultaneously (Girdhar et al., 2023), using a contrastive loss*".

Reviewer comment (Reproducibility):
How easily will it be to improve the current labels given that they were acquired using self-reported metrics and tracing it back seems impossible?

Author response:
This really depends on the target of interest, but one way to improve upon the self-reported labels is to use labels for standard medical diagnosis such as lab/clinical recordings and standard questionnaires for certain health conditions. This requires collecting such data form informed participants, and the complexity/difficulty could vary from condition to condition.

Reviewer comment (Experiments):
**Exploration of KoLeo Regularization**: An area of potential exploration is the specific impact of the KoLeo regularization on the model's performance. Ablation studies that incrementally remove or vary the strength of KoLeo regularization could provide clarity on its role and efficacy. Such an analysis would help in understanding whether the regularization is crucial for the model's success, and to what extent it contributes to the overall performance. This is particularly important as the unique characteristics of biosignals may interact with regularization techniques differently compared to other domains.

Author response:
We thank the reviewer for pointing this out. We have now performed an experiment where we dropped KoLeo regularization from our pre-training framework to study its importance in isolation. As detailed in the **updated Table 3** and **updated Appendix Table 12**, we observed that removing KoLeo regularization from our pre-training resulted in a drop in performance, evaluated with smooth effective rank and downstream demographic variables’ prediction, for both PPG and ECG. This demonstrates the specific impact of KoLeo regularization and we have now added to the paper: “*We observed that for both PPG and ECG modalities, removing KoLeo regularization from our objective function resulted in reduced evaluation metrics (Table 3 and Appendix Table 12), demonstrating its impact on the model’s performance*”.

Reviewer comment (Reproducibility):
**Inclusion of All Results**: Transparency and completeness in research reporting are essential for reproducibility and peer review. Therefore, it is recommended that the authors include all results, particularly those referenced as "results not shown" within the main body of the paper or the appendix. Having access to these results would allow the scientific community to fully evaluate the findings, methodologies, and claims made within the paper. It would also enhance the credibility and utility of the work for those looking to build upon it.

Author response:
This is an important point. We have now added the following to the manuscript, providing further information and evidence: 1) **new Appendix Table 10**, including downstream performance evaluation of PPG and ECG embeddings in predicting demographic variables when the number of pre-training segments in PPG is similar to that for ECG in Table 1, 2) **new Appendix Table 11**, including demographic variables’ performance evaluation calculated at segment-level granularity, where each segment contributes one and only one sample in the downstream training/evaluation.

Reviewer comment (Experiments):
**Augmentation Impact Analysis**: The paper would benefit from a deeper dive into the effects of various augmentation techniques on the performance of the biosignal models. Unlike image data, where the impact of different augmentations is well-studied, the domain of biosignals remains relatively unexplored in this aspect. An appendix providing detailed results and analysis of how different augmentations influence the learning process would be invaluable. This could include which augmentations contribute most to model robustness or performance, and any augmentation-specific phenomena observed with biosignals. Given the novelty of the field, such insights could be highly influential for future research in biosignal analysis.

Author response:
We appreciate the reviewer for this thoughtful comment. We have now added details about the selection of our augmentation functions/probabilities in the **new Appendix section A.1** and have done a **new Ablation 5.2.5** with the **new Appendix Fig. 7** regarding the effect of single augmentation functions in our pre-training framework: “*We studied the effect of single augmentation functions for each modality. To do so, for both PPG and ECG, we kept each of the augmentation functions in isolation separately (with probability 1), and repeated our pre-training framework from scratch. We observed that PPG pre-training was more sensitive to the choice of single augmentation functions (more variance in performance), and channel permute and cut out were the most effective augmentation functions in isolation for PPG and ECG, respectively (Appendix Fig. 7). Future work can study more optimal ways to design modality-specific augmentation modules*”.

Reviewer comment (Experiments):
**Demographic details missing**: The paper would benefit greatly from a more detailed presentation of demographic information related to the participants whose data underpin the foundation models. Such information is essential for assessing the diversity and representativeness of the dataset, which in turn, influences the generalizability of the model across various populations. The current omission of granular demographic details leaves a gap in understanding the scope of the model's applicability. It is recommended that the authors include statistics on age, gender, ethnicity, and other pertinent demographic factors. This would not only enhance the transparency of the research but also allow for a more nuanced evaluation of the model's performance across different demographic groups.

Author response:
This is a great suggestion. We have now added the **new Appendix Fig. 5** demonstrating our dataset's (AHMS) demographics (including age, BMI, biological sex, and ethnicity), and we believe it enhances the transparency of our work.

Reviewer comment (Reproducibility):
**Concern Regarding Dataset Availability**: The dataset from the Apple Heart and Movement Study (AHMS) is central to this research, offering immense value to the broader scientific community. However, the paper doesn't clarify if the dataset will be open-sourced upon acceptance. The release of this dataset, along with the associated models and code, is pivotal for reproducibility and further research. If the authors don't have ownership of the dataset, detailed instructions on sourcing it would be essential. Clarity on this matter is requested, as it will influence the final assessment of the paper.

Author response:
We would like to thank the reviewer for their comment about this. Given the sensitivity of health data and that such data is collected under informed consent, we cannot release the data to general public. However, we have added details for how researchers can request for sourcing the dataset and the code, “*the aggregated data that support the findings of this study can be made available on request from the corresponding author. Request for data will be evaluated and responded to in a manner consistent with the specific language in the study protocol and informed consent form. Similarly, code for all data analyses may be available upon request from the corresponding author. Requests for code will be evaluated and responded to in a manner consistent with policies intended to protect participant confidentiality and language in the study protocol and informed consent form*”, which is now added to the **new Reproducibility statement** of the paper.

Reviewer comment (Experiments):
There are potential concerns on the technical methodology front. While this study is the product of extensive training and evaluation, much of the methodology draws from pre-existing studies. Although there are several SSL studies tailored for time-series data, particularly in the realm of biosignals in healthcare, the authors did not extensively compare different model architectures. Readers might be keen to discern whether biosignal SSL performance is more contingent upon scale or the model architecture itself.

Author response:
We thank the reviewer for their thoughtful comment. First, we agree with the reviewer that our study is the product of "*extensive training and evaluation*". We would like to emphasize that “*biosignal SSL*” is largely under-explored at this scale compared to other domains of deep learning and there are no universally-agreed-upon model for wearable biosignals (PPG/ECG) that we can really compare our models to. However, to address the reviewer’s comment, we did a **new Ablation 5.2.4** and **new Table 4** by comparing the performance of our default 1D-EfficientNet model with our variation of 1D-ResNet and 1D-ViT to enable comparison with alternative model architectures and model sizes. As we mentioned in text with different model sizes, different model architectures can reach to reasonable accuracies addressing reviewer’s point about “*SSL performance is more contingent upon scale or the model architecture itself*”. We now also mention that: “*For comparisons, we used smooth effective rank as a crude proxy of general downstream performance, and observed that different encoder architectures, with different model sizes, can achieve relatively similar performance, demonstrating that the performance is not unique to the 1D-EfficientNet encoder architecture (Table 4). We observed that 1D-EfficientNet model yielded similar performance to these alternative encoders with significantly smaller number of parameters, which was one of the main reasons we picked 1D-EfficientNet as our default backbone encoder given less memory footprint when potentially running on a wearable device. An interesting future direction is scaling up the model size further, particularly with transformer-based models (e.g., 1D-ViT) given their scalability (Kaplan et al., 2020), and larger datasets to study its effect on downstream performance*".

Reviewer comment (Experiments):
While this study encompasses two modalities, it seems that the authors have considered them in isolation. Pretrained models have shown effectiveness across varied modalities like images and language. It would be beneficial for the authors to delve deeper into this aspect.

Author response:
This is a great point. We respectfully believe that multi-modal pre-training, similar to CLIP (“*Learning Transferable Visual Models From Natural Language Supervision*”, Radford & Kim et al., 2021) is beyond the scope of our current work, and merits a separate detailed investigation on its own. However, we agree with the reviewer that this is an interesting future direction and we have now added this as a potential future direction in the **updated Discussion** section: "*Another future area of investigation of investigation is multi-modal pre-training by: 1) using a multi-modal encoder that takes multiple modalities (e.g., PPG and ECG) as input, 2) or CLIP-style multi-modal pre-training by supervising one modality with another one (Radford et al., 2021), or training multiple encoders for multiple modalities simultaneously (Girdhar et al., 2023), using a contrastive loss*".

Reviewer comment (Evaluation):
The pretrained embedding for PPG appears to encapsulate more information than its ECG counterpart. This raises a question: given that PPG is passively sampled and ECG is actively collected by users, could this disparity in data collection methods influence such an outcome? Additionally, conventional clinical diagnoses often rely on 12-lead ECG or periodic information like HRV derived from ECG. It would be valuable if the authors could elucidate more why the ECG embedding doesn't seem as informative as the PPG.

Author response:
We thank the reviewer for their comment. While we agree with the reviewer that ECG signals are conventionally used for diagnosing certain medical conditions, we would like to emphasize that, unlike ECG, PPG is relatively under-explored in the medical field and harder to annotate (e.g., it's easy to visually inspect ECG based on the P-QRS-T waveform, however, there are limited gold-standard manual/visual procedures like that for PPG). As we had xplained in the Results section, we believe there are likely two reasons for PPG signals being better than ECG: “*ECG signals may contain less information specific to these conditions, and/or our pre-training framework is more optimal for PPG compared to ECG (see Ablation 5.2.2 and Discussion)*” and had mentioned in the Discussion section: "*Our pre-training framework generalizes to PPG and ECG biosignals, however, we believe future work could gain improvements by incorporating modality-specific design choices*". Related to this, we have now added a new Appendix Table 10 showing that PPG still outperforms ECG with similar number of pre-training segments, shedding light on the fact that PPG embeddings being more informative than ECG embeddings is not due to the number of pre-training segments. Also, we do not believe that the data collection disparity (active vs. passive) could cause this amount of information gap between PPG/ECG, but testing that requires some infrastructural changes to the Apple watch and how these biosignals are recorded, which is beyond the scope of our work.

Reviewer comment (Presentation):
Even if the authors intend to display only aggregated information, it would be beneficial for them to include a representative visualization of both PPG and ECG signals. This would provide readers, especially those without a healthcare background, with a clearer understanding.

Author response:
This is a great suggestion and we have now added representative examples of PPG/ECG processed signals in the **new Appendix Fig. 6**.

Reviewer comment (Evaluation):
In Figure 2, both PPG and ECG embeddings demonstrate good separability based on subject IDs. However, I'm curious if two subjects with similar demographic attributes should be distinctly separated.

Author response:
This is a good question. Given that our models are trained on unlabeled data, there are various factors that could determine the information encoded in the embeddings from PPG/ECG beyond only demographics (that is evident from Fig. 1a and we know that PPG/ECG embeddings contain more information than just demographics and measured heart rate). In addition to that, T-SNE is a nonlinear dimensionality reduction (Maaten & Hinton, 2008), and the local distances in T-SNE dimensions (what is visually depicted in Fig. 2a) do not linearly represent the global distances. Furthermore, based on the decent predictive power of PPG/ECG embedding about demographics, we know that demographics are encoded well in the embedding subspace (256-D). Given these three factors, we believe that the participants who have similar demographic variables may be close in the embedding subspace (at least in some feature dimension, as evident from decent demographics prediction performance), but may not necessarily lie on top of each other in the T-SNE subspace given other information content in PPG/ECG and that T-SNE would still non-linearly separate them if they’re not exactly on top of each other in the 256-D embedding subspace.

Reviewer comment (Evaluation):
While ECG and PPG are pivotal for evaluating physiological status, there are also numerous other parameters to consider, such as HRV, heart rate, and possibly activity levels. The authors might wish to discuss this aspect further.

Author response:
We totally agree with the reviewer and believe that future work can investigate the relationship between health conditions, PPG/ECG embeddings, and other factors such as HRV/HR/activity levels. We also believe that metrics such as HR/HRV/activity levels, still provide valuable information regarding one’s health status and should not be neglected. To address this, we have added to the **updated Discussion** section: “*Also, we would like to note that while PPG/ECG embeddings are predictive of health conditions, other biomarkers including but not limited to heart rate, heart rate variability (HRV), and physical activity still provide valuable insight regarding one's health status, and their relationship to PPG and ECG embeddings should be examined further in future work*”.

Reviewer comment (Reproducibility):
No discussion around model/data release.

Author response:
We thank the reviewer about their thoughtful comment. Given the sensitivity of health data and that such data is collected under informed consent from participants, we cannot release the data to general public. Also, we are not aware of any public dataset that contains PPG/ECG recorded from wearable devices at this scale, which is the cornerstone of our manuscript, results and evaluations, that we respectfully believe is valuable for the researchers. However, to guide the researchers on how to source the data and the code, we have now added a **new Reproducibility statement**: “*the aggregated data that support the findings of this study can be made available on request from the corresponding author. Request for data will be evaluated and responded to in a manner consistent with the specific language in the study protocol and informed consent form. Similarly, code for all data analyses may be available upon request from the corresponding author. Requests for code will be evaluated and responded to in a manner consistent with policies intended to protect participant confidentiality and language in the study protocol and informed consent form*”.

Reviewer comment (Novelty):
The paper could better address previous research. For example, it claims that previous works employed biosignals recorded in clinical or controlled experimental settings, however, both [1] and [2] used large-scale _free-living_ datasets in the wild.

Author response:
We thank the reviewer for pointing out an improvement regarding this. We would like to clarify that throughout the manuscript, we had meant to claim that our manuscript is the first study on “*large-scale PPG and ECG data*”, however, we agree with the reviewer that some of the mentioned work could have been better discussed in our paper, and **we have made improvements to the Introduction and Related work sections** to address the reviewer’s comment about the prior work and to include them. The changes for this comment are: 

- We modified the Introduction section: “*Self-supervised learning often does not require explicit labels, making it suitable to pre-train foundation models on unlabeled biosignals, and has been recently proven successful for health applications (Hallgrimsson et al., 2018; Cheng et al., 2020; Kostas et al., 2021; Sarkar & Etemad, 2022; Mohsenvand et al., 2020; Gopal et al., 2021; Kiyasseh et al., 2021; Mehari & Strodthoff, 2022; Wu et al., 2020; Spathis et al., 2021; Tang et al., 2021; Yuan et al., 2023; Lai et al., 2023). While there have been recent efforts to apply SSL on wearable accelerometer data in free-living conditions (Spathis et al., 2021; Yuan et al., 2023), other SSL work have mostly used biosignal modalities such as PPG, ECG, and electroencephalogram (EEG) collected in clinical or controlled experimental settings (Cheng et al., 2020; Kostas et al., 2021; Sarkar & Etemad, 2022; Mohsenvand et al., 2020; Gopal et al., 2021; Kiyasseh et al., 2021; Mehari & Strodthoff, 2022; Lai et al., 2023)*“.

- We modified the Related work section: “*Recent work have used self-supervised learning for wearable accelerometer signals in free-living conditions and large dataset (Spathis et al., 2021; Yuan et al., 2023)*”.

- We would like to respectfully clarify that we still believe our claim in Abstract “*To the best of our knowledge, this is the first study that builds foundation models using large-scale PPG and ECG data collected via wearable consumer devices — prior works have commonly used smaller-size datasets collected in clinical and experimental settings.*” is still accurate in consideration of the works the reviewer has mentioned as they use accelerometer data, which is a different modality compared to PPG/ECG.

Reviewer comment (Experiments):
Regarding the augmentations, it is not clear whether assigned probabilities are found through hyper-parameter tuning or heuristics. The reviewer points the authors to [3] for further experimental decisions about the order and impact of these augmentations.

Author response:
We thank the reviewer for this careful comment. We have now added details about the selection of our augmentation functions/probabilities in the **new Appendix section A.1** and have done a **new Ablation 5.2.5** with **new Appendix Fig. 7** regarding the effect of single augmentation functions in our pre-training framework. We now mention that “*we did not comprehensively tune our augmentation module probabilities, we started with a preliminary set of probabilities in our early experiments based on prior work (Cheng et al., 2020; Tang et al., 2021), for instance, cut out was shown to be the most effective augmentation in (Cheng et al., 2020) and we assigned a higher probability to it. We did not tune these probabilities afterwards and the only change made was to make the ECG augmentation probabilities stronger (2x probability) to allow for more mismatch between positive pair representations*” and that “*we studied the effect of single augmentation functions for each modality. To do so, for both PPG and ECG, we kept each of the augmentation functions in isolation separately (with probability 1), and repeated our pre-training framework from scratch. We observed that PPG pre-training was more sensitive to the choice of single augmentation functions (more variance in performance), and channel permute and cut out were the most effective augmentation functions in isolation for PPG and ECG, respectively (Appendix Fig. 7). Future work can study more optimal ways to design modality-specific augmentation modules*”.

Reviewer comment (Reproducibility):
There are very limited details about the availability of the dataset. The reviewer questions whether it should be assumed private and whether there are any public datasets that could be used to replicate (some of) the results of the paper, requesting discussion around these points.

Author response:
We thank the reviewer about their thoughtful comment. Given the sensitivity of health data and that such data is collected under informed consent from participants, we cannot release the data to general public. Also, we are not aware of any public dataset that contains PPG/ECG recorded from wearable devices at this scale, which is the cornerstone of our manuscript, results and evaluations, that we respectfully believe is valuable for the researchers. However, to guide the researchers on how to source the data and the code, we have now added a **new Reproducibility statement**: “*the aggregated data that support the findings of this study can be made available on request from the corresponding author. Request for data will be evaluated and responded to in a manner consistent with the specific language in the study protocol and informed consent form. Similarly, code for all data analyses may be available upon request from the corresponding author. Requests for code will be evaluated and responded to in a manner consistent with policies intended to protect participant confidentiality and language in the study protocol and informed consent form*”.

Reviewer comment (Experiments):
Given the number of participants, the paper could attempt to increase the sequence length of the signals and assess whether longitudinal/day-level temporal dynamics can impact downstream tasks.

Author response:
We thank the reviewer for this comment. While this is a good suggestion, we respectfully believe this is beyond the scope of our current manuscript because the PPG/ECG data in our dataset is not sampled regularly and this is not immediately testable; PPG is collected intermittently and passively in Apple Watch and ECG is collected when initiated by the participants, as explained in 4.1. However, we think exploring longitudinal variations in PPG/ECG embeddings is a valuable future direction so we have now mentioned in the **updated Discussion section**: "*Future work can also investigate the longitudinal changes in PPG and ECG embeddings, and whether accounting for those can improve downstream predictions*".

Reviewer comment (Evaluation):
The linear probing results and performance analysis should be put in context to previous works like [4], [2], and [5].

Author response:
This is a great point and we thank the reviewer for pointing us to the related work. We have now added a detailed discussion regarding this in the **new Appendix section A.4** and mentioned that in the main Discussion: “*Prior work have looked into predicting certain demographic variables via representation learning from biosignals: 1) Using activity, sleep, and heart rate, BMI was predicted with ROC AUC of 0.697 (with the binary class cut-off at 30* $\text{kg}/\text{m}^2$ *similar to us), which is significantly smaller than that with our PPG/ECG embeddings in Table 2. Also, age was predicted with ROC AUC of 0.701 (with the binary class cut-off at 31 years) which is not directly comparable to our evaluation due to different cut-offs (Hallgrimsson et al., 2018). 2) A prior work pre-trained models using wearable accelerometer and heart rate data in free-living conditions (Spathis et al., 2021), and predicted age, BMI, and biological sex with ROC AUC of 0.676, 0.694, and 0.934, respectively, where the binary class cut-off was set to the median for continuous variables such as age and BMI. While the biological sex prediction is more accurately predicted with PPG and ECG embeddings in Table 2, the age and BMI predictions are not directly comparable due to different factors including different cut-off for binary labels and different dataset distribution, e.g., the distribution of the age in AHMS (Fig. 5) is different from the dataset used in this prior work (age ranging from 35 to 65). 3) Using the heart rate time-series data, biological sex was predicted with 0.703 and 0.672 Micro and Macro F1 score (Wu et al., 2020) - these reported numbers are lower than that for PPG embeddings (0.973 and 0.969, respectively) and for ECG embeddings (0.892 and 0.869, respectively). This prior work also reports age prediction which is not directly comparable to us due to different class buckets*”.

Reviewer comment (Experiments):
The parameter size of the final model seems quite low considering the dataset size and number of participants. The paper could justify the word "foundation model" in its title by scaling up the experiments. For example, it would have been very exciting to find the Chinchilla-optimal parameter size for this sort of data. The reviewer would appreciate discussion around whether there are overfitting issues with bigger models and considerations of different architectures like Transformers.

Author response:
We thank the reviewer for pointing this out. To better address the reviewer’s comment, we did a **new Ablation 5.2.4** and **new Table 4** by comparing the performance of our default 1D-EfficientNet model with our variation of 1D-ResNet and 1D-ViT to enable comparison with alternative model architectures and model sizes: “*For comparisons, we used smooth effective rank as a crude proxy of general downstream performance, and observed that different encoder architectures, with different model sizes, can achieve relatively similar performance, demonstrating that the performance is not unique to the 1D-EfficientNet encoder architecture (Table 4). We observed that 1D-EfficientNet model yielded similar performance to these alternative encoders with significantly smaller number of parameters, which was one of the main reasons we picked 1D-EfficientNet as our default backbone encoder given less memory footprint when potentially running on a wearable device”. We respectfully believe further analyses on this front is beyond the scope of our current manuscript, and therefore we have framed it as a future direction: “An interesting future direction is scaling up the model size further particularly with transformer-based models (e.g., 1D-ViT given their scalability (Kaplan et al., 2020), and larger datasets to study its effect on downstream performance*".