Paper: M-BioBERTa: Modular RoBERTa-based Model for Biobank-scale Unified Representations (ICLR-2024)

Reviewer comment (Presentation):
Presentation is unclear. Many important details are missing.

Author response:
1. and 2. We have tried to enhance the clarity of our paper by rephrasing critical sections. Additionally, we have included comprehensive explanations in the appendix, detailing pivotal aspects like tokenization, ablations, and feature extraction processes.

Reviewer comment (Evaluation):
The evaluation is unclear. Lots of experiments are needed to justify the model design.

Author response:
1. and 2. We have tried to enhance the clarity of our paper by rephrasing critical sections. Additionally, we have included comprehensive explanations in the appendix, detailing pivotal aspects like tokenization, ablations, and feature extraction processes.

Reviewer comment (Experiments):
Why Roberta model? It would be great to justify it.

Author response:
1. We selected a BERT-based model due to existing research that has demonstrated its effectiveness. Investigating alternative transformer models as baselines could be a valuable direction for future research.

Reviewer comment (Experiments):
How exactly is pre-training conducted? The authors mention that masked modeling is used. But here which part is masked? Since there are multiple modalities, is it modality-specific masking? Or others? Have the authors experimented with any varieties of masking strategies?

Author response:
2. We have updated the appendix with a new section to elucidate the Masked Language Modeling (MLM) objectives, which were previously unclear. This section explains the dual application of the MLM objective: firstly, in pretraining the encoder for each modality with modality-specific token sequences, and secondly, in jointly training the four encoders with the decoder. In this latter phase, masking is exclusively applied to the decoder’s sequence, and correspondingly, the calculation of the MLM loss is confined to the decoder's output.

Reviewer comment (Writing):
"The genomics data, sourced from SNPs, was encoded to display only the minor and major allele in the sequence." What does this mean? Every sample has both minor and major alleles? That does not make sense?

Author response:
3. We have corrected the error concerning the SNPs in the tokenization section in our paper, thanks to your observation, and have updated the respective paragraph to ensure its accuracy.

Reviewer comment (Reproducibility):
It would be great for the authors to give an exact example of what the input looks like.

Author response:
4. Recognizing the unconventional nature of our tokenization approach, we have expanded the tokenization section in the appendix with a detailed example to help better comprehension.

Reviewer comment (Experiments):
Is there ablations on which modality is most useful? For example, is genetics useful at all since some works have shown that EHR information overrides with genetics signals.

Author response:
5. An "Ablations" section has been added to the appendix, presenting an in-depth exploration of how key components influence model performance.

Reviewer comment (Experiments):
Why are only 700 SNPs used? There are 800K genotyped SNPs, if not considering imputed arrays. These could be used for pre-training.

Author response:
6. The choice of SNPs was based on their confirmed relation to Major Depressive: The choice of SNPs was motivated by their already established role to MDD, as our current model is not intended for SNP discovery at GWAS scale, such as linear mixed models, but to support cross-modal fusion and interpretation. We acknowledge the limitations imposed by computational constraints on the number of SNPs included. The ablations section offers insights into the minimal contribution of the genetic modality to overall performance. Future research could focus on better incorporating this genetic modality to enhance its effectiveness.

Reviewer comment (Experiments):
The authors used MMD disease as the prediction task. The data curation is also based on this task, which is limited.

Author response:
7. We apologize for any confusion and have clarified that only the genetics profile is specifically curated for MDD, not the broader UK BioBank data.

Reviewer comment (Presentation):
What are clusters in the Figure 2 and 3? There are little information with the current set of labels. Could the authors change to other more meaningful labels?

Author response:
9. We have updated the legends and captions of relevant figures to enhance their clarity and comprehensiveness.

Reviewer comment (Writing):
What is "Mean Absolute Percentage Error"?

Author response:
10. We have revised the caption of 'Table 5: Performance Metrics for Regression Tasks' to include the definition of the MAPE acronym, standing for Mean Absolute Percentage Error. In our paper, we utilize MAPE as a standard metric for regression tasks. To briefly explain MAPE here, it measures the absolute error as a percentage of actual values.

Reviewer comment (Evaluation):
Why do AUROC and AUPRC differ so much? For example in Table 4, E10-14, XGB has 0.859 the best AUROC but 0.162 worst AUPRC?

Author response:
11. The use of different metrics, such as AUPRC (Area Under the Precision Recall Curve), is particularly relevant in our case due to the heavily imbalanced nature of our datasets (as detailed in "Table 3: Statistics of Datasets for fine-tuning"). AUPRC is a more suitable metric for performance evaluation in such scenarios; see Saito, Takaya, and Marc Rehmsmeier. "The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets." PloS one 10.3 (2015): e0118432.

Reviewer comment (Evaluation):
There is a limited discussion on the scaling behavior of the proposed model architecture.

Author response:
In response to your observation about model size and its impact on performance, we have incorporated an analysis in the appendix. This new section delves into the correlation between model size and final performance metrics, supplying a deeper understanding of our model's scalability and efficiency.

Reviewer comment (Experiments):
The paper does not include ablation studies to directly demonstrate the benefits of key components like the temporal embeddings.

Author response:
Recognizing the value of component analysis in our model, we have introduced an "Ablations" section in the appendix. This section examines the influence of various components, notably the temporal embedding and diverse modalities, on the overall results.

Reviewer comment (Experiments):
Other transformer baselines are not considered for several downstream benchmarks.

Author response:
The primary objective of employing the RoBERTa baseline was to assess the impact of the proposed alterations on predictive efficacy. A further research direction of considerable interest involves examining the performance of alternative transformer models and investigating the implications of applying comparable alterations to these models.

Reviewer comment (Experiments):
How sensitive is M-BioBERTa to the choice of pretraining objectives beyond MLM? Could other self-supervised tasks (T5, UL2) further improve the representations?

Author response:
Your question about the exploration of different pretraining objectives is indeed intriguing. While this aspect was not covered in our current research, we acknowledge its potential significance and consider it a promising avenue for future research.

Reviewer comment (Experiments):
What is the impact of pretraining the encoders separately versus jointly pretraining the full model?

Author response:
We regret any confusion caused by the presentation of our models in the paper. To clarify, in "Table 2: Model Performance Comparison Based on ICD Accuracy," we compare three distinct models: "M-BioBERTa" (the jointly trained model), "pM-BioBERTa" (employing pretrained encoders), and "fpM-BioBERTa" (which freezes encoder weights, training only the decoder). We have revised the relevant section of the paper to elucidate this comparison more effectively, ensuring a clearer understanding of the distinct methodologies and their respective impacts on model performance.

Reviewer comment (Experiments):
The main drawback is around the deleted baselines. Apart from XGB the selected baselines are rather weak. Even for the selected baselines, the feature selection and processing should be discussed in more details.

Author response:
We appreciate your emphasis on the necessity for baseline models. Following your guidance, we have incorporated the missing baseline models pertinent to both regression and classification tasks. It's important to note that the inclusion of these models did not alter the previously reported best metrics results. We intentionally chose baseline models that are commonly used in healthcare settings, but we recognize the possibility of including additional models in subsequent studies. To enhance transparency, we have appended a detailed explanation of our feature selection method for these baseline models in the appendix.

Reviewer comment (Writing):
The method description is rather convoluted and difficult to follow. The main claim of the architecture is rather muddied and difficult to review for importance.

Author response:
To clarify our methodology, we have revised the relevant sections to ensure a more coherent and accessible presentation of our methods. We have also incorporated a concise statement of goals in the introduction for clearer understanding.

Reviewer comment (Experiments):
The modalities while discussed in the beginning is under-analyzed in terms of their contribution for predictive/modeling performance. The authors should consider ablation of modalities.

Author response:
Acknowledging the importance of understanding the individual contributions of different modalities, we have included an "Ablations" section in the appendix. This section investigates the impact of each modality in two distinct scenarios: firstly, when a single modality supplements the disease sequences, and secondly, when one modality is excluded from the final fpM-BioBERTa architecture.

Reviewer comment (Experiments):
For temporal embeddings, have the authors considered standards methods such as cosine embeddings?

Author response:
We regret any ambiguity in our first explanation about embedding methods and revised the relevant paragraph in the temporal embedding section. To clarify, our approach integrates all embedding methodologies of RoBERTa, including token type embedding and learned positional embedding, in conjunction with temporal embedding. We have opted for learned positional embedding instead of cosine embedding, aligning with current trends in the field.