Paper: Rethinking the bert-like pretraining for dna sequences (ICLR-2024)

Reviewer comment (Writing):
In the description of Table 2, what are MCC and PCC? The reviewer is aware that they are later explained in Experiment section, but what they are and what they do need to be briefly explained in the description of Table 2 as well for the readers who are not in life sciences field.

Author response:
**Q1**: In the description of Table 2, what are MCC and PCC? The reviewer is aware that they are later explained in Experiemnt section, but what they are and what they do need to be briefly explained in the description of Table 2 as well for the readers who are not in life sciences field.

**A1**: I have put the following content into the description of Table 2 or Supplementary Material.

Matthews Correlation Coefficient (MCC) and Pearson Correlation Coefficient (PCC) are both statistical measures used to assess the relationship between variables. The maximum values of MCC and PCC are both 100%. The larger these two indicators are, the better the model performance is.

1. **Matthews Correlation Coefficient (MCC)**: MCC is primarily used in binary classification problems to measure the performance of binary classification models. The formula for MCC is:
   $$
   MCC = \frac{TP \times TN - FP \times FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}
   $$
   
   where:
   -  TP  = Number of True Positives
   -  TN  = Number of True Negatives
   -  FP  = Number of False Positives
   -  FN  = Number of False Negatives

   True Positives are instances where the model predicts the sample as positive, and it is indeed positive. True Negatives are instances where the model predicts the sample as negative, and it is indeed negative. False Positives are instances where the model predicts the sample as positive, but it is actually negative. False Negatives are instances where the model predicts the sample as negative, but it is actually positive.

2. **Pearson Correlation Coefficient (PCC)**: PCC is used to measure the strength and direction of the linear relationship between two continuous variables. Let $\hat{y_i}$ be the model's predicted value, and $y_i$ be the actual value of the sample. The formula for PCC is:
  $$
   PCC = \frac{\sum_{i=1}^n (\hat{y_i} - \bar{\hat{y}})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^n (\hat{y_i} - \bar{\hat{y}})^2 \sum_{i=1}^n (y_i - \bar{y})^2}}
  $$

   where:
   - $\hat{y_i}$ = Predicted value of the sample by the model
   - $y_i$ = Actual value of the sample
   - $\bar{\hat{y}}$ = Mean of the model's predicted values
   - $\bar{y}$ = Mean of the actual values

   EMP, TF-M, TF-H, PD, CPD, and SSP are classification tasks, so the evaluation metric for the model on these tasks is MCC. EAP is a regression task, therefore the evaluation metric for the model on this task is PCC.

Reviewer comment (Experiments):
The authors say they progressively expand the masking boundary to prevent easy learning. Shouldn't it be the other way around? The reviewer believes that the masking boundary should progressively contract. What is the point of showing the shortcut first and complicating the learning? The model would already learn the shortcut if the masking boundary progressively expanded. An additional experiment on this needs to be conducted.

Author response:
**Q2**: The authors say they progressively expand the masking boundary to prevent easy learning. Shouldn't it be the other way around? The reviewer believes that the masking boundary should progressively contract. What is the point of showing the shorcut first and complicating the learning? The model would already learn the shortcut if the masking boundary progressively expanded. An additional experiment on this needs to be conducted.

**A2**:

Curriculum learning was proposed  by Yoshua Bengio in 2009. It is a training strategy that mimics the human learning process by presenting the examples in an easy-to-difficult order. It has been shown to improve the performance and convergence stability of various Large language model. 

The paper in ACL2020 that explore curriculum learning for Large language Model, titled *Curriculum Learning for Natural Language Understanding*. This paper proposes a curriculum learning approach that reviews the training set in a crossed way to distinguish easy and hard examples, and arranges a curriculum for Large language Model on various NLU tasks. We wanted to increase the difficulty of the DNA sequence pre-training task and at the same time ensure the stability of the training, so we adopted a course learning solution.

In the following table, the best performance results are represented by **boldface**, and the second best performance results are represented by <u>underline marks</u>. We tried consecutive mask6, 12, 14, 28 and 40 tokens during the entire pre-training process. Finally, it was found that this RandomMask from mask 6 to mask 14 achieved the best performance.
| mask            | avg.  | EMP    | TF-M   | TF-H   | PD     | CPD    | SSP    | EAP    |
|-----------------|-------|--------|--------|--------|--------|--------|--------|--------|
| Mask 6 (DNABERT)| 70.73 | 51.81  | 60.40  | 64.10  | 93.05* | **71.88** | 85.44  | 68.43  |
| Mask 12         | 71.98* | 52.96* | 67.12* | 67.05  | 92.01  | 71.05* | 86.62* | 67.03  |
| Mask 14         | 71.80 | 51.44  | 66.86  | 66.60  | 92.59  | 70.67  | 85.31  | 69.16* |
| Mask 28         | 70.94 | 50.95  | 65.94  | 67.75* | 91.03  | 70.70  | 83.33  | 66.85  |
| Mask 40         | 69.94 | 49.93  | 66.19  | 67.54  | 90.21  | 70.01  | 82.35  | 63.37  |
| RandonMask      | **74.85** | **65.83** | **67.96** | **69.73** | **93.40** | 70.24  | **87.20** | **69.56** |

We further tested both options from easy to hard and from hard to easy. In the table below, we divide pre-training into two stages equally. Mask 6 to 12 means masking 6 tokens continuously in the first stage and masking 12 tokens continuously in the second stage. Mask 12 to 6 means masking 12 tokens continuously in the first stage and masking 6 tokens continuously in the second stage. It can be found that the former has better performance.
| strategy    | avg.  | EMP   | TF-M | TF-H | PD    | CPD   | SSP  | EAP  |
|-------------|-------|-------|------|------|-------|-------|------|------|
| Mask 6 to 12| **73.06** | **59.22** | **67.83** | **66.37** | **92.81** | 71.51 | **86.20** | **67.38** |
| Mask 12 to 6| 70.64 | 53.15 | 63.47 | 65.18 | 90.42 | **71.93** | 83.95 | 66.37 |

Reviewer comment (Experiments):
[Clarity] The experiment and result section is relatively short. Some more clarifications would be helpful. For example, (a) when describing the baseline, the author mentioned “All models are trained on human genome and fine-tuned on the benchmark datasets with identical settings.” The “identical” setting does not bring clarity on how the mask is generated. Does this mean all the algorithms are trained and fine tuned with 15% token masked? This setting is slightly different than the original setting in the “Nucleotide Transformer” paper.

Author response:
**A to Clarity**: We will additionally add detailed settings to the revised paper and open source our code.

a) "Pretraining": For Nucleotide Transformer, hyenaDNA, and DNABERT, we used the open source pre-training weights from huggingface. When we pretrain DNABERT using RandomMask (DNABERT+RandomMask), we just modified the strategy for masking with RandomMask. The Other settings, such as the model structure, training steps, and learning rate, remained consistent with the settings of DNABERT's paper. We have made changes to the introduction of pretraining in the experimental section.

b) "Finetuning": In the fine-tuning stage, the fine-tuning parameters for NT are detailed in the A.7 of supplementary material's final section. The fine-tuning parameters for DNABERT and DNABERT+RandomMask are identical and their specific values can be found in the A.7 of supplementary materials. HyenaDNA used the fine-tuning parameters, being consistent with HyenaDNA's paper, can also be found in the A.7 of supplementary materials.

Reviewer comment (Reproducibility):
[Clarity] “Finetuning” section discussed the dataset in a detailed way; however, the details of the algorithm setup in the fine tuning is not discussed.

Author response:
**A to Clarity**: We will additionally add detailed settings to the revised paper and open source our code.

a) "Pretraining": For Nucleotide Transformer, hyenaDNA, and DNABERT, we used the open source pre-training weights from huggingface. When we pretrain DNABERT using RandomMask (DNABERT+RandomMask), we just modified the strategy for masking with RandomMask. The Other settings, such as the model structure, training steps, and learning rate, remained consistent with the settings of DNABERT's paper. We have made changes to the introduction of pretraining in the experimental section.

b) "Finetuning": In the fine-tuning stage, the fine-tuning parameters for NT are detailed in the A.7 of supplementary material's final section. The fine-tuning parameters for DNABERT and DNABERT+RandomMask are identical and their specific values can be found in the A.7 of supplementary materials. HyenaDNA used the fine-tuning parameters, being consistent with HyenaDNA's paper, can also be found in the A.7 of supplementary materials.

Reviewer comment (Novelty):
The authors might consider refining the focus of this work. The true contribution appears to be the improvement of the overlapping strategy tokenization for DNA pretraining, which diverges from the broader theme of "rethinking the pretraining for DNA sequence."

Author response:
We named our paper as "Rethinking the BERT-like Pretraining for DNA Sequences" based on the following two points: 

1) The primary influences on BERT-like pretraining methods are the **tokenizer** and **mask**. Our paper discusses both of these key factors: 1) We discuss the impacts of DNA **tokenizer** in both pretraining and fine-tuning stages, as shown in Figure 1, Figure 2, Table 2 of manuscript and Table 1 of Supplementary Material. In Figure 1 of manuscript, we show the comparisons between commonly used BERT-like DNA tokenizer and NLP tokenizer. In Figure 2 of manuscript, we show the impact of overlapping and non-overlapping tokenizer on the pre-training process and the underlying representation of the model. In Table 2 of manuscript, we show the influence of overlapping and non-overlapping tokenizer on the fine-tuning stage. In Table 1 of the supplementary material, we also compare the performance of 6mer with BPE, with the support of RandomMask, the performance of 6mer is much higher than the most advanced BPE model. 2) We investigate the methods of **mask** in BERT-like pretraining. In Figure 1, we compare different mask approaches in DNA and NLP. In Table 4 of the supplementary material, we discussed the impact of the number of consecutive mask tokens on performance. In Table 5 of the supplementary material, we discussed pre-training mask task from easy to difficult and from difficult to easy. Based on these ablation experiments, we found the optimal setting strategy for RandomMask.
3) Our contribution includes **three experimental observations** related to BERT-like DNA pretraining and **a simple yet effective method**. Reviewer SHyo acknowledged the inspirational significance of these progressive observations, while Reviewer 45WA affirmed that our novel insights differ from traditional views, offering valuable insights for numerous related studies and applications. Based on these observations, our article introduces a simple yet effective method called RandomMask. Therefore, our contribution is not just in optimizing the tokenizer, but in providing three new observations regarding BERT-like DNA pretraining and presenting an effective method based on these insights.

Reviewer comment (Theory):
The motivation behind the study is somewhat unclear. Although the authors identify three potential challenges -- rapid convergence, the risk of under-training, and the potential for sparse attention -- they do not adequately explain how RandomMask addresses or mitigates these issues.

Author response:
This study gives a comprehensive understanding and a specifically tailored approach for bert-like DNA pretraining. Based on a series of exploratory experiments, three insightful observations are given, followed by a simple-but-effective method RandomMask for solving the rapid convergence and under-training problems. Below, we give more experimental analysis supporting the observed improvements of RandomMask:

1) In Figure 3 of Supplementary Material, we can see that DNABERT's loss will quickly decrease to an extremely low value. If RandomMask is used, the loss will increase at the start of each stage, giving it enough space to decrease, so we will see a decrease at each stage in the loss curve with RandomMask. Reviewer 45WA affirmed this point. He emphasized that we enhanced the generalization of the model by increasing the difficulty of the task and reducing the overfitting problem caused by rapid convergence that gradually increasing the complexity during the training could mitigate the overfitting issue and achieve better performance by combining fast convergence and generalizability.
2) In the Figure 1 of Supplementary Material, we can see that the middle layer only pays attention to the [CLS] token, while the first and last layers capture a variety of features. Further, in Figure 2 of Supplementary Material, we see that all tokens in the DNABERT's middle layer pay excessive attention to [CLS]. Therefore the middle layer is not sufficiently trained to capture sequence features. However, as showed in Figure 2 of Supplementary Material, if pre-training with RandomMask, these layers of the model are activated. The attention weights of these layers learn to pay attention to other tokens, and the attention patterns are more diverse. Diverse attention weights indicate that these layers are mroe adequately trained to capture sequence features as the first and last layers do.
3) What RandomMask wants to solve is the fixed attention weight pattern of the middle layers in DNABERT. For sparse attention, different tokens and different layers usually show diverse sparse patterns. However, in Figure 1 and 2 of Supplementary Material, we see that the different layers in the middle of DNABERT and all tokens show the same attention weight pattern, that is, concentrated attention [CLS]. Thus, the problem we're trying to solve and sparse attention are two different things. In Figure 2 of Supplementary Material, we see that adding RandomMask will diversify the attention weight pattern of layers in the middle of DNABERT.

Reviewer comment (Experiments):
There is a lack of experimental analysis supporting the source of the observed improvements, which is crucial for substantiating the paper's main claims. For example, besides the quantitative improvements, does the rapid convergence and under-training still exist after applying RandomMask?

Author response:
This study gives a comprehensive understanding and a specifically tailored approach for bert-like DNA pretraining. Based on a series of exploratory experiments, three insightful observations are given, followed by a simple-but-effective method RandomMask for solving the rapid convergence and under-training problems. Below, we give more experimental analysis supporting the observed improvements of RandomMask:

1) In Figure 3 of Supplementary Material, we can see that DNABERT's loss will quickly decrease to an extremely low value. If RandomMask is used, the loss will increase at the start of each stage, giving it enough space to decrease, so we will see a decrease at each stage in the loss curve with RandomMask. Reviewer 45WA affirmed this point. He emphasized that we enhanced the generalization of the model by increasing the difficulty of the task and reducing the overfitting problem caused by rapid convergence that gradually increasing the complexity during the training could mitigate the overfitting issue and achieve better performance by combining fast convergence and generalizability.
2) In the Figure 1 of Supplementary Material, we can see that the middle layer only pays attention to the [CLS] token, while the first and last layers capture a variety of features. Further, in Figure 2 of Supplementary Material, we see that all tokens in the DNABERT's middle layer pay excessive attention to [CLS]. Therefore the middle layer is not sufficiently trained to capture sequence features. However, as showed in Figure 2 of Supplementary Material, if pre-training with RandomMask, these layers of the model are activated. The attention weights of these layers learn to pay attention to other tokens, and the attention patterns are more diverse. Diverse attention weights indicate that these layers are mroe adequately trained to capture sequence features as the first and last layers do.
3) What RandomMask wants to solve is the fixed attention weight pattern of the middle layers in DNABERT. For sparse attention, different tokens and different layers usually show diverse sparse patterns. However, in Figure 1 and 2 of Supplementary Material, we see that the different layers in the middle of DNABERT and all tokens show the same attention weight pattern, that is, concentrated attention [CLS]. Thus, the problem we're trying to solve and sparse attention are two different things. In Figure 2 of Supplementary Material, we see that adding RandomMask will diversify the attention weight pattern of layers in the middle of DNABERT.

Reviewer comment (Experiments):
The comparison in Observation 1 does not seem to be an apples-to-apples comparison. Overlapping represents more patterns and creates longer sequences for the same DNA length. It would be beneficial to understand if the conclusion holds for different lengths of k-mer.

Author response:
Follow the reviewer's suggestions, we investigate the effect of sequence length. The following table is an expansion of Table 2 in the paper. The row named Same-lenth is to copy the Non-overlapping tokens five times to obtain the same sequence length as Overlapping.
(i.e. Non-overlapping: token1 token2; Same-lenth: token1 token2 token1 token2 token1 token2 token1 token2 token1 token2 token1 token2.)

The best performance results are represented by **boldface**, and the second best performance results are represented by *.
| NT              |    avg. |   EMP |   TF-M |   TF-H |    PD |   CPD |   SSP |   EAP |
|----------------|--------|------|-------|-------|------|------|------|------|
| Non-overlapping | 58.68 | 45.37* | 39.81 | 55.25 | 88.43* | 62.56 | 80.39 | 38.94* |
| Same-lenth      | 60.42* | 44.88 | 47.59* | 60.57* | 86.96 | 63.98* | 80.96* | 37.98 |
| Overlapping     | **68.69** | **46.47** | **61.99** | **63.95** | **90.88** | **68.55** | **84.34** | **64.67** |

| DNABERT         |    avg. |   EMP |   TF_M |   TF_H |    PD |   CPD |   SSP |   EAP |
|----------------|--------|------|-------|-------|------|------|------|------|
| Non-overlapping | 60.24*   | 43.65* | 34.87  | 54.50*  | 87.62* | 65.82* | 79.91 | 55.31* |
| Same-lenth      | 57.66 | 42.98 | 38.60*  | 53.27  | 85.33 | 64.09 | 80.76* | 38.62 |
| Overlapping     | **70.49**   | **51.81** | **59.60**  | **63.55**  | **91.76** | **72.84** | **85.44** | **68.43** |

1) An interesting phenomenon. In NT that uses non-overlapping 6mer for pre-training, stretching the sequence length will indeed produce obvious gains in TF-M, TF-H and CPD. Combined with Table 1 in the paper, the common feature of these three tasks is that the DNA sequence length is short. The DNA sequence lengths of EMP, PD, SSP and EAP are 500, 300, 400 and 250 nucleotides respectively. However, the DNA sequence lengths of TF-M, TF-H and CPD are 100, 100 and 70 nucleotides respectively, and these are shorter than others.

2) But in general, the overlapping tokenizer to obtain more diverse tokens get better performance than simply lengthening the sequence length (Same-lenth) both of overlapping pretraining (DNABERT) model and non-overlapping pretraining model (NT).

Reviewer comment (Writing):
The introduction is somewhat verbose, indirectly causing the first two weaknesses and making the paper hard to read.

Author response:
Thanks for your valuable suggestions, we have considered all the suggestions and improved the paper's presentation as much as we can. Besides, we will ask senior researchers who have published more than ten top conference papers to revise the manuscript to ensure that the article is concise and easy to read.

Reviewer comment (Presentation):
Placing Figure 1 and Table 1 on page 1 would improve readability, given that the main content describing Figure 1 and Table 1 is in the first page.

Author response:
Thanks for your valuable suggestions, we have considered all the suggestions and improved the paper's presentation as much as we can. Besides, we will ask senior researchers who have published more than ten top conference papers to revise the manuscript to ensure that the article is concise and easy to read.

Reviewer comment (Reproducibility):
The experimental settings in Section 3 lack detailed descriptions, potentially making reproduction difficult and potentially misleading.

Author response:
Thanks for your valuable suggestions, we have considered all the suggestions and improved the paper's presentation as much as we can. Besides, we will ask senior researchers who have published more than ten top conference papers to revise the manuscript to ensure that the article is concise and easy to read.

Reviewer comment (Writing):
A thorough proofreading could enhance the clarity of writing and word choice.

Author response:
Thanks for your valuable suggestions, we have considered all the suggestions and improved the paper's presentation as much as we can. Besides, we will ask senior researchers who have published more than ten top conference papers to revise the manuscript to ensure that the article is concise and easy to read.

Reviewer comment (Evaluation):
How does the RandomMask improve the internal representations - Can you see visible differences in embedding representations for downstream tasks (e.g Biotype Embeddings shown in HyenaDNA)?

Author response:
**Q2**: How does the RandomMask improve the internal representations - Can you see visible differences in embedding representations for downstream tasks (e.g Biotype Embeddings shown in HyenaDNA) 

**A2**: Since HyenaDNA does not open source the code and biotype dataset for this task. Therefore we tried it on our dataset.

Following the motivation of biotype, we selected EMP-Yeast, TF-Human, TF-Mouse and Prom-Human, covering three species and three functional originals. We used DNABERT and DNABERT + RandomMask to generate embeddings for these datasets, and performed t-SNE for visualization. We put the visualization results in A.8 of the Supplementary Material. 

We can get the following conclusion. Compared with the original DNABERT, the model pre-trained using RandomMask can produce clearer boundaries and the separation between different species is more obvious.

Reviewer comment (Experiments):
How does RandomMask compare on the benchmark datasets listed in HyenaDNA?

Author response:
**Q3**: How does RandomMask compare on the benchmark datasets listed in HyenaDNA?

**A3**: Follow the reviewer’s suggestion, we compare RandomMask on the benchmark datasets listed in HyenaDNA:

(1) 18 of the 25 downstream task datasets open sourced by HyenaDNA are already included in the GUE benchmark we used (including EMP, TF-M, PD, CPD, SSP and EAP). These results are showed in the Table 3 of the manuscript. The results of the remaining 7 datasets are as follows.

(2) In the table below, the first row shows the results of CNN on the 7 datasets. The second row is the result of two layers of Transformers. The third row is the result of two layers of Hyena. These three rows of results are all from the original paper of HyenaDNA. We also test open source DNABERT and our DNABERT + RM (RandomMask), and show the results in the fourth and fifth rows respectively. Because the results in the original HyenaDNA's paper are accurate to one decimal place, our test results in this table are also kept to one decimal place. The best performance results are represented by **boldface**, and the second best performance results are represented by *.
|                       | Avg.       | Coding vs Intergenomic | Human vs Worm | Human Enhancers Cohn | Human Enhancers Ensembl | Human OCR Ensembl | Human Regulatory | Human Nontata Promoters |
|------------------------|--------------|------------------------|---------------|----------------------|-------------------------|-------------------|------------------|------------------------|
| CNN                   | 80.7         | 87.6                   | 93.0          | 69.5                 | 68.9                    | 68.0              | <u>93.3</u>     | 84.6                   |
| Transformer           | 84.4        | 88.8                   | 95.6          | 70.5                 | 83.5                    | 73.0              | 91.5            | 87.7                   |
| HyenaDNA              | 88.3* | 87.6                   | 96.5          | 73.8                 | 89.2*             | 80.9              | **93.8**        | **96.6**               |
| DNABETR              | 87.8        | 93.2*           | **97.0**      | 74.3*          | 89.0                    | 81.0*       | 88.5            | 91.7                   |
| DNABERT + RM            | **89.4**    | **94.5**               | 96.9*   | **76.6**             | **91.3**                | **82.7**          | 90.3            | 93.4*            |

Comparing the last two rows, we can find that except Human vs Worm, DNABERT+RM is always better than DNABERT. And the average performance of DNABERT+RM far exceeds DNABERT. This proves the superiority of RandomMask. Compared to other methods, DNABERT+RM achieves the best performance on four of the seven datasets. DNABERT+RM achieve second-best performance on two of the three datasets where performance was not the best. And DNABERT+RM are the best in terms of average performance across seven datasets.