From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Thank you for your detailed and constructive feedback
comment: Dear Reviewer Anyp,

Thank you for taking the time to provide a detailed post-rebuttal update justification. We now clearly understand your concerns regarding novelty and empirical analysis, which will greatly help to improve our work.

However, we still believe that our work does provide _" interesting machine learning/genomics insight and show results in toys dataset"_ and has merit for acceptance because:

 This is the **first work** that explores and evaluates the effectiveness of **PMI-guided MLM** (from a methods perspective) and **DNABert-like models** (from a genomics perspective) in a **few-shot setting**. No prior works in sequences (both for language and genomics) explored the **few-shot setting**.

We once again **thank you for your constructive feedback**, as it contributed to significantly improving the paper.
 
Best regards,

Authors of Paper 4658

From: ICLR.cc/2023/Conference/Paper4658/Reviewer_Anyp
title: Extra post-rebuttal update justification
comment: I thank the authors for their reply.  I acknowledge that the extra experiments make the results a bit more reliable (specially the 10-run on few-shot evaluation), but that was not enough to change my perspective.

Here is the summary of my justification:
- Novelty: As mentioned before (also by other reviewers), this paper—similar to DNABert—has very limited novelty wrt any machine/representation learning contribution. The proposed method is an adaptation (by adding some tricks) of a previous method applied to one type of sequence (genomics) rather than another (language). Moreover, we already know that Bert-style objective work(-ish) in genomics sequence, so a PMI-version of Bert would probably work as well.
- That would not be problem if the experimental results would be interesting and meaningful in practice. However, I don’t think that is the case. The tasks/datasets used (Prom-core, Prom-300 and Conh-enh) are not relevant in practice for genomics/biology. These datasets are very toy-ish and very disconnected from actual genomics reality. A method that perform slightly better on these very unrealistic toy tasks will unlikely be very useful in any genomics/biology application. 
- I would be inclined to accept this paper if either: (i) it has an interesting machine learning/genomics insight and show results in toys dataset, OR  (ii)  the method is an adaptation from NLP to genomics but have considerable good/useful results in relevant tasks.

- Extra comments:
	- Another issue with the evaluation is that most of the contributions are on an artificial hold-out set (which we already have the data of). As I mentioned, those kinds of models are much more valuable if they can provide any new biological insight (eg, understand effect of mutations on the genome). Section 6.3 (“Motif Analysis”) very briefly covers this topic, but it is only a small part of the results.
	- Transcriptional regulation (in which promoters/enhancers/silencers play a role at) rely on very complicated mechanisms and require interaction of multiple regulatory elements—requiring a very large sequence context that  DNABert (PMI-guided or not) cannot dream on achieving. Moreover, the relationship of those elements varies a lot with respect  to external variables (eg,  the cell type the data comes from). The datasets used on this paper ignore all of those complications and consider a very simple problem of binary classification (with an unrealistic data distribution).
	- I also disagree that “Enformer is just another competing baseline like DNABert/LOGO”. The modeling is fundamentally different as it takes a much larger input sequence (100-200kbp)—this is known to be important to transcriptional regulation tasks. The analysis provided by Enformer (and many other papers that are cited by them or cite them) is fundamentally different than those provided by eg DNABert. They show much more interesting results than a simple binary classification performance on toy tasks, predict effect of genetic variants in multiple MPRA datasets, perform many motif analysis, etc.

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: A gentle reminder as discussion deadline ends today
comment: Dear Reviewer 61nZ,

Thank you for your positive appreciation of the paper. We sincerely hope that we are able to address your concern about improving the reliability of empirical results. 

Since the **discussion deadline ends today, we sincerely request you to consider improving the paper evaluation score** if all your concerns are addressed.

1. We have now **updated Tables 1 and 2 with multiple-run results**. We report the mean and standard deviation of accuracy and AUC by running the few-shot experiments ten times by randomly choosing _t_ different training (fine-tuning) examples per class and a randomly generated seed in each run. We use the same set of ten randomly generated seeds for all the experiments to maintain fair comparison.

 2. We have also **added the results for DNABert trained to 120K steps in Table 1**. We use the pretrained weights the authors provided in their codebase at https://github.com/jerryji1993/DNABERT. We observe the effectiveness of PMI-guided models at 10K steps, which performs comparably with the DNABert model trained for 120K steps in the Prom-core setting. PMI-guided models outperform the ORI model in all few-shot settings of Prom-300 and Cohn-enh, except for the 500-shot setting of Cohn-enh, where it performs comparably.

It will be great if you could let us know whether there are any further questions about the revised paper or the response.

Thanks and regards,

Authors of Paper 4658

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: A gentle reminder as discussion deadline is ending 
comment: Dear Reviewer 61nZ,

Thank you for your positive appreciation of the paper. 

As the deadline for the discussion period ends in three days, we sincerely hope that we are able to address your concern about improving the reliability of empirical results.

1. We have now **updated Tables 1 and 2 with multiple-run results**. We report the mean and standard deviation of accuracy and AUC by running the few-shot experiments ten times by randomly choosing t different training (fine-tuning) examples per class and a randomly generated seed in each run. We use the same set of ten randomly generated seeds for all the experiments to maintain fair comparison.

2. We have also **added the results for DNABert trained to 120K steps in Table 1**. We use the pretrained weights the authors provided in their codebase at https://github.com/jerryji1993/DNABERT. We observe the effectiveness of PMI-guided models at 10K steps, which performs comparably with the DNABert model trained for 120K steps in the Prom-core setting. PMI-guided models outperform the ORI model in all few-shot settings of Prom-300 and Cohn-enh, except for the 500-shot setting of Cohn-enh, where it performs comparably.

It will be great if you could let us know whether there are any further questions about the revised paper or the response.

Based on the above-mentioned points, we **sincerely request you to consider improving the paper review score**. 

Thanks and regards,

Authors of Paper 4658

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Request to elaborate on "analysis not being convincing" of post-rebuttal update comment
comment: Dear Reviewer Anyp,

We thank you for your post-rebuttal feedback. In our response and the revised paper, we **completed all your requested experiments and answered point-wise each of your concerns mentioned in the Weaknesses part of your review**.

We also took **considerable effort to evaluate a fourth task of identifying important functional genetic variants**, which is also used by DNABert authors. This task allows us to discover significant variants. It is explained in detail in the "Relevance of biological justification" response under point 3.
The **results and detailed discussion is added in Figure 2 and Section A.3.1 of the Appendix of the revised paper**. 

It would be great if you could **kindly point out (or provide more details) the specific analysis** you considered not convincing.

Based on the above-mentioned points, we request you to **kindly consider an improvement in the evaluation score** of the paper.

Best regards,

Authors of Paper 4658

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Response to Post-rebuttal update comment regarding novelty
comment: Dear Reviewer Anyp,

We thank you for your post-rebuttal feedback. 

However, we strongly posit that **the statement "(PMI-guided MLM has already been applied in sequences)" is incorrect**. It is the first time PMI-guided MLM has been applied to gene sequences. The **baseline model, Fixed PMI-guided Masking Vocabulary (PMI-VOCAB), is also our own construction and is a contribution in itself**.

The PMI Masking paper by Levine et al. 2020 is limited to only natural language processing tasks and is **not applied to gene sequences**.

On top of that, we observe issues with the PMI-VOCAB model and propose a novel model that mitigates those issues. This is empirically evident as PMI-best outperforms PMI-VOCAB in all settings. We also highlight these points in the paper and the response. Therefore, we claim that the proposed PMI-guided MLM adaptation to the gene sequences domain is non-trivial and novel.

Therefore, please re-consider the statement you are using to claim limited novelty, i.e., PMI-guided MLM has already been applied in sequences. 

We would sincerely encourage you also to consider the reviewers QUAU and 61nZ (**both with a confidence of 4**) rated **3** on the "Technical Novelty And Significance," whereas **reviewer ARuE has a reduced confidence rating of 3**. We request you to also consider this point.

Best regards,

Authors of Paper 4658

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Looking forward for more discussion
comment: Dear Reviewer QUAU,

Thank you for your review. We sincerely hope our response is able to address your concern about the weaker reimplementation of DNABert and LOGO as compared to the original version.

As the deadline for the discussion period is approaching, we would appreciate it if you could let us know whether there are any further questions about the revised paper or the response.

Best regards,

Authors of Paper 4658

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Thank you for the feedback
comment: Dear Reviewer ARuE,

Thank you for your feedback and for taking the time to go through the response and the revised paper.

Best regards,

Authors of Paper 4658

From: ICLR.cc/2023/Conference/Paper4658/Reviewer_ARuE
title: Thanks for the response
comment: Thanks for the notes. I think the paper benefits from highlighting the points mentioned here. I am willing to increase my score by one; sorry that the score is not more encouraging. I still believe the significance of the proposed method is limited for acceptance to ICLR.

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Looking forward for more discussion
comment: Dear Reviewer ARuE,

Thank you for your review. We sincerely hope our response is able to address your concerns.

As the deadline for the discussion period is approaching, we would appreciate it if you could let us know whether there are any further questions about the revised paper or the response.

Best regards,

Authors of Paper 4658

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Looking forward for more discussion
comment: Dear Reviewer Anyp,

Thank you for your review. We sincerely hope our response is able to address your concerns.

As the deadline for the discussion period is approaching, we would appreciate it if you could let us know whether there are any further questions about the revised paper or the response.

Best regards,

Authors of Paper 4658

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Looking forward to more discussion
comment: Dear Reviewer 61nZ,

Thank you for your review. We hope our response is able to address your concern about improving the reliability of empirical results. 

As the deadline for the discussion period is approaching, we would appreciate it if you could let us know whether there are any further questions about the revised paper or the response.

Best regards,

Authors of Paper 4658

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Common Questions Asked
comment: **Novelty Concerns**

This has been addressed in detail in the comment to Reviewer ARuE. We found unique challenges in adapting a PMI-based algorithm to gene sequences, both in the definition of the PMI scores and in the implementation of the PMI token vocabulary and the PMI-based masking strategy for masked language models.

**Generalizability of Experimental Results**

We have now added detailed experiments such that

- Multiple runs (10) of the fine-tuning models have been reported, and gains have been reported in the few-shot approach. This was performed with multiple seeds and multiple runs as well.
- The original model, trained over 120k steps, has also been included for comparison.

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Summary of Rebuttal Revision
comment: **Summary**

In the updated version of the paper, we have added the following:

- Justification on why the addition of PMI masking is novel and non-trivial in its adaptation for gene sequences in the **Introduction** and **Methodology** section
- We have **completed all the experiments proposed by the reviewers**, where the general trend remains intact. We subsequently update the **Experiments** section of the paper, namely:
  - Added fully trained models (on 120k steps for DNABERT) for comparison (ORI 120K model)
  - Each result of Tables 1 and 2 is updated where we repeat the experiments **ten** times, each time with a different training data split and a random seed. We then reported the accuracy and the AUC metric's mean and standard deviation values.
  - Added analysis on a new task of identifying functional genetic variants, where we observe PMI Masking-guided DNABert performs better than the original DNABert (ORI) model in the 10-shot Prom-core setting. Please see Section A.3.1 and Figure 2 of the Appendix.
  - Added clarification details regarding experimental setup. Specifically, we provide strong empirical justification for considering the setting of evaluating pretrained model variants trained on 10K steps.
- **New insights**: In the case of **DNABert**, PMI-best outperforms all the baseline models for the task of Prom-300 and Cohn-enh, except for 1000-shot Prom-300 setting, which also includes the original DNABert model trained for 120K steps (ORI 120K model). ORI 120K model outperforms PMI-best on low data settings (10, 50, and 100-shot) of the Prom-core task, whereas PMI-best outperforms again in 500 and 1000-shot settings.

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Justification for choice of evaluation datasets and Relevance of biological justification
comment: **Relevance of Biological Justification**

1. This paper is the first step toward adapting PMI masking to the genomic setting. The evaluation pertaining to gene sequence classification tasks, we would like to point out that it is part of a standard evaluation setup used in the gene sequence modeling domain and has been used in many papers in this domain, such as LOGO, BigBird, and DNABERT.
2. The Enformer model is a competing baseline in the genome sequence modeling domain, just like DNABert and LOGO. However, there is no clear standard for utilizing the Enformer as a baseline model. Our proposed PMI masking strategy works for any simple, straightforward transformer architecture and should ideally also be adapted to the Enformer model.
3. We thank you for the interesting suggestion of looking into the evaluation methods used by the Enformer paper. We went through the paper and found the task of identifying novel variants from genomic tracks to be very interesting. To further address your comment, we now evaluate a fourth task of identifying important functional genetic variants, which is also used by DNABert authors. This task allows us to discover significant variants.
  - However, we do not have access to the specific code used by the authors to evaluate the importance of a given variant using Clinvar. We have written to the authors and are waiting to hear from them.
  - Therefore, we did the next best thing. The results and detailed discussion is added in Section A.3.1 of the Appendix. As depicted in Figure 2, in the 10-shot setting, we observe that the PMI-guided DNABert model consistently reports higher overlap in terms of important functional genetic variants over the original model (ORI) in different top-N settings (5000 \<= N \<= 50000). Thus, we conclude that PMI-masking helps incorporate intrinsic (or relevant) genomics information into ORI models like DNABert and LOGO. The performance improvement is wider than just gene sequence classification tasks. We will look to evaluate settings other than 10-shot for the sake of completeness.
4. Regarding the comment, "When training models in genomics (or other biological datasets), it is much more interesting if the model can show some insights about the biology surrounding what we want to model.", we would like to point out that it is already tackled in the paper to a certain extent in the Motif Analysis section (Section 6.3), under the heading "PMI-based rankings capture motifs present in finetuning datasets."
  - Quoted from the paper: We observe that most of the 6-mers that match the discovered motifs are ranked very high. Within the top 25 percentile of RANK, that is a rank of 1024. We further observe that our top-1 ranked 6-mer is present in motifs of both enhancers and promoters based on de-novo motif discovery and is also mentioned in previous biomedical literature (Chow et al., 1991). The TATA box, a well-known motif for promoters (rows 3 of Table 3) is ranked at 242; the best TATA box motif is TATATA, which has a PMI rank of 15.

[1] Dhillon, G. S., Chaudhari, P., Ravichandran, A., & Soatto, S. (2019, September). A Baseline for Few-Shot Image Classification. In International Conference on Learning Representations.

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Novelty of PMI Masking Adaptation Strategy and Improving the Soundness of Empirical Results 
comment: Thank you for your insightful comments and detailed suggestions.

**Novelty Concerns** : same comment as given to the first reviewer ARuE regarding novelty of PMI Masking adaptation strategy to genomic setting.

**Soundness of Empirical Results:**

1. For few-shot algorithms, it is common in the few-shot literature (Dhillon et al., 2020) [1] to perform training without using any validation set. We had mentioned the justification already in the paper in Section 5.2. We also provide a few more references that use a similar approach.
  - Quoting from the paper: We follow the standard evaluation setup used in the few-shot text classification setting (Schick & Schutze, 2021a;b). Thus, we assume that we do not have access to a validation dataset (development set) to optimize the hyperparameters.

2. We completely agree with your comment regarding the issue of using only one train/test data in a few-shot setting. Following your suggestion, we have now updated Tables 1 and 2 with multiple-run results. We report the mean and standard deviation of accuracy and AUC by running the experiments **ten** times by randomly choosing _t_ different training (fine-tuning) examples per class and a randomly generated seed in each run. We use the same set of ten randomly generated seeds for all the experiments to maintain fair comparison.

3. Thanks for the suggestion for comparing with the original model trained till convergence. We have now added the results for DNABert trained to 120K steps in Table 1, we use the pretrained weights that the authors have provided in their codebase at [https://github.com/jerryji1993/DNABERT](https://github.com/jerryji1993/DNABERT). We observe the effectiveness of PMI-guided models at 10K steps, which performs comparably with the DNABert model trained for 120K steps in the Prom-core setting. PMI-guided models outperform the ORI model in all few-shot settings of Prom-300 and Cohn-enh, except for the 500-shot setting of Cohn-enh, where it performs comparably.
  - Therefore, we suggest that our choice of model evaluation at 10K pretraining steps is in line with the convergence of perplexity as depicted in Figure 1 of the Appendix section. We have now provided results for the fully trained DNABert on 120K steps using the original authors' pretrained model weights provided by the codebase. We further observe empirically from the performance comparison of the original DNABert model trained for 120K steps versus the same model trained on 10K steps from Table 1 that: the ORI 10K model outperforms the ORI 120K model in 500 and 1000-shot settings. However, ORI 120K model outperforms ORI 10K model in lower data settings (10, 50, and 100-shot).

**Relevance of the Biological Justification**: We provide it as a separate comment in the same thread

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Novelty of PMI masking adaptation strategy to genomic setting and Improved empirical justification
comment: We thank you for your review.

We strongly contest the comment w.r.t novelty of PMI masking adaptation to genomic sequences. The very idea of introducing PMI in a genomic setting is novel and produces significantly improved results. However, we go beyond and propose an improved version:

1. The following points (quoting directly from the paper) mention the difference between PMI masking (Levine et al. 2021) and our proposed adaptation of PMI masking to gene sequences in the Introduction section:
  - However, the standard PMI formulation (Levine et al., 2021) sometimes favors tokens with a low frequency of occurrence. To adapt to the genomic setting, we modify the PMI metric to mitigate the above-mentioned issue.
  - The main difference from our work is that they use it to develop a large masking vocabulary in the NLP domain, while we use the PMI score as a ranking function to choose the span of tokens to mask.
2. The closest adaptation (non-trivial) of the PMI masking (Levine et al. 2021) paper is the PMI-VOCAB baseline model that we use in our experiments. Our proposed PMI masking method outperforms the PMI-VOCAB model in all few-shot settings w.r.t DNABert. PMI-VOCAB possesses the following two issues, which we directly mitigate through our proposed PMI Masking strategy:
  - **Determining the masking vocabulary size:** Levine et al. 2021 use a small-scale evaluation of an n-gram's collocation quality as a function of its $PMI_n$ rank. They create an ad hoc dataset composed of 1000 n-grams that they manually label as the collocated terms or not (for more details, please see Section A of the Appendix of Levine et al. paper). It is difficult to create such an ad-hoc dataset in the gene sequence modeling setting because: (i) gene sequences are continuous, and such semantically-meaning unit (like words or tokens in NLP) is not available in the genomic domain. (ii) The manual annotation process will require medical domain experts' involvement and, thus, a significant human effort.
  - The standard PMI formulation (Levine et al., 2021) **favors tokens with a low frequency of occurrence**. This significantly reduces the efficacy of PMI masking and leads to the wastage of PMI masking-based pretraining steps because, during the MLM training stage, such high PMI tokens will rarely appear as masked tokens (given their low frequency of occurrence).
3. We mitigate the above two issues in the following manner in our proposed PMI adaptation to gene sequences:
  - Instead of using an absolute importance notion of the PMI score to create a masking vocabulary, we use the PMI score for ranking purposes, where we measure the relative importance among DNA sequence k-mer tokens during MLM training (i.e., tokens with a higher PMI score have a higher chance of being masked). **[Added this sentence in the Introduction section]**
  - Add a discounting factor to the $PMI_n$ formulation of Levine et al. 2021, as shown in Equation 2, which penalizes rare tokens. We also would like to point out that the design of the PMI score is directly motivated by its usage during the MLM training stage and is not ad-hoc in nature.
4. We further observe the effectiveness of PMI-guided models at 10K steps and report comparable performance with the DNABert model trained for 120K steps in the Prom-core setting. PMI-guided models outperform in all few-shot settings of Prom-300 and Cohn-enh, except for the 500-shot setting of Cohn-enh, where it performs comparably.

We thus contest that the very fact of using PMI for genes is novel and non-trivial, and its usage shows direct improvement by doing just 10K pretraining steps.

Thank you for pointing out the inconsistency in the n-gram and k-mer definitions. We have now unified the terminology and consistently use k-mers for ease of understanding.

Concerning the organization of empirical studies, we have now updated Tables 1 and 2 with multiple-run results. We report the mean and standard deviation of accuracy and AUC by running the experiments **ten** times by randomly choosing _t_ different training (fine-tuning) examples per class and a randomly generated seed in each run. We use the same set of ten randomly generated seeds for all the experiments to maintain a fair comparison.

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Empirical justification for training models at 10K steps instead of 120K
comment: Thank you for your detailed review and positive appreciation of the paper.

Our choice of model evaluation at 10K pretraining steps is in line with the convergence of perplexity as depicted in Figure 1 of the Appendix section. We have now provided results for the fully trained DNABert on 120K steps using the original authors' pretrained model weights provided by the codebase. We further observe empirically from the performance comparison of the original DNABert model trained for 120K steps versus the same model trained on 10K steps from Table 1 that: the ORI 10K model outperforms the ORI 120K model in 500 and 1000-shot settings. However, ORI 120K model outperforms ORI 10K model in lower data settings (10, 50, and 100-shot). This may be because the pretrained model memorizes the gene sequences instead of learning actual (intrinsic) genomic knowledge. Thus its contribution to performance improvement is limited to very low data settings.

Additionally, the authors of DNABert do not provide strong justification for training for 120K steps nor provide any results or model checkpoints for intermediate pretraining steps. We thus posit that it is not justified to conclude that pretraining for 120K steps should always perform better than a model trained for 10K steps.

The empirical rationale for choosing 10k steps is already mentioned in the paper (Pretraining topic of the Implementation Details section). Quoted from Section 5.3 of the paper:

… We select such a setup for two reasons —

(i) to explore different pretrained model variants in a reasonable time because we observe that the perplexity of DNABert (SOTA model) has converged to a low score of 2.526 and is stable over the last 3000 pretraining steps (see Figure 1). 

(ii) As observed by Levine et al. (2021), PMI-masking learns fast and is thus quite efficient to reap the benefit even with a lower number of pretraining steps. We use the same hyperparameter setting as the original SOTA model (see Section A.2 to know more about training details and hyperparameters used).

From: ICLR.cc/2023/Conference/Paper4658/Authors
title: Improving the reliability of empirical analysis
comment: Thank you for your review, and we completely agree with your comment. We have now updated Tables 1 and 2 with multiple-run results. We report the mean and standard deviation of accuracy and AUC by running the few-shot experiments **ten** times by randomly choosing _t_ different training (fine-tuning) examples per class and a randomly generated seed in each run. We use the same set of ten randomly generated seeds for all the experiments to maintain fair comparison.

Regarding the comment about running it "3 times in the normal setting", we do not report the results as our claim is limited to the few-shot setting. However, we observe in 500-shot and, more specifically, in the 1000-shot setting that the performance improvement of PMI masking-guided models over the original SOTA model is limited. One reason may be that the effect of pretraining (extra information) becomes negligible when a good amount of fine-tuning data is available.

From: ICLR.cc/2023/Conference/Paper4658/Reviewer_ARuE
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper develops a strategy to improve on NLP models motivated by biological applications. Instead of random masking of tokens, the paper adapts Pointwise Mutual Information (PMI) masking (used previously in NLP) for gene sequence modeling.
strength_and_weaknesses: Without questioning the results, the contribution of the method is severely limited. The PMI measure and masking scheme has been developed elsewhere. Unfortunately the adaptation of the previous methods to gene sequences, while might be a contribution to that biological domain, does not have significant intellectual merit for ICLR community. The write up is also not in the publication format. The terminology between k-mers and n-grams are used interchangeably. Some of the derivations are not clear. For example, in (1) p(.) is not defined. It is not clear how to go from (1) to (2) and how we switch from a probability measure to frequency measure. The empirical studies can be strengthened by having confidence intervals and better organization.
clarity,_quality,_novelty_and_reproducibility: Not in a publication level.
summary_of_the_review: The paper can be strengthen by a better exposition of contributions and structure to the empirical results.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4658/Reviewer_QUAU
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors describe an improvement over existing transformer models DNABert and LOGO are transformers trained with MLM for computing gene representations. The approach proposes to use PMI (pointwise mutual information) to identify DNA spans that occur much more than expected and then proposes to mask these with higher probability than other spans.

The approach follows existing NLP works such as SpanBERT (Joshi et al 2020) and PMI masking (Levine et al 2020). However this approach improves on PMI masking by using ranking instead of a fixed masking vocabulary.

The authors evaluate on 3 benchmarks (2 promoter and 1 enhancer) in a variety of few-shot settings, outperform a reimplementation of SOTA models LOGO and DNABert. The authors also find that PMI-based rankings have some commonality with DNA “motifs”.
strength_and_weaknesses: Strengths:
* the authors report SOTA in 3 tasks, compared to reimplementations.
* the authors plan to open-source the their full pre-training and fine-tuning setup, which appears to not have been done for previous SOTA models.

Weaknesses:
* the authors only train their models for 10K steps instead of for example 120K steps (as was reportedly done in DNABert). This suggests that their reimplementation of DNABert and LOGO is weaker than in the original papers.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and high quality. The authors will release code for reproducibility. The PMI ranking approach seems to be mildly novel compared to the PMI-masking they cite from Levine et al 2020.
summary_of_the_review: The paper has positive results on multiple tasks compared to SOTA baselines. It successfully brings to genetics an improvement from NLP (PMI masking) and contributes a small but interesting modeling improvement (PMI ranking).
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper4658/Reviewer_Anyp
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: In this paper, the authors propose to improve performance of previously published work on unsupervised training on genomics sequence using masked language models (in particular DNABERT and LOGO). These models tend to use uniform random masking strategy during training (that is, randomly uniform tokens are chose at each training step). The authors leverage some previous work on NLP—PMI masking—to improve the masked language modeling on genomics sequence. The authors argue that the PMI-based approach masks tokens more efficiently than uniform masking and therefore achieve better results in some simple benchmarks. They show results in a few genomics toy-tasks to support the claim.

======== Post-rebuttal update ========

I appreciate the extra experimental results provided by the authors. However, I agree with reviewer ARuE and believe that this work is not ready for publication. The novelty is very limited (PMI-guided MLM has already been applied in sequences) and the analysis is not very convincing. For that reason, I keep my score to 3.

strength_and_weaknesses: ### Strengths
- The idea of developing/improving unsupervised/self-supervised training for genomics data is of extreme important.  As pointed by the authors, the lack of large amount of labeled data makes the line of work extremely useful in practice for genomics/healthcare applications.

### Weaknesses
- Although I believe this is an important application, I feel this paper lacks contribution to fit the scope of this conference.
- First is the lack of novelty. The paper does not provide any novel contribution on the context of machine learning/representation learning. The proposed approach is a simple extension (ie, the PMI-masking strategy—previously published) built on the top of previously published models (ie, DNABERT and LOGO).
- Second, the experimental result is not very convincing. Below are some weaknesses I spot:
	- The experimental results is not very clear and sometimes difficult to follow.
	- The authors seems to not make use of any validation set (only train/test sets). This is a weird as the model has some hyperparameters to be tuned.
	- The authors evaluate on a “few shot setting” (considering 10 to 1000 samples), but seems to only train/test one time (with seed 42). This is not good practice, as it is well known that those models have high variance and performance can change a lot depending on what (few shot) samples it is chosen to finetune. Ideally multiple training should be done (considering different t samples at each time) and results should be reported in terms of mean/std.
	- The authors trained the baseline models (DNABERT and LOGO) for only a small fraction of time. IT would be nice to compare the PMI-version with the original models trained until convergence.  It is very likely that performance would not be different and the PMI-masking only helps to converge faster instead of performing better (we would need those numbers to have any certainty).
- Third, the task used to evaluate (promoter and enhancer predictions) are not very relevant in practice nor biologically meaningful. Training a simple binary classifier on a hand-selected dataset is not very insightful, as we (humans) already know the labels of each instance. When training models in genomics (or other biological datasets), it is much more interesting if the model can show some insights about the biology surrounding what we want to model. I recommend the authors look at the Enformer paper (Avzec et al, Nature Methods 2021), and the references therein, for more meaningful evaluations.
clarity,_quality,_novelty_and_reproducibility: - The paper is not very clear to read, it contains many typos/grammatical errors and—more importantly--wrong biological definitions (eg, promoter is not defined correctly on section 5.1).
- The paper lacks any novel/insightful contribution.
- The paper lacks implementation details. It does provide the source code.
summary_of_the_review: Based on my comments above, I recommend the current version of this work for rejection. Although developing/improving unsupervised/self-supervised learning model for genomics data is very important, I believe this work is not ready for publications (see weaknesses section above).
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper4658/Reviewer_61nZ
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper aims to improve the pre-trained language models for the genome. Existing SOTA methods such as the DNABERT train the model with a modified Masked Language Modeling objective that randomly selects continuous spans of k-mers to mask. This paper proposes to use Pointwise Mutual Information to guide the masking procedure.
strength_and_weaknesses: Strength:
  - This paper is clearly written and easy to follow.
  - The problem this paper aims to solve is important.
  - The proposed method is efficient and effective, especially in the few-shot setting. If the empirical analysis is reliable, researchers in this community may achieve a good performance boost in the few-shot setting by simply replacing the `DNABERT` or `LOGO` model with the ones this paper provided.

Weaknesses:
  - My main concern is about the reliability of the empirical analysis since the reported results are all single-run results. Since the model fine-tuning in the few-shot setting is highly unstable, empirical results on single runs are not convincing enough. I would suggest the authors run the experiments multiple times (at least 10 times in the few-shot settings and 3 times in the normal settings) and report the *mean* and *std* to make the results more reliable.
clarity,_quality,_novelty_and_reproducibility: Clarity: Good.

Quality: Good.

Novelty: Good.

Reproducibility: Easy to reproduce.
summary_of_the_review: This paper proposes a simple yet effective modification of existing pre-trained genome language models. Though the reported results look good, it is less reliable since they are all single-run results. I would suggest a weak reject for now. I may increase my scores if more convincing and comprehensive empirical results are provided.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold