Paper: BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks (ICLR-2024)

Reviewer comment (Novelty):
This is not original research and a substantial fraction of the paper is concerned with background material.

Author response:
As outlined above, we consider our submission well within what is welcomed by ICLR, and would appreciate your evaluation and feedback accordingly. 

We are convinced that the experiments, materials and findings provided by BEND constitute original research, and are not a mere review of existing materials.

Reviewer comment (Experiments):
What is the performance of non LM models on these tasks?

Author response:
Thank you for this question - including the performance of non LM models in the benchmark is of key interest to us, as we think it is absolutely necessary to put LM performance into context of existing non-LM approaches. We have made changes to the “Supervised baselines” section to highlight that the baseline CNN on one-hot encoded DNA does not use DNA LMs, and that all expert methods that we evaluated on the benchmark data do not use DNA LMs either.

Reviewer comment (Experiments):
The supervised learning baselines are useful, but are rather weak. For example, in the context of histone modifications, a model such as Sei which is trained on much larger datasets has achieved much higher accuracy (although the results are not strictly comparable). I expect such a model to perform much better than the DNA LM approach.

Author response:
It is true that for many tasks, there are available methods that are trained with supervision on larger datasets and that they should yield higher accuracy by both architectural innovation as well as transfer learning from multiple datasets. 

However, a key objective of curating expert methods for BEND was that they could be evaluated on the same data on comparable terms as far as possible, so that LM performance can be put in context directly (see also answer below regarding literature methods in Table 3). For Sei, which was trained on ~22k tracks, it would be difficult to answer in direct comparison whether unsupervised DNA LM features are beneficial for predicting histone modifications in a given dataset. The Basset (Kelley 2017) model, which we trained on the exact same data as the CNN models operating on DNA LM features, allows us to compare performance directly - revealing that DNA LM features can be beneficial, but also that fully supervised approaches are very competitive when the data is large. We have updated the “Supervised baselines” section to better explain the role of the task-specific methods sourced from literature.

Reviewer comment (Experiments):
Regarding the enhancer prediction task: This is a case where additional forms of data such as DNA accessibility and DNA contacts are able to provide additional hints that make this problem more tractable, and likely provide much higher accuracy than reported here (see e.g. the EPCOT model from the Liu lab).

Author response:
We fully agree that experimental functional genomics data renders enhancer prediction a more tractable problem, and should in practice always be leveraged for developing state-of-the-art enhancer prediction methods. That being said, the focus of BEND is evaluating DNA LMs - and as such, we were interested in probing to what extent (unsupervised) DNA LMs encode features useful for this task, comparing against baseline approaches and existing methodology (Enformer). 

In fact, based on the observed results, we would absolutely advise against using currently available DNA LMs without additional inputs for this task in practice. We have extended the results discussion to highlight that our findings refer to enhancer annotation without using supporting experimental data as input.

Reviewer comment (Evaluation):
In Table 3 the authors provide accuracy from the literature. Not clear how relevant those numbers are since they are based on potentially very different datasets. It should be noted in the manuscript that those numbers should be taken with a grain of salt.

Author response:
We realize now that labeling this row as “Literature” was ambiguous. In fact, all performance metrics provided in the table were computed on BEND’s datasets and we can confirm that they are thus directly comparable. “Literature” was merely meant to convey that these are task-specific expert methods curated from existing literature. 

We have now renamed the row to “Expert method” to better communicate its meaning, and have added an explanation to section 4-Modeling.

Reviewer comment (Evaluation):
The authors note that "reasoning over very long contexts, as e.g. required for finding enhancers, is still challenging." Is this too much to expect from these models? (i.e expect them to reason over long sequence contexts). Perhaps we should be content with LMs producing representations that are based on short contexts, and leave long-range contexts for downstream models?

Author response:
Thank you for raising this interesting question! We tend to agree that it might be too much to expect self-supervised models to learn features that capture genomic organization over up to 50k bp by masked language modeling only. With recent DNA LMs increasingly claiming improved scaling to longer input lengths, we think it is crucial to establish long-range benchmark tasks to investigate quantitatively whether scaling the input length is really helping. Our results do not suggest that there is a direct effect of pretraining on longer ranges on long-range task performance. 

Should we therefore be content with what we have? This is hard to answer in general, as it very much depends on the task that needs to be solved. For many genomic problems, having long-range understanding would undoubtedly be beneficial. We hypothesize that the future might bring hybrid approaches, where masked language modeling allows a model to learn local context features, and readily available functional genomics data provides a supervised signal for learning long-range dependencies.

Reviewer comment (Reproducibility):
The authors say they provide "An adaptable benchmarking framework for preparing embeddings and training lightweight supervised models." This point requires some elaboration - I assume you are referring to the code in the (currently anonymous) github repository.

Author response:
This is correct - BEND is set up in a way that allows new tasks to be added in a straightforward manner (a bed file with coordinates, splits and labels,  and YAML-formatted configuration that defines the task). We also provide instructions for adding new LMs, and strive to keep the repo updated with regards to new DNA LMs (e.g. by having added the NT-V2 LMs since the original submission).

The BEND repository has sphinx documentation that highlights how adding new embedders and tasks works in practice - the documentation source is included in the anonymized repository. We apologize that this is inconvenient to browse for review with the constraints imposed by anonymity, but will be hosted in proper HTML form eventually.

Reviewer comment (Experiments):
You had noted that your benchmark's advantage is a mix of short and long sequence classification; however, I think you could have chosen better problems for long sequence tasks, e.g. gene expression prediction; also, the histone modification prediction task might have benefitted from longer sequence contexts.

Author response:
This is a great point - tasks that are derived from genome-wide functional genomics experiments can technically be defined over *any* range (up to the limits imposed by chromosomes) - as seen in e.g. the recent Borzoi model (Linder 2023), that defined expression prediction over 524 kbp. Within the benchmark, where the defined aim is to investigate LMs and compare the utility of their pretrained features, we opted for a length that fits the context of all included DNA LMs, and have therefore chosen to predict histone modifications from 512bp, the context window of DNABERT. We absolutely agree that longer context should be beneficial to these tasks (provided that an LM learned to exploit the context). 

Establishing BEND’s data in bed-based format also makes it straightforward to change the flanking context at will, which we hope renders the materials more reusable and flexible for future LMs. We mention this in section 3, and have provided a more detailed discussion in section A.2.

For BEND, we aimed at specifically establishing long-range tasks that have a natural biological context length imposed by the task - rather than the context length being driven by computational scaling constraints. Since very long context ranges are common in the genome and its regulation, we believe it is vital for the future utility of DNA LMs to benchmark and aim for good performance on tasks that take such ranges into account.

Reviewer comment (Writing):
A comment regarding splice site classification: You note that "Moreover, there are cases in which a short-sequence task represents a simplification compared to real-world applications, as exemplified by SS-containing sequences. In genome annotation, classifying SSs is a subproblem of the gene finding task and would typically not be performed on its own." This is mixing the issue of short sequences and how a splice site classifier would be used. From my experience, SS classification is a relatively easy task that does not really need long sequences; while it is used in gene finding, it has other applications related to gene annotation; the condition-dependent version of the problem stands on its own (see e.g. the recent paper on the pangolin deep learning model).

Author response:
Thanks for this comment. Indeed, splice site prediction has applications beyond the context of gene finding - we have now rewritten this section to better express the original idea (SS prediction performance being insufficient to reason about gene finding performance) without unduly claiming a limited scope of SS prediction on its own.

Reviewer comment (Writing):
"Gene-adjacent regulatory elements are referred to as cis, and distant ones as trans." The definition of cis and trans is not a matter of distance!

Author response:
Thank you for bringing this to our attention - we have removed this erroneous statement.

Reviewer comment (Theory):
However, I feel that the task formulation for long sequences lacks depth and isn't entirely persuasive.

Author response:
Including feedback from all reviewers, we have now clarified various aspects of the underlying experimental data of the two long-range tasks, and extended our description of how their formulation follows existing task-specific work (AUGUSTUS Stanke & Waack 2003, Enformer Avsec et al. 2021). Formulating tasks in a way that enables them to be evaluated by previous task-specific methods was of key interest when establishing BEND. We would be happy to address any further comments regarding their persuasiveness.

Reviewer comment (Experiments):
When benchmarking DNA Language models, it's crucial to explore the intricacies of training at least one of these models from scratch. This would provide a comprehensive insight into their potential and the limitations of pretraining.

Author response:
Indeed, we were interested in training LMs from scratch. We trained both a dilated CNN as well as an AWD-LSTM DNA LM that we included in all experiments. The dilated CNN was motivated by the fact that this architecture had shown good performance in GPN on A. thaliana (Benegas et al. 2023), whereas the AWD-LSTM was considered a very simple, but computationally cheap model to put gains of more expensive architectures into context.  
As suggested, having this comparison does give a more rounded perspective: On many tasks, the dilated CNN shows competitive performance with medium-sized transformer-based DNA LMs. That being said, the variety of models provided by Nucleotide transformer allows us to reason about pretraining intricacies, even when models of the scale of NT-MS are out of reach to reproduce in this work (reported: 128 A100s for 28 days). The comparison of NT-V2 (now added to the manuscript) and NT-MS to the other NT-family models reveals that a multispecies pretraining dataset can yield better understanding of gene structure unique among all investigated models. To our best knowledge, this was not reported before.

Reviewer comment (Experiments):
A side-by-side comparison with a DNA Language model trained from scratch is essential. Such an analysis would give a more rounded perspective on the strengths and shortcomings of the existing methods.

Author response:
Indeed, we were interested in training LMs from scratch. We trained both a dilated CNN as well as an AWD-LSTM DNA LM that we included in all experiments. The dilated CNN was motivated by the fact that this architecture had shown good performance in GPN on A. thaliana (Benegas et al. 2023), whereas the AWD-LSTM was considered a very simple, but computationally cheap model to put gains of more expensive architectures into context.  
As suggested, having this comparison does give a more rounded perspective: On many tasks, the dilated CNN shows competitive performance with medium-sized transformer-based DNA LMs. That being said, the variety of models provided by Nucleotide transformer allows us to reason about pretraining intricacies, even when models of the scale of NT-MS are out of reach to reproduce in this work (reported: 128 A100s for 28 days). The comparison of NT-V2 (now added to the manuscript) and NT-MS to the other NT-family models reveals that a multispecies pretraining dataset can yield better understanding of gene structure unique among all investigated models. To our best knowledge, this was not reported before.

Reviewer comment (Presentation):
Figure 1 appears to inaccurately represent the length of certain genomic features. For example, the majority of exons in the human genome are shorter than 200 base pairs, with an average length between $120-170$ base pairs. Furthermore, promoter lengths usually range from 100 to 1000 base pairs. The diagram should at least offer a rough indication of these lengths. It's also crucial to highlight that exons and introns alternate in their appearance, a detail that the figure should encompass.

Author response:
Thank you for the feedback, we have updated the figure. The figure now has updated lengths, and we have added an additional exon and intron to highlight their alternation. We have also extended the figure’s caption accordingly.

Reviewer comment (Novelty):
In the introduction, I'd suggest mentioning foundational works like DeepSEA alongside DeepBind when discussing supervised learning on DNA. These groundbreaking studies have left an indelible mark on the field.

Author response:
Thank you for the suggestion - We have now ensured that DeepSEA is mentioned at the same time as DeepBind.

Reviewer comment (Evaluation):
Concerning the enhancer annotation, there is some uncertainty regarding the authenticity of the ground truth. The ABC method is based on inference and might not reflect a direct experimental outcome.

Author response:
It is correct that the ABC method in itself is a predictive method and should not be considered ground truth. We want to avoid uncertainty in the labels of the benchmark datasets and therefore did not consider ABC-predicted enhancers for BEND. Rather, all included enhancers were validated directly using either CRISPRi-FlowFISH (Fulco et al 2019) or crisprQTL (Gasperini et al 2019).  
We have updated the datasheet in the supplement to explicitly state that ABC-predicted enhancers are not part of the dataset.

Reviewer comment (Experiments):
Clarity is sought on the dataset splitting methodology — why was the data from the pretraining phase included? What about considering a leave-chromosome-out approach to splitting? The methods to prevent data leakage should also be elucidated.

Author response:
Having adequate splits for reliably quantifying performance is of key interest to us. As outlined in our answer further down below, we have ensured that individual datasets are split according to applicable best practices, either using a leave-chromosome-out approach, or splitting based on sequence homology for the gene finding task. The approach is now better highlighted in the text. These steps should largely prevent data leakage between partitions.  
Regarding data from the pre-training phase, following established practices in the protein language modeling field, we do not consider input sequence overlap between unlabeled pre-training data and labeled task data as leakage. We understand that data leakage from pre-training can be an issue when evaluating e.g. LLMs on text, where it is possible that the answer (=label) of a data point is encountered in conjunction with the data point during pre-training. When pre-training on unlabeled genomes, followed by evaluation on classification tasks, this naturally cannot occur. We therefore consider our approach aligned with established best practices and are confident that leakage risks are minimized.

Reviewer comment (Evaluation):
The decision to categorize variant effect prediction as a binary problem raises questions. Given the intricate nature of genomic variations and the subsequent implications, framing this as a regression task might be more appropriate.

Author response:
Indeed, variant effect prediction is a problem that allows for many different formulations, and for BEND, we opted to frame it as a classification task as done previously in DeepSEA and Nucleotide Transformer. Ultimately, the decision whether a task is categorized as classification or regression is driven by the available data. The underlying data that we used (eQTL+background SNPs, pathogenic+benign SNPs) provides categorical labels, making it a natural choice to render it as a classification problem.  
We fully agree that variant effect regression, as often seen in the protein LM field (e.g. Meier et al 2021), would be a valuable complementary task - we are however not aware of any experimental data on the human genome that would be comparable. That being said, we find that the formulation as a binary task allows us to gain insights into what features LMs learn - as exemplified by NT-MS’ unique understanding of splice site variants.

Reviewer comment (Experiments):
It's crucial, especially with such intricate datasets, that downstream models be evaluated using a leave-chromosome-out strategy to guarantee robust results.

Author response:
Thank you for raising this important point. We agree that this is important and we should have clarified our splitting strategy better. We do actually split by chromosome for all tasks except for gene finding (where sequence homology-based splitting based on the coding sequence is more appropriate and commonly used). We have now ensured that the splitting strategy is described in detail in the supplementary datasheet for each task, and added a statement in section 3-Tasks and Datasets explaining that we follow a chromosome-based splitting approach in general.

Reviewer comment (Experiments):
The absence of a simple CNN baseline for the variant effect prediction task is a notable omission. Such a baseline would not only validate the implemented supervised models but also provide an insight into the relative performance of more complex models.

Author response:
Thank you for this suggestion, we should clarify why there is no CNN baseline for variant effect prediction. This task was rendered as a zero-shot classification task (following Dalla-Torre et al 2023), using the LM’s embedding cosine distance as the predictor. Therefore, no supervised models were trained on the task, and a supervised shallow CNN baseline would not be applicable for comparison.  We have now updated section 4-Modeling to discuss this accordingly.