Paper: Embed-Search-Align: DNA Sequence Alignment using Transformer models (ICLR-2024)

Reviewer comment (Experiments):
The experimental validation is the weak point of this paper, with claims of efficiency yet no experimental evidence of any resource requirements and scaling such as time and memory.

Author response:
**[Q2] What are the runtime resource requirements?**

The model requires a GPU with >10GB of vRAM to run inference on DNA-ESA. The model uses 20GB of CPU memory during inference. This is an initial version of the model, and as noted in Sec. Future Work, will continually become more performant using methods such as model distillation, compilation, and distributed processing techniques. Efficient transformers are still an area of active research which have seen notable improvements over the last few years (e.g. FastAttention, Long-range architectures such as HyenaDNA and similar).

Reviewer comment (Experiments):
There are no experiments on real data, or evaluation of the effect on downstream variant calling.

Author response:
**[Q1] Have you evaluated the method on real data/data with noise?**

Before we discuss the additional evaluation results we have included in the revised version, we wanted to address why we used ART as a simulation toolkit. An important consideration was that for such reads, we know the ground truth (making evaluation much more precise) while ensuring that the reads have the same distribution of errors as one would expect from real world sequencers. 

In fact, ART is an industry-standard simulator that is used to benchmark almost every new aligner that is introduced. ART mimics the variants, insertions/deletions (INDELs) and other error models found in reads generated by expensive Illumina machines. The ART simulator used in our experiments was fine-tuned on an Illumina MiSeqv3 machine model (https://www.illumina.com/systems/sequencing-platforms/miseq/specifications.html). 
BWA-Mem and several other aligners have been validated on such simulated data: Table 1 here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2705234/ demonstrates performance on WGSIM from SAMTools, which is a more primitive version of ART (https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-14-184). ART was also used in the simulation study of the 1000 Genomes datasets to profile the quality of the resource: https://www.niehs.nih.gov/research/resources/software/biostatistics/art/index.cfm

We also want to note that Pure Reads serve an important purpose: They establish an upper bound on the performance of the aligner (as noisy reads would only have lower performance), allowing us to benchmark the maximum possible performance for Transformer-based LLM models as shown in Fig. 2. These pure-read results make it superfluous to benchmark the existing Transformer models on noisy data / real data. 

In the revision: While we believe that ART reads with $Q_{PH} >= 30$  are sufficient to benchmark aligners -- Next-gen Illumina machines generate reads at Phred score $Q_{PH} >= 30$, we address the reviewers’ request by evaluating DNA-ESA on reads with $Q_{PH} \in [10,20]$, $I,D = 1$%; see Table 4 (left).

We are pleased to report that DNA-ESA performance at $Q_{PH} \in [10,20]$, $I,D = 1$% is only about $1$% less than the performance reported at $Q_{PH} \in [30,60]$, $I,D = 0.01$%.

Performance on real read data: In Table 4, we have also included the performance of DNA-ESA on real reads generated by PacBio CCS ($Q_{PH} = 20$): and read length $\in \{250, 350, 500\}$. 

Here, too, DNA-ESA performs as well on real read data as on ART-generated reads reported in our original submission.  This further supports our original evaluation platform.

Reviewer comment (Experiments):
Generalisation to other species is very interesting, however the chosen species are all closely related.

Author response:
**[Q3] Does the model generalize to less related organisms? Presumably there will be some dependence on the degree of homology.**

We thank Reviewer 3 for their curiosity in understanding whether the model generalizes to less related organisms. Accordingly, we have expanded the inter-species experiments in the transfer learning section (Table 3) to include Thermus Aquaticus -- an ancient extremophile bacterium -- and Acidobacteriota reference sequences: two species that are highly dissimilar to the rest. We are pleased to report that the model -- originally trained to align reads to fragments on Chr. 2 of humans -- continues to perform well (in fact even with top-K = 1, the recall is >93%)! This adds further support to the claim that the DNA-ESA architecture truly learns the alignment task rather than the sequences themselves seen during training.

Reviewer comment (Experiments):
Have you evaluated the method on real data?

Author response:
**[Q1] Have you evaluated the method on real data/data with noise?**

Before we discuss the additional evaluation results we have included in the revised version, we wanted to address why we used ART as a simulation toolkit. An important consideration was that for such reads, we know the ground truth (making evaluation much more precise) while ensuring that the reads have the same distribution of errors as one would expect from real world sequencers. 

In fact, ART is an industry-standard simulator that is used to benchmark almost every new aligner that is introduced. ART mimics the variants, insertions/deletions (INDELs) and other error models found in reads generated by expensive Illumina machines. The ART simulator used in our experiments was fine-tuned on an Illumina MiSeqv3 machine model (https://www.illumina.com/systems/sequencing-platforms/miseq/specifications.html). 
BWA-Mem and several other aligners have been validated on such simulated data: Table 1 here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2705234/ demonstrates performance on WGSIM from SAMTools, which is a more primitive version of ART (https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-14-184). ART was also used in the simulation study of the 1000 Genomes datasets to profile the quality of the resource: https://www.niehs.nih.gov/research/resources/software/biostatistics/art/index.cfm

We also want to note that Pure Reads serve an important purpose: They establish an upper bound on the performance of the aligner (as noisy reads would only have lower performance), allowing us to benchmark the maximum possible performance for Transformer-based LLM models as shown in Fig. 2. These pure-read results make it superfluous to benchmark the existing Transformer models on noisy data / real data. 

In the revision: While we believe that ART reads with $Q_{PH} >= 30$  are sufficient to benchmark aligners -- Next-gen Illumina machines generate reads at Phred score $Q_{PH} >= 30$, we address the reviewers’ request by evaluating DNA-ESA on reads with $Q_{PH} \in [10,20]$, $I,D = 1$%; see Table 4 (left).

We are pleased to report that DNA-ESA performance at $Q_{PH} \in [10,20]$, $I,D = 1$% is only about $1$% less than the performance reported at $Q_{PH} \in [30,60]$, $I,D = 0.01$%.

Performance on real read data: In Table 4, we have also included the performance of DNA-ESA on real reads generated by PacBio CCS ($Q_{PH} = 20$): and read length $\in \{250, 350, 500\}$. 

Here, too, DNA-ESA performs as well on real read data as on ART-generated reads reported in our original submission.  This further supports our original evaluation platform.

Reviewer comment (Reproducibility):
What are the runtime resource requirements?

Author response:
**[Q2] What are the runtime resource requirements?**

The model requires a GPU with >10GB of vRAM to run inference on DNA-ESA. The model uses 20GB of CPU memory during inference. This is an initial version of the model, and as noted in Sec. Future Work, will continually become more performant using methods such as model distillation, compilation, and distributed processing techniques. Efficient transformers are still an area of active research which have seen notable improvements over the last few years (e.g. FastAttention, Long-range architectures such as HyenaDNA and similar).

Reviewer comment (Experiments):
Does the model generalise to less related organisms? Presumably there will be some dependence on degree of homology.

Author response:
**[Q3] Does the model generalize to less related organisms? Presumably there will be some dependence on the degree of homology.**

We thank Reviewer 3 for their curiosity in understanding whether the model generalizes to less related organisms. Accordingly, we have expanded the inter-species experiments in the transfer learning section (Table 3) to include Thermus Aquaticus -- an ancient extremophile bacterium -- and Acidobacteriota reference sequences: two species that are highly dissimilar to the rest. We are pleased to report that the model -- originally trained to align reads to fragments on Chr. 2 of humans -- continues to perform well (in fact even with top-K = 1, the recall is >93%)! This adds further support to the claim that the DNA-ESA architecture truly learns the alignment task rather than the sequences themselves seen during training.

Reviewer comment (Writing):
I felt that the paper is a very dense read for the general ML audience at ICLR for folks who do not have DNA sequencing background, and it will be great to make the paper more accessible.

Author response:
**[Q1] “I felt that the paper is a very dense read for the general ML audience at ICLR for folks who do not have DNA sequencing background, and it will be great to make the paper more accessible.”**

Acknowledging the interdisciplinary nature of our work, we concur with the reviewer's perspective that it is our responsibility, as the authors, to bridge the gap between cutting-edge techniques in NLP and the longstanding challenges in bioinformatics and genomics. To address this, we have taken the following steps:

In Eq. 1, we define a read in its simplest form: a sequence of bases. In Eq.s 2 and 3, we write the objective of sequence alignment through the application of the sharding rule [P1]. Furthermore, in Section 3.1, we present the rationale behind a straightforward inequality constraint Eq. 4 (modified now to incorporate homologs / repeats) within the representation space.

It is worth noting that our approach to motivating sequence alignment, as outlined, is itself innovative and designed to resonate with the machine learning community. We remain optimistic that the reviewer finds merit in this novel perspective.

Reviewer comment (Evaluation):
The performance for short reads is worse than long reads, given that short reads are more commonly used, this may affect how this system can be actually used.

Author response:
**[Q2] “The performance for short reads is worse than long reads, given that short reads are more commonly used, this may affect how this system can be actually used.”**

The reviewer rightly notes that short-read aligners are more commonly used than long-read aligners. Our approach does not intend to replace existing short-range alignment methods. Rather, methods for sequencing have seen rapid advances in cost and their ability to produce increasingly longer reads (PacBio and Oxford Nanopore, up to 10s of kilobases long), and accordingly, aligners to map such long reads are also of recent interest in the bioinformatics community.

Reviewer comment (Experiments):
Limited demonstration of applications in downstream genomic tasks.

Author response:
Re: the additional downstream tasks: We would like to note that Sequence Alignment, unlike the several downstream tasks that other Transformer-DNA models address, is a significantly larger task that on its own unlocks solving several subtasks -- such as the chromosome-specific and across-species alignment demonstrated in Table 2,3. We plan to apply our model to downstream classification tasks. As we note in the last paragraph of the paper (Concluding Remarks) and in the General Comments,  this model can be applied to alignment tasks concerning the Pan Genome and on a complementary task of Genome Assembly. This is because unlike traditional aligners, *the trained DNA-ESA model constructs a general mapping for sequences of nucleotides to a projection space* (irrespective of the reference from which it is derived). As demonstrated in Figure 1, global patterns in this space emerge naturally, and one can simply trace along these paths to reconstruct a reference genome.  We intend this paper to be a first step in such directions.

Reviewer comment (Presentation):
Can the authors comment on how this paper is a good fit for ICLR, and the steps the author may take to make this paper more accessible to the ML audience?

Author response:
**[Q3] “Can the authors comment on how this paper is a good fit for ICLR, and the steps the author may take to make this paper more accessible to the ML audience?”**

Representation learning holds a pivotal role within the framework of ICLR. The treatment of DNA sequences demands a computational approach grounded in deep learning, as evidenced by numerous recent works in the field [1,2,3]. Notably, the guidelines for ICLR explicitly outline a track dedicated to "Applications in biology, and related fields."

In our endeavor to make our paper accessible to the broader ICLR community, which includes individuals with expertise in machine learning, we kindly direct the reviewer's attention to the first part of the response. We believe our work aligns with the conference's objectives, particularly within the specified track.


[1] Nguyen, Eric, et al. "Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution." arXiv preprint arXiv:2306.15794 (2023).

[2] Dalla-Torre, Hugo, et al. "The nucleotide transformer: Building and evaluating robust foundation models for human genomics." bioRxiv (2023): 2023-01.

[3] Ji, Yanrong, et al. "DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome." Bioinformatics 37.15 (2021): 2112-2120.