Paper: Biological Sequence Editing with Generative Flow Networks (ICLR-2024)

Reviewer comment (Novelty):
Poorly described relation to previous research and unclear if compared with the right methods.

Author response:
Existing machine learning methods within the domain of biological sequences including the ones discussed in the related works have predominantly concentrated on generating novel *de novo* sequences with desired properties from scratch. Therefore, there is an inherent risk of deviating significantly from naturally occurring sequences compromising safety and predictability. In contrast, this paper explores the generation of edited sequences that enhance the properties of existing sequences, ensuring the edited counterparts maintain similarity to their original counterparts. Here, we take an input seed sequence and modify a few elements to improve its properties. 

To underscore the limitations of existing biological sequence design methods, we conducted experiments on AMP and CRE datasets using GFlowNet to generate sequences from scratch. Our observations reveal that GFlowNet-generated sequences for the CRE dataset differ from existing sequences at 76.29% of locations, and for the AMP dataset, the difference is found in 95.24% of locations. In contrast, the proposed GFNSeqEditor demonstrates the ability to generate new edited sequences differing from existing ones in less than 35% of locations across all datasets. Moreover, employing existing biological sequence methods the difference between generated sequences and existing ones cannot be controlled while using the proposed GFNSeqEditor the amount of edits can be controlled by hyperparameters. 

In order to highlight the differences between biological sequence editing and biological sequence design problems we have added a discussion to the introduction of the revised manuscript. This revision can be found on pages 1 and 2 highlighted in red. Also, we extend the related works discussion in Appendix D and the revisions are highlighted in red.

Furthermore, another aspect that distinguishes the proposed GFNSeqEditor from prior works is that existing methods are not able to combine multiple sequences to create a new sequence. In section 4.3, the proposed algorithm combines sequences to generate a new sequence similar to its parent sequences. In section 4.3, we discuss that this can also be used for sequence length reduction which has important applications in vaccine therapies.

Reviewer comment (Reproducibility):
That can provide consistency condition, Bengio et al. (2021) formulates ow-matching loss function as follows: LFM(s; θ) = log P∀s′:s′→s Fθ(s′ → s) P′′′′ Fθ(s → s′′) . (4) Moreover, as an alternative objective function Page 3 — Sure but which do you use?

Author response:
We used trajectory balance loss in equation (5) to train the flow function. On page 7 before section 4.1 of the revised version, we made clear that we used trajectory balance loss. The revision is highlighted in red.

Reviewer comment (Writing):
2  R(x) Page 3 — Not defined here.

Author response:
In general the reward function $R(x)$ can be any function and it is determined by the environment. In experiments we consider the sequence property as the reward since in experiments our goal was to maximize those properties.

Reviewer comment (Reproducibility):
GFlowNet’s ow function Fθ(·) to identify sub-optimal positions of x, and subsequently replace the sub-optimal parts with newly sampled edits based on the stochastic policy π(·). pretrained ow function Fθ(·) Page 4 — At this point its implementation is not clear or why it can be pre trained.

Author response:
In the first paragraph of section 3 we add that “Assume that the flow function $F_\theta(\cdot)$ is trained on an available offline training data.”

Reviewer comment (Writing):
6  Pa′∈A Page 4 — It should be made clear that the given character x_t always belongs to the available actions.

Author response:
We add a comment on page 5 that “Assume that $x_t \in \mathbb A$, $\forall t$ meaning that $x_t$ is always in the available actions.”

Reviewer comment (Reproducibility):
7  chosen by the algorithm Page 4 — Point out how or, alternatively, where you describe it.

Author response:
To clarify how to choose hyperparameters $\delta$, in this revised version after equation (6) we add that “Choosing larger $\delta$, it is more probable that the algorithm identifies $x_{t+1}$ as sub-optimal”. Furthermore, to clarify how to choose hyperparameter $\sigma$ after equation (7) we add that “The relation between $\sigma$ and the algorithm performance will be analyzed in section 3.3 and Appendix E.”

Reviewer comment (Theory):
9  RF,T represents the reward of a sequence with length T generated using the ow function Fθ(·) Page 5 — It IS the reward. Formulate it as a rv with the distribution induced by the flow function, with reference to the correct equation.

Author response:
Based on this comment, we revise the notation and in the revised version $R_{F,T}$ denotes the expected reward of a sequence with length $T$ generated using the flow function $F_\theta(\cdot)$.

Reviewer comment (Writing):
10  Levenshtein Page 6 — Spelling.

Author response:
We checked the spelling and we believe it is correct.

Reviewer comment (Evaluation):
Higher binding activity is preferable Page 6 — In what sense? It may not be so in a biological system.

Author response:
We followed the prior studies such as Trabucco et al., (2021) and Jain et al., (2022) in which the goal in the TFbinding dataset is to increase the binding activity.

Reviewer comment (Presentation):
13  14.34 Page 7 — This should be in bold, right?

Author response:
We made14.34 bold in Table 1.

Reviewer comment (Experiments):
One notable concern is the difference in the starting points for experimentation. While this work leverages pretrained GFN models, other baselines begin from scratch. This discrepancy could potentially lead to an unfair comparison.

Author response:
We would like to clarify that other baselines also use a pretrained model to perform editing. While we trained GFlowNet for GFNSeqEditor, we trained an MLP as the proxy model for Ledidi and Directed Evolution (DE) baselines. Ledidi and DE are highly dependent on the proxy model to evaluate their candidate edits and use these evaluations to proceed to the next iteration for editing. Also, we trained a Transformer for the Seq2Seq baseline and Seq2Seq employs the trained transformer model to perform sequence editing. Please note that:
1. We used exactly the same training data to pretrain GFlowNet and models used by the other baselines. 
2. Due to the space limit, we put the experimental details in Appendix C.1.

Therefore, we believe that comparisons in the paper are fair since all baselines use a trained model and all models are trained on the same training data. Thanks to this comment to improve the clarity of the experiment section, we expand implementation details explanations in Appendix C.1 and also in section 4 we refer the readers to Appendix C.1.

Reviewer comment (Experiments):
The baseline comparisons should extend beyond the scope of other generative models and optimization techniques in biological sequence design. It would be beneficial to incorporate baseline methods such as offline model-based optimization [1], which are tailored to extrapolate sequences from offline datasets, thereby yielding "improved sequences."

Author response:
We would like to clarify that biological sequence design methods such as offline model-based optimization models including the ones discussed in [1] do not tackle the problem of sequence **editing** and as a result, they are not appropriate to be directly used for biological sequence editing for two main reasons: 
1. Biological sequence design models usually generate entirely new sequences from scratch. Therefore, sequences generated by these methods are expected to be greatly different from existing sequences that need editing. Therefore, it might be infeasible to generate diverse sequences using biological sequence design models that are similar to the existing sequences. 
2. The difference between generated sequences and existing ones cannot be controlled by employing biological sequence design methods while using the proposed algorithm by setting hyperparameters one can control the number of edits.

To underscore the limitations of biological sequence design methods, we generate new sequences from scratch using GFlowNet and we observe that on average for CRE dataset the generated sequences are different from existing sequences (i.e. sequences in the datset) in 76.29% of locations and it is not possible to control the amount of differences between generated sequences and existing ones. Also, for AMP dataset the generated sequences are different from existing ones in 95.24% of locations. In contrast, the proposed GFNSeqEditor demonstrates the ability to generate new edited sequences differing from existing ones in less than 35% of locations across all datasets. To highlight the differences between biological sequence editing and biological sequence design problems we have added a discussion to the introduction of the revised manuscript. The revision can be found on pages 1 and 2 highlighted in red.

Reviewer comment (Experiments):
To provide a more comprehensive perspective on the proposed approach, the paper could benefit from the inclusion of promising evolutionary algorithms specifically designed for biological sequence design [2].

Author response:
Thanks for drawing our attention to the evolutionary algorithm in [2]. Note that [2] implements a standard directed evolutionary algorithm for sequence design. Our directed evolutionary baseline is the same as the evolutionary algorithm presented in Algorithm 1 of [2]. We cite the reference [2] in the revised version.

Reviewer comment (Experiments):
Conducting a thorough comparison with GFN baselines would be valuable in demonstrating the relative strengths and weaknesses of the proposed method when contrasted with models of similar architecture.

Author response:
Existing GFN baselines can be used for biological sequence design from scratch and as a result, they suffer from the same problems as those of other biological sequence design methods discussed above when it comes to performing sequence editing. However, since GFlowNet can generate sequences token by token sequentially, there is a naive approach to employ GFlowNet for sequence editing. In this approach, the first tokens of a sequence $x$ can be given to GFlowNet and then the GFlowNet generates the rest of the tokens. We add this as a baseline to Table 1 in the revised version and the baseline is called GFlowNet-E. For example, for the CRE dataset, the GFlowNet-E gets the first 130 tokens and generates the rest of 70 tokens. Compared to GFNSeqEditor, this approach lacks the flexibility to choose edit locations. GFNSeqEditor outperforms GFlowNet-E and this indicates the effectiveness of sub-optimal position identification of GFNSeqEditor in equation (7).

Reviewer comment (Reproducibility):
Could you please elaborate on the process you used for pre-training GFN?

Author response:
**Response to Q1.** We trained an active learning based GFlowNet model following the setting in Jain et al. (2022). In the active learning setting, at each round of active learning $t \times K$ candidates generated by GFlowNet are sampled and then top $K$ samples based on scores given by a proxy are chosen to be added to the offline dataset. Here offline dataset refers to an initial labeled dataset. To train the GFlowNet, we employed the same proxy models as those used by other baseline methods. For all datasets, we set the number of active learning rounds to $1$, with $t$ equal to $5$ and $K$ equal to $100$. We parameterize the flow using a MLP comprising two hidden layers, each with a dimension of $2048$, and $|\mathbb A|$ outputs corresponding to individual actions. Throughout our experiments, we employ the trajectory balance objective for training. Adam optimizer with $(\beta_0, \beta_1) = (0.9, 0.999)$ is utilized during the training process. The learning rate for $\log Z$ in trajectory balance loss is set to $10^{-3}$ for all the experiments. The number of training steps for TFbinding, AMP and CRE are $5000$, $10^6$ and $10^4$, respectively. The remaining hyperparameters were configured in accordance with the settings established in Jain et al. (2022).

Due to space limitations, we provided the training details of employed GFlowNet models in Appendix C.1. In this revised version, we provided more training details in the main text of the paper. This revision can be found on page 7 before section 4.1 highlighted in red.

Reviewer comment (Experiments):
Have you conducted a comparison with baseline models (e.g., Seq-to-Seq) using the pretrained GFN as a component?

Author response:
**Response to Q2.** Our baselines in the initial submission do not use the trained GFN since they do not construct sequences token by token sequentially. However, all baselines perform editing using a trained model. All models including GFN and the transformer of Seq2Seq are trained on the same training data.

Reviewer comment (Experiments):
Is this algorithm more beneficial than naive search algorithms based on pretrained policies, such as beam-search or MCTS?

Author response:
**Response to Q4.** Generating a new sequence from scratch by GFlowNet can be viewed as a search algorithm based on a pretrained policy. The benefit of using the learned policy by GFlowNet compared to other policies is that the probability of constructing a sequence by GFlowNet is proportional to the sequence reward. Moreover, naive search algorithms such as beam-search and MCTS usually need to observe the intermediate reward. However, when it comes to constructing a new object sequentially, the intermediate reward is not available and the reward is given when the object is fully constructed. Therefore, naive search algorithms may not work well in generating new objects. GFlowNet is able to learn to generate a new object with probability proportional to its final reward when there is not any intermediate reward. Leveraging GFlowNet by the proposed algorithm is its benefit compared to naive search algorithms.

Reviewer comment (Experiments):
Have you performed experiments related to Theorem 1 and 2? Inclusion of such experiments would likely enhance the overall quality of the paper.

Author response:
**Response to Q5.** We performed experiments related to Theorems 1 and 2. Figure 3 studies the effect of hyperparameters $\delta$ and $\lambda$ on the property  improvement and edit percentage. Furthermore, in Figure 4, we illustrate the impact of changing $\sigma$ on property improvement and edit diversity for GFNSeqEditor. These results corroborate the theoretical analyses outlined in Theorems 1 and 2 in section 3.3 as well as Theorem 3 in Appendix E.

Reviewer comment (Reproducibility):
Lack of Training Details: The paper lacks sufficient information regarding the training process of the policy. It should provide more details on the training data used, the methodology for updating parameters, and the specific hyperparameters employed in the process.

Author response:
## Training Details
Due to the space limit, we put the training details in Appendix C.1. To address this comment, we provide more information about the training process of the flow function in the main text of the revised paper. Please see page 7 where we add that ``*To train models associated with baselines and the proposed GFNSeqEditor, we partition each dataset into a 72% training set and an 18% validation set. The remaining 10% constitutes the test set, employed to evaluate the performance of methods in sequence editing tasks. The trained flow function $F_\theta(\cdot)$ employed by GFlowNet-E and the proposed GFNSeqEditor, is an MLP comprising two hidden layers, each with a dimension of $2048$, and $|\mathbb A|$ outputs corresponding to actions. Throughout our experiments, we employ the trajectory balance objective for training the flow function. Detailed information about training the flow function can be found in Appendix C.1.*''  This revision is highlighted in red in the revised version. Also, we expand the implementation details in Appendix C.1.

Reviewer comment (Novelty):
Unclear Literature Review: The literature review in the paper needs improvement. It is not adequately clear what the main contribution of the proposed method is, and how it distinguishes itself from existing work, particularly in relation to the utilization of GFlowNet for sequence generation. The paper should provide a more explicit and comparative analysis of related work.

Author response:
## Literature Review
To address this comment, we expand the literature review in the introduction and we discuss our contribution relative to the biological sequence design methods including GFlowNet. This revision can be found on pages 1 and 2 highlighted in red. Also we extend the detailed related work discussion in Appendix D. Please find below the discussion about the contributions that distinguish this paper from prior works. 

### Biological Sequence Design
Existing machine learning methods within the domain of biological sequences have predominantly concentrated on generating novel sequences with desired properties from scratch. Therefore, there is an inherent risk of deviating significantly from naturally occurring sequences compromising safety and predictability. In contrast, this paper addresses the generation of new edited sequences that enhance the properties of existing sequences, ensuring the edited ones maintain similarity to their existing counterparts. Here, we take an input seed sequence and modify a few elements to improve its property. To underscore the limitations of existing biological sequence design methods, we conducted experiments on AMP and CRE datasets using GFlowNet to generate sequences from scratch. Our observations reveal that on average GFlowNet-generated sequences for the CRE dataset differ from existing sequences at 76.29% of locations, and for the AMP dataset, the difference is found in 95.24% of locations. In contrast, the proposed GFNSeqEditor demonstrates the ability to generate new edited sequences differing from existing ones in less than 35% of locations across all datasets. Moreover, employing existing biological sequence methods the difference between generated sequences and existing ones cannot be controlled while using the proposed GFNSeqEditor the amount of edits can be controlled by hyperparameters.

### Sequence Combination
Also another aspect that distinguishes the proposed GFNSeqEditor from prior works is that existing methods are not able to combine multiple sequences to create a new sequence. In section 4.3, the proposed algorithm combines sequences to generate a new sequence similar to its parent sequences. In section 4.3, we discuss that this can be used for sequence length reduction which has important applications such as vaccine therapies.

### GFlowNet
Existing GFlowNet methods can be used for biological sequence design from scratch and as a result, they suffer from the same problems as those of other biological sequence design methods discussed above when it comes to performing sequence editing. However, since GFlowNet can generate sequences token by token sequentially, there is a naive approach to employ GFlowNet for sequence editing. In this approach, the first tokens of a sequence $x$ can be given to GFlowNet and then the GFlowNet generates the rest of the tokens. We add this as a baseline to Table 1 in the revised version and the baseline is called GFlowNet-E. For example, for the CRE dataset, the GFlowNet-E gets the first 130 tokens and generates the rest of 70 tokens. Compared to GFNSeqEditor, this approach lacks the flexibility to choose edit locations and GFNSeqEditor outperformed GFlowNet-E. This indicates the effectiveness of sub-optimal position identification of GFNSeqEditor.

Reviewer comment (Novelty):
Ambiguity in Key Innovation: The claim that GFNSeqEditor can produce novel sequences with improved properties lacks clarity regarding the key innovation driving these contributions. The paper should better articulate what novel techniques or insights lead to the claimed improvements, thereby enhancing the reader's understanding of the method's unique value.

Author response:
## Key Innovation:
We would like to emphasize that the primary goal of GFNSeqEditor is not to produce novel sequences and the focus is different from biological sequence design. While GFNSeqEditor can indeed be utilized in conjunction with biological-sequence generation models to create novel sequences with improved properties, as empirically examined in section 4.2, the primary intent of GFNSeqEditor is not centered around generating entirely novel sequences. Instead, the core purpose of the proposed GFNSeqEditor lies in generating a diverse array of edits for a given seed sequence $x$, ensuring that the edited sequences exhibit similarity to $x$ while simultaneously displaying improved properties. To highlight the key innovations that contributed to the performance gain of GFNSeqEditor compared to other baselines, in this revised version we expand the discussion about the results in Table 1 and Figure 3. We explain that the proposed sub-optimal identification method outlined in equation 7 and the proposed editing policy in equation 9 are the key innovations to achieve improved performance compared to other baselines. This revision is highlighted in red on page 7 in section 4.1.