Paper: Encoding Unitig-level Assembly Graphs with Heterophilous Constraints for Metagenomic Contigs Binning (ICLR-2024)

Reviewer comment (Evaluation):
It shows that the overall performance is heavily reliant on the choice of downstream clustering methods. Using Kmeans on the learned representation yields surprisingly poor results, which raises the question of whether the representation learning is beneficial in this application.

Author response:
Thanks for the comments.

Q1: Applying K-means to the learned embeddings often yields poor performance, primarily due to two reasons: a) imbalanced bin sizes (i.e., varying from a couple to hundreds), and b) inefficient incorporation of information from constraints derived from single-copy marker genes. To tackle these challenges, we designed a combined binning operation that integrates contig-level label propagation while maximizing constraint satisfaction (refer to equation 5).
It is a good question to ponder whether representation learning is beneficial for metagenomic contig binning. From my perspective, the primary strength of utilizing representation learning techniques, as opposed to traditional algorithms, lies in the efficient embedding of all relevant information together, bypassing the need for intricate algorithm design and moving towards large language models. Within our model, it is possible to concurrently represent the unitig-level assembly graph, nucleotide sequences, and supplementary information from single-copy marker genes. 
In Figure 6, our ablation study on Sim100G reveals that UnitigBin-Learning (applying the K-Means algorithm to learning representations) yields unsatisfactory results. This is primarily attributed to the large number of bins and the challenge of imbalanced bin sizes, ranging from 3 to 652. We also assessed UnitigBin-Learning on Sim20G, a dataset with fewer bins and a less pronounced issue of imbalanced bin sizes. We achieved an F1 score of 0.93 (second only to the 0.97 obtained by RepBin) and an ARI of 0.87 (comparable to 0.96 for VAMB and 0.95 for RepBin). These results demonstrate competitiveness, significantly surpassing non-representation learning-based algorithms like MetaCoAG, which achieved F1 and ARI values of 0.54 and 0.79, respectively. Therefore, models capable of producing effective representations prove advantageous for metagenomic contig binning.

Reviewer comment (Novelty):
Considering the additional computational complexity introduced when learning directly from Unitig assembly graphs, it is not entirely clear why the proposed method can capture additional information that may have been overlooked in prior work that learned from reconstructed contig assembly graphs.

Author response:
Q2: We choose to learn the unitig-level assembly graph rather than the contig-level assembly graph to preserve the original data and avoid introducing errors in constructing the contig-level assembly graph from the unitig-level assembly graph.  The output from assemblers (such as metaSPAdes and metaFlye) is essentially a unitig-level assembly graph. Previous graph-based models devised strategies to construct a contig-level assembly graph from the unitig-level assembly graph to simplify it for the task of binning contigs. However, this construction process introduces errors, indicating that linked edges in the contig-level assembly graph may not necessarily belong to the same bin (e.g., the error rate for Sim50G is 7.3%). The motivation behind using the assembly graph is rooted in the homophily assumption, where linked unitigs/contigs are likely to belong to the same bin. The introduction of erroneous edges in the contig-level assembly graph adds complexity to embedding or learning. To circumvent these errors, we directly focus on the original unitig-level assembly graph.

Reviewer comment (Experiments):
Is there any evidence or an intuitive explanation of what additional information is captured by learning directly from the unitig assembly graph instead of the reconstructed contig assembly graph, and how this contributes to the improvement in binning performance? Conducting an ablation study by running a similar framework directly on the unitig assembly graph and demonstrating performance differences could be helpful for illustration.

Author response:
Q2: We choose to learn the unitig-level assembly graph rather than the contig-level assembly graph to preserve the original data and avoid introducing errors in constructing the contig-level assembly graph from the unitig-level assembly graph.  The output from assemblers (such as metaSPAdes and metaFlye) is essentially a unitig-level assembly graph. Previous graph-based models devised strategies to construct a contig-level assembly graph from the unitig-level assembly graph to simplify it for the task of binning contigs. However, this construction process introduces errors, indicating that linked edges in the contig-level assembly graph may not necessarily belong to the same bin (e.g., the error rate for Sim50G is 7.3%). The motivation behind using the assembly graph is rooted in the homophily assumption, where linked unitigs/contigs are likely to belong to the same bin. The introduction of erroneous edges in the contig-level assembly graph adds complexity to embedding or learning. To circumvent these errors, we directly focus on the original unitig-level assembly graph.

Reviewer comment (Writing):
The paragraph on Page 3 in the Problem Statement only talked about the notation but does not specify what the task is, or what the desired output should be, making it unclear what exact problem is to be solved.

Author response:
Q1: The goal of binning metagenomic contigs is to group nucleotide sequences (e.g., ATCG) into separate clusters. Our input data includes the assembly graph at the unitig level, nucleotide sequences, and additional constraints obtained from single-copy marker genes. 
We simplify the clustering challenge of sequences and frame it as a task of graph learning and constraint satisfaction. In detail, we aim to incorporate two key pieces of information: a)homophily information within the assembly graph, indicating that unitigs/contigs linked together in the assembly graph are more likely to belong to the same bin, and b)heterophilious relations of constraints, stating that unitigs/contigs within the same constraint set must belong to distinct bins. The ideal situation for contig binning is that all contigs are grouped into distinct clusters corresponding to the constituent genomes and satisfy previous heterophilic constraints. 
i) In the learning phase, our primary focus is on acquiring the homophily structural information of the unitig-level assembly graph while simultaneously integrating heterophilious constraints in a soft manner, accomplished through the inclusion of a penalty function (refer to the equation 3). The objective function of the learning phase mainly consists of two components: the loss of the reconstructed graph that captures homogeneous information in the assembled graph and the loss of optimizing heterogeneous constraints to maximize constraint satisfaction. 
ii) During the binning phase, our attention shifts to optimizing the heterophilous constraints and synergizing label propagation with maximizing constraint satisfaction (refer to equation 5). The primary components of the binning phase's objective function include labeling unannotated contigs using the homogeneous assembly graph and incorporating a loss term to maximize constraint satisfaction. The binning process, specifically the minimization of constraint violations, is illustrated in Figure A3(b). It can be observed that as the training/binning epochs increase, the number of violating heterophilious constraints decreases. 
iii) Finally, as there are still some binning results violating the constraints, we introduce a refinement method to eliminate such violations.

Reviewer comment (Theory):
It is not clear what objective is aimed at in the problem formulation, which raises the question of how to ensure that the objective function in each component maps to the objective of binning.

Author response:
Q1: The goal of binning metagenomic contigs is to group nucleotide sequences (e.g., ATCG) into separate clusters. Our input data includes the assembly graph at the unitig level, nucleotide sequences, and additional constraints obtained from single-copy marker genes. 
We simplify the clustering challenge of sequences and frame it as a task of graph learning and constraint satisfaction. In detail, we aim to incorporate two key pieces of information: a)homophily information within the assembly graph, indicating that unitigs/contigs linked together in the assembly graph are more likely to belong to the same bin, and b)heterophilious relations of constraints, stating that unitigs/contigs within the same constraint set must belong to distinct bins. The ideal situation for contig binning is that all contigs are grouped into distinct clusters corresponding to the constituent genomes and satisfy previous heterophilic constraints. 
i) In the learning phase, our primary focus is on acquiring the homophily structural information of the unitig-level assembly graph while simultaneously integrating heterophilious constraints in a soft manner, accomplished through the inclusion of a penalty function (refer to the equation 3). The objective function of the learning phase mainly consists of two components: the loss of the reconstructed graph that captures homogeneous information in the assembled graph and the loss of optimizing heterogeneous constraints to maximize constraint satisfaction. 
ii) During the binning phase, our attention shifts to optimizing the heterophilous constraints and synergizing label propagation with maximizing constraint satisfaction (refer to equation 5). The primary components of the binning phase's objective function include labeling unannotated contigs using the homogeneous assembly graph and incorporating a loss term to maximize constraint satisfaction. The binning process, specifically the minimization of constraint violations, is illustrated in Figure A3(b). It can be observed that as the training/binning epochs increase, the number of violating heterophilious constraints decreases. 
iii) Finally, as there are still some binning results violating the constraints, we introduce a refinement method to eliminate such violations.

Reviewer comment (Experiments):
In the experiment setup, it is unclear how the training/testing/validation is conducted.

Author response:
Q2:The binning of metagenomic contigs emphasize the extraction of features from the nucleotide sequences and assembly graph, possibly incorporating additional information from references or single-copy marker genes. And then organize these contigs into distinct clusters corresponding to the constituent genomes. Thus, contigs binning can be conceptualized as an unsupervised clustering problem. 
In our paper, we utilize 12 datasets—six assembled by metaSPAdes and six assembled by metaFlye. We execute our proposed UnitigBin along with selected baselines on these datasets to generate the final binning results. In the experiment, we primarily employ CheckM v1.1.3 and AMBER v2.0.2 to assess the performance of UnitigBin and the baseline methods. In Table 2, all runs of UnitigBin and the baseline methods are performed five times on Sim20G, and the best binning assignment (evaluated using the AMBER tool) is reported. Additionally, we also present the average values along with the corresponding standard deviation values. Significance tests are also conducted, and the results are presented below.
| Methods   | F1(bp)         | F1(seq)        | ARI(bp)        | ARI(seq)       | C<5;P>90  |
| --------- | -------------- | -------------- | -------------- | -------------- | --------- |
| Maxbin2   | 0.985(0.005)   | 0.634(0.010)   | 0.987(0.003)   | 0.784(0.008)   | 19.3(0.6) |
| SemiBin   | 0.982(0.007)   | 0.541(0.004)   | 0.954(0.024)   | 0.410(0.008)   | 19.0(0.0) |
| VAMB      | 0.983(0.007)   | 0.604(0.012)   | 0.985(0.009)   | 0.969(0.007)   | 18.3(0.6) |
| MetaBAT2  | 0.629(0.025)   | 0.313(0.011)   | 0.400(0.009)   | 0.224(0.008)   | 4.0(0.0)  |
| GraphMB   | 0.950(0.031)   | 0.603(0.022)   | 0.552(0.012)   | 0.364(0.017)   | 9.3(0.6)  |
| CCVAE     | 0.981(0.004)   | 0.617(0.004)   | 0.790(0.004)   | 0.431(0.034)   | 13.3(0.6) |
| RepBin    | 0.955(0.012)   | 0.447(0.011)   | 0.959(0.004)   | 0.148(0.009)   | 15.7(0.6) |
| MetaCOAG  | 0.955(0.006)   | 0.591(0.002)   | 0.988(0.003)   | 0.783(0.004)   | 14.7(0.6) |
| UnitigBin | 0.995(0.005)\* | 0.953(0.003)\* | 0.992(0.003)\* | 0.780(0.006)\* | 19.7(0.6) |

Reviewer comment (Evaluation):
While the results reported in Table 2 are promising, it is unclear how significant they are.

Author response:
Q2:The binning of metagenomic contigs emphasize the extraction of features from the nucleotide sequences and assembly graph, possibly incorporating additional information from references or single-copy marker genes. And then organize these contigs into distinct clusters corresponding to the constituent genomes. Thus, contigs binning can be conceptualized as an unsupervised clustering problem. 
In our paper, we utilize 12 datasets—six assembled by metaSPAdes and six assembled by metaFlye. We execute our proposed UnitigBin along with selected baselines on these datasets to generate the final binning results. In the experiment, we primarily employ CheckM v1.1.3 and AMBER v2.0.2 to assess the performance of UnitigBin and the baseline methods. In Table 2, all runs of UnitigBin and the baseline methods are performed five times on Sim20G, and the best binning assignment (evaluated using the AMBER tool) is reported. Additionally, we also present the average values along with the corresponding standard deviation values. Significance tests are also conducted, and the results are presented below.
| Methods   | F1(bp)         | F1(seq)        | ARI(bp)        | ARI(seq)       | C<5;P>90  |
| --------- | -------------- | -------------- | -------------- | -------------- | --------- |
| Maxbin2   | 0.985(0.005)   | 0.634(0.010)   | 0.987(0.003)   | 0.784(0.008)   | 19.3(0.6) |
| SemiBin   | 0.982(0.007)   | 0.541(0.004)   | 0.954(0.024)   | 0.410(0.008)   | 19.0(0.0) |
| VAMB      | 0.983(0.007)   | 0.604(0.012)   | 0.985(0.009)   | 0.969(0.007)   | 18.3(0.6) |
| MetaBAT2  | 0.629(0.025)   | 0.313(0.011)   | 0.400(0.009)   | 0.224(0.008)   | 4.0(0.0)  |
| GraphMB   | 0.950(0.031)   | 0.603(0.022)   | 0.552(0.012)   | 0.364(0.017)   | 9.3(0.6)  |
| CCVAE     | 0.981(0.004)   | 0.617(0.004)   | 0.790(0.004)   | 0.431(0.034)   | 13.3(0.6) |
| RepBin    | 0.955(0.012)   | 0.447(0.011)   | 0.959(0.004)   | 0.148(0.009)   | 15.7(0.6) |
| MetaCOAG  | 0.955(0.006)   | 0.591(0.002)   | 0.988(0.003)   | 0.783(0.004)   | 14.7(0.6) |
| UnitigBin | 0.995(0.005)\* | 0.953(0.003)\* | 0.992(0.003)\* | 0.780(0.006)\* | 19.7(0.6) |

Reviewer comment (Evaluation):
One motivation described in the Introduction is that the presented approaches will respect the constraints, but the evaluation of how this is achieved compared to other methods is missing; it would be useful to show this either through an explicit metric (or clarify if an existing metric already does this) or through an example.

Author response:
Q3: One of the objectives of our proposed UnitigBin model is to maximize the satisfaction of heterophilious constraints. Therefore, we calculate the ratio of binning results that violate constraints. The process of minimizing the number of violating constraints is illustrated in Figure A3(b). Additionally, the ratio of violating constraints achieved by UnitigBin and selected baselines is presented in the table below (on Sim100G). UnitigBin fulfills nearly all constraints, with only a 0.9% violation of heterophilious constraints, a lower rate compared to other baseline methods.
| MaxBin2     | SemiBin     | VAMB        | MetaBAT2 | GraphMB      | CCVAE        | RepBin      | MetaCoAG    | UnitigBin   |
| ----------- | ----------- | ----------- | -------- | ------------ | ------------ | ----------- | ----------- | ----------- |
| 1712/127449 | 2226/180044 | 2623/152948 | 37/1313  | 42244/170476 | 22627/167211 | 8854/177910 | 2069/153026 | 1686/182366 |
| 1.34%       | 1.24%       | 1.92%       | 2.82%    | 24.80%       | 13.53%       | 4.98%       | 1.35%       | 0.90%       |

Reviewer comment (Theory):
Suggest to motivate the readers more by explicitly describing the problem formulation, including what objective is desired; ideally this can be formulated as a mathematical objective function and tied to the choice of loss function in the later learning component.

Author response:
Q1: The goal of binning metagenomic contigs is to group nucleotide sequences (e.g., ATCG) into separate clusters. Our input data includes the assembly graph at the unitig level, nucleotide sequences, and additional constraints obtained from single-copy marker genes. 
We simplify the clustering challenge of sequences and frame it as a task of graph learning and constraint satisfaction. In detail, we aim to incorporate two key pieces of information: a)homophily information within the assembly graph, indicating that unitigs/contigs linked together in the assembly graph are more likely to belong to the same bin, and b)heterophilious relations of constraints, stating that unitigs/contigs within the same constraint set must belong to distinct bins. The ideal situation for contig binning is that all contigs are grouped into distinct clusters corresponding to the constituent genomes and satisfy previous heterophilic constraints. 
i) In the learning phase, our primary focus is on acquiring the homophily structural information of the unitig-level assembly graph while simultaneously integrating heterophilious constraints in a soft manner, accomplished through the inclusion of a penalty function (refer to the equation 3). The objective function of the learning phase mainly consists of two components: the loss of the reconstructed graph that captures homogeneous information in the assembled graph and the loss of optimizing heterogeneous constraints to maximize constraint satisfaction. 
ii) During the binning phase, our attention shifts to optimizing the heterophilous constraints and synergizing label propagation with maximizing constraint satisfaction (refer to equation 5). The primary components of the binning phase's objective function include labeling unannotated contigs using the homogeneous assembly graph and incorporating a loss term to maximize constraint satisfaction. The binning process, specifically the minimization of constraint violations, is illustrated in Figure A3(b). It can be observed that as the training/binning epochs increase, the number of violating heterophilious constraints decreases. 
iii) Finally, as there are still some binning results violating the constraints, we introduce a refinement method to eliminate such violations.

Reviewer comment (Experiments):
Suggest to describe the experiment setup, including what training/testing/validation data is included, and clarify the statistical significance of the result comparison; if this is not feasible, provide justification and identify future steps.

Author response:
Q2:The binning of metagenomic contigs emphasize the extraction of features from the nucleotide sequences and assembly graph, possibly incorporating additional information from references or single-copy marker genes. And then organize these contigs into distinct clusters corresponding to the constituent genomes. Thus, contigs binning can be conceptualized as an unsupervised clustering problem. 
In our paper, we utilize 12 datasets—six assembled by metaSPAdes and six assembled by metaFlye. We execute our proposed UnitigBin along with selected baselines on these datasets to generate the final binning results. In the experiment, we primarily employ CheckM v1.1.3 and AMBER v2.0.2 to assess the performance of UnitigBin and the baseline methods. In Table 2, all runs of UnitigBin and the baseline methods are performed five times on Sim20G, and the best binning assignment (evaluated using the AMBER tool) is reported. Additionally, we also present the average values along with the corresponding standard deviation values. Significance tests are also conducted, and the results are presented below.
| Methods   | F1(bp)         | F1(seq)        | ARI(bp)        | ARI(seq)       | C<5;P>90  |
| --------- | -------------- | -------------- | -------------- | -------------- | --------- |
| Maxbin2   | 0.985(0.005)   | 0.634(0.010)   | 0.987(0.003)   | 0.784(0.008)   | 19.3(0.6) |
| SemiBin   | 0.982(0.007)   | 0.541(0.004)   | 0.954(0.024)   | 0.410(0.008)   | 19.0(0.0) |
| VAMB      | 0.983(0.007)   | 0.604(0.012)   | 0.985(0.009)   | 0.969(0.007)   | 18.3(0.6) |
| MetaBAT2  | 0.629(0.025)   | 0.313(0.011)   | 0.400(0.009)   | 0.224(0.008)   | 4.0(0.0)  |
| GraphMB   | 0.950(0.031)   | 0.603(0.022)   | 0.552(0.012)   | 0.364(0.017)   | 9.3(0.6)  |
| CCVAE     | 0.981(0.004)   | 0.617(0.004)   | 0.790(0.004)   | 0.431(0.034)   | 13.3(0.6) |
| RepBin    | 0.955(0.012)   | 0.447(0.011)   | 0.959(0.004)   | 0.148(0.009)   | 15.7(0.6) |
| MetaCOAG  | 0.955(0.006)   | 0.591(0.002)   | 0.988(0.003)   | 0.783(0.004)   | 14.7(0.6) |
| UnitigBin | 0.995(0.005)\* | 0.953(0.003)\* | 0.992(0.003)\* | 0.780(0.006)\* | 19.7(0.6) |

Reviewer comment (Experiments):
It is not clear from the experiment results why GNN is needed; suggest providing case studies to show how GNN is able to capture the information that the other approaches cannot, since introducing GNN is considered the main contribution of the work.

Author response:
Q4: Traditional contig binning methods depend on statistical information derived solely from contigs, neglecting homophily information present in the assembly graph. Homophily suggests that sequences connected in the assembly graph are more likely to belong to the same species. Typically, statistics-based approaches tend to exclude short sequences, often shorter than 1,000 bp, due to the instability of features extracted from nucleotide sequences. GNNs can leverage the homophily present in the assembly graph to assign labels to these short sequences.