Paper: Contrastive Predict-and-Search for Mixed Integer Linear Programs (ICLR-2024)

Reviewer comment (Novelty):
I find one work in related work[1] very similar to this paper. Both of these two papers propose to utilize contrastive learning in solving MILP and share the core idea of generating positive and negative samples. The only difference is the operation after the contrastive learning part, the ICML paper[1] uses large neighborhood search (LNS) and this ICLR paper uses Predict and Search (PaS). Actually, I think this paper is covered by the ICML paper, as PaS could be regarded as a variant of LNS. Though the authors do mention this ICML paper in the related work, they do not discuss the difference between their work and the ICML paper, nor compare it as a baseline.

Author response:
We would like to thank the reviewer for reading our rebuttal and your valuable suggestion on addressing the differences between ConPaS and CL-LNS in the main paper. We agree that it is important for the readers to understand the differences. We have added a paragraph at the end of the related work section highlighted in blue to address this issue. Please kindly let us know if any concerns remain.

Reviewer comment (Novelty):
Though the idea of utilizing contrastive learning in MILP looks interesting, I consider the current usage of contrastive learning to be more like an incremental part. In this work, solving MILP basically relies on the performance of PaS. I am not sure if this contribution is good enough for ICLR. To me, this work is more like using contrastive learning to find a better initialization for PaS, of which the application is limited.

Author response:
We thank the reviewer for the feedback and suggestions. Regarding the weaknesses and questions concerning the novelties and choices of MIP benchmark, please kindly refer to the general responses.

In addition, we would like to further discuss how this work could be applied beyond PaS to answer your 2nd question: ConPaS is more versatile since the prediction coming out of its ML model can be useful in different ways. An example is to warm start LNS as mentioned earlier. In addition, one could leverage the ML prediction from ConPaS to assign variable branching priorities and/or generate cuts in tree searches such as branch-and-bound (or branch-and-cut) search. We defer the deployment of ConPaS in different algorithms to future work.

We also want to clarify that Neural Diving is a more restricted variant of ConPaS and PaS, where it corresponds to setting \Delta = 0 in PaS that allows no change of the assigned values once they’re fixed in the search.

Reviewer comment (Experiments):
The effect of hyperparameter k0 and k1 is evaluated. This hyperparameter is important because it controls the tradeoff between the feasibility and quality of predicted solutions. However, considering that MILP instances generally have different scales of integer variables, a specific number of integer variables may not be a good choice. It may be better to use the coverage rate (i.e., the ratio of fixed variables in the entire set of integer variables when using predict methods like Neural Diving) to control the fixed number of integer variables.

Author response:
We thank the reviewer for the feedback and suggestions.
Regarding the weaknesses, please refer to the discussions on the novelties of the work and choices of benchmark in the general response to all reviewers. 

Below are our responses to answer the question regarding hyperparameters and coverage rates:

We agree that using coverage rates as alternatives to model k0 and k1 would be more helpful when the instances are diverse in size. In our paper, we described a systematic way in Section 5.1 “Hyperparameters” to tune both k0 and k1 based on a percentage of the number of variables (10%-50%). We believe that this hyperparameter tuning method is easy to follow. We report the results of different k0 for CA to demonstrate how tuning could be done.

Reviewer comment (Experiments):
Some studies indicate that each instance has a unique optimal coverage rate (https://arxiv.org/abs/2308.00327), so evaluating the effect of k0 by just computing an average number on one dataset (CA) may not help readers configure their own prediction model properly.

Author response:
We thank the reviewer for the feedback and suggestions.
Regarding the weaknesses, please refer to the discussions on the novelties of the work and choices of benchmark in the general response to all reviewers. 

Below are our responses to answer the question regarding hyperparameters and coverage rates:

Regarding the optimal coverage rate studied in [Yoon et al., 2023], it is important for methods like Neural Diving (ND) since it requires training a separate model for each coverage rate. With an optimal coverage rate identified, it helps overcome the training inefficiency of ND. However in ConPaS, instead of fixing all variables according to the prediction, we let the MIP solver explore regions around the prediction that allows more flexibility and room for inaccuracy in prediction, therefore removing the need for an accurate coverage threshold. 


[Yoon et al., 2023] Threshold-aware Learning to Generate Feasible Solutions for Mixed Integer Programs. Arxiv 2023.

Reviewer comment (Evaluation):
Since the proposed approach is based on predict-and-search, it cannot guarantee the optimality or feasibility. This limitation is not discussed or analyzed properly in this paper. For example, there is no empirical study on the feasibility ratio on the test instances. The authors should also conduct experiments on more constrained problems. Furthermore, it is somewhat unfair to compare the anytime performance with SCIP, since the proposed method (as well as predict-and-search) essentially solves a much simpler problem than SCIP since some variables are fixed.

Author response:
ConPaS is a solution construction heuristic and we acknowledge that our approach doesn’t guarantee optimality or feasibility. Similarly, this drawback also applies to Neural Diving (ND) [Nair et al, 2020] and PaS [Han et al, 2023]. However, in a distributional setting where one needs to solve similar MIP instances over and over again, approaches like ND and ConPaS can be particularly helpful if it is able to predict solutions that are empirically feasible and of high quality. This is indeed true according to our experiments - on the five MIP benchmarks (including one in the Appendix), we achieve a 100% feasibility rate using a consistent set of hyperparameters on each benchmark, confirming the applicability of these approaches. However, we also acknowledge that ConPaS (or ND and PaS) is not universally applicable to all MIP solving, especially on more constrained problems. For example, using MIP for scientific discoveries when the solutions are sparse could be extra challenging [Deza et al., 2023] and often we need to design other approaches tailored to them. We have added the discussion in the conclusion section.

Reviewer comment (Experiments):
The authors collected training data using Gurobi, but only compared the test performance with SCIP. I cannot see any reason why not compare with Gurobi at test time.

Author response:
Thank you for this comment. We use Gurobi to collect data since Gurobi typically runs a lot faster than SCIP. For data collection, we set the time limit to an hour for Gurobi. We could easily replace Gurobi with SCIP for data collection and get the same-quality training data but this comes at the cost of 4-8 times (4-8 hours per instance)  longer runtime on average. Due to our limited computational resources, using Gurobi for data collection is more practical for us.

We have included results on Gurobi in Appendix Section D.2 in the updated draft. We show that ConPaS still outperforms Gurobi and PaS significantly in terms of both primal integral and primal gap.

Reviewer comment (Evaluation):
The authors used two ways to collect negative samples, but only report their empirical performance, without a deep analysis on which way is more reasonable.

Author response:
The main motivation to design negative samples this way is that we want them to be close to positive samples in the input space but actually with very different quality (i.e., near miss). From a theoretical point of view, the InfoNCE loss we use has the property that it will automatically focus on hard negative pairs (i.e., samples with similar representation but of very different qualities) and learn representations to separate them apart (See e.g., [Tian 2022]). While our approach is built upon a theoretical understanding of contrastive learning, we acknowledge that our work designs the negative samples heuristically and does not aim for theoretical impacts. On the other hand, we believe that our work contributes a new principled method that demonstrates strong empirical performance in challenging domains.

Reviewer comment (Evaluation):
The authors did not show the results of how accurate the solution prediction is.

Author response:
Regarding the accuracy of the predicted solutions, we would like to point out that the prediction accuracy doesn’t strongly correlate with the performance of the downstream task where the predictions are used (in this paper, the downstream task is the search phase). The ML model is trained on multiple solution samples and when deployed in the search, we use only a part of the predictions controlled by the hyperparameters. Therefore, there is no standard way to quantify the accuracy of the ML predictions in this setting that captures the downstream performance.

We report the prediction accuracy quantified by the classification accuracy over all binary variables (with the threshold set to 0.5) in the following table. We report it for both PaS and ConPaS-LQ on the MVC and CA problems on 100 validation instances. The accuracy is the fraction of correctly classified variables averaged over 50 positive samples for each instance, and we report the average accuracy over 100 validation instances. Since the classification accuracy is sensitive to the threshold, we also report the AUROC. 
On the MVC instances, though ConPaS has a lower accuracy (w.r.t. Threshold =0.5), it has higher AUROC than PaS. On the CA instances, their accuracies and AUROCs are similar. We would like to again point out that a better accuracy/AUROC doesn't necessarily indicate a better downstream task performance, even though we believe they are correlated. 


|           | MVC      |       | CA       |       |
|-----------|----------|-------|----------|-------|
|           | Accuracy | AUROC | Accuracy | AUROC |
| PaS       | 81.2%    | 0.88  | 88.3%    | 0.87  |
| ConPaS-LQ | 76.9%    | 0.91  | 86.9%    | 0.86  |