Paper: CAMBranch: Contrastive Learning with Augmented MILPs for Branching (ICLR-2024)

Reviewer comment (Presentation):
The illustration of experimental results requires to be improved. The bars missing detailed values on it. Compared with histograms, tables could be more compact for demonstration.

Author response:
Thanks for pointing this out. Actually, we have displayed the table-format results in Appendix Table 5. In the revised version, we will place Table 5 into the main text.

Reviewer comment (Experiments):
Missing comparisons to other ML approaches. Employing auxiliary tasks is an effective way to improve the training efficiency. Empirically, I found the simple auxiliary task employed in [1] can efficiently promote the training efficiency. Thus, can you provide more results comparing this data augmentation approach to other auxiliary tasks?

Author response:
Maybe missing a bibliography [1]. Could you please add it? Thanks for your feedback!

In Gasse et al. (2019), experiments results have demonstrated that GCNN outperforms other ML approaches by a large margin. Therefore, we did not include these ML approaches and CAMBranch will certainly outperform them now that CAMBranch’s performance is close to GCNN’s. 

Furthermore, it's essential to note that CAMBranch is a plug-and-play method, ensuring ease of implementation without conflicts with other auxiliary methods in most cases, and can be combined with other auxiliary methods. This adaptability and compatibility contribute to its versatility and potential for integration with other methods.

Reviewer comment (Experiments):
Marginal improvement in Figure 1. It seems the improvement of CAMBranch is not significant. Maybe a lighter GNN with fewer hidden layers and hidden nodes can achieve similar IL accuracy but requires less expert data.

Author response:
Currently, the model architecture of CAMBranch follows Gasse et al. (2019) and it only has ~10k parameters, light enough compared with those extremely deep graph neural networks like DeeperGCN. The hidden size is only 64 and extremely small for graph neural networks. The hidden layer is only one. 

To explore whether lighter GNN with less expert data can achieve similar IL accuracy or not, we conducted additional experiments. In our additional experiments, conducted with the same 10% training data (less expert data), we trained the GCNN while adjusting the hidden size to 32 and 16. Imitation results on Set Covering are shown in the following. The results highlight that merely reducing the hidden size leads to negligible performance changes. In contrast, our proposed CAMBranch significantly outperforms these approaches.

|Model|acc@1|acc@5|acc@10|
|---|---|---|---|
|GCNN (10%, hidden size 16)|59.56 ± 0.70|82.75 ± 0.98|91.27 ± 0.67|
|GCNN (10%, hidden size 32)|59.65 ± 0.30|83.36 ± 0.39|91.63 ± 0.22|
|GCNN (10%, hidden size 64)|58.98 ± 0.69|82.97 ± 0.52|91.61 ± 0.44|
|CAMBranch (10%, hidden size 64)|**65.27 ± 3.92**|**89.46 ± 2.98**|**96.14 ± 1.90**|

Significantly, when considering MILP solving, while imitation learning (IL) accuracy holds importance, it isn't the primary metric of concern. In contrast, time, Nodes, and wins are the primary metrics and can actually measure the performance of a neural branching strategy. This discussion is detailed in the Appendix E Discussion section. It's crucial to note that while Strong Branching serves as the expert strategy for imitation learning, it might not represent the optimal branching strategy. Therefore, IL accuracy in imitating Strong Branching merely stands as an intermediate result and cannot fully demonstrate the models’ performance. 

The true essence lies in the metrics evaluated during the inference stage—solving instances fast, and generating fewer nodes. Figures 2, 3, and 4 display the results, demonstrating CAMBranch's superior performance in these crucial metrics, especially evident in handling significantly harder instances.

Reviewer comment (Theory):
**Fidelity of Imitation Learning:** The effectiveness is predicated on the assumption that imitation learning can closely approximate the decisions made by Strong Branching. However, Strong Branching is known for its nuanced decision-making process, which considers a multitude of factors and potential future states. Replicating this complexity through imitation learning is challenging. If the learning model fails to encapsulate the depth of Strong Branching's strategy, the resultant branching decisions could be significantly less efficient, negating the primary advantage of CAMBRANCH.

Author response:
As depicted in the results presented by Gasse et al. (2019), the imitation learning paradigm demonstrates remarkable superiority in this domain. Since CAMBranch follows Gasse et al. (2019), CAMBranch also demonstrates superior performance. Therefore, there is no such concern.

Reviewer comment (Experiments):
**Sensitivity to Hyperparameter Tuning:** Machine learning models, particularly those used in imitation learning, are sensitive to hyperparameter settings. The performance of CAMBRANCH could be highly dependent on the choice of learning rate, batch size, and other hyperparameters. Finding the optimal configuration can be a time-consuming process that requires extensive experimentation and computational resources.

Author response:
Since our CAMBranch follows Gasse et al. (2019) and the model backbone are extremely light graph neural networks, grid search for hyperparameter tuning does not need too many resouces. Actually, both CAMBranch and Gasse et al.(2019) are not sensitive to the hyperparameter tuning, further streamlining the optimization process.

Reviewer comment (Experiments):
**Generalizability and Robustness Concerns:** The diversity of MILP problems poses a significant challenge to the generalizability of CAMBRANCH. Different MILP instances can vary drastically in terms of size, structure, and complexity. CAMBRANCH must demonstrate robust performance across a wide array of problems to be considered a viable alternative to existing B&B methods. Additionally, the model's robustness to adversarial inputs or unusual problem structures remains to be thoroughly evaluated.

Author response:
As responding in Question 1, “we focus on homogeneous MILP solving, i.e., the training and test MILP instances are the same type, which is a popular setting in industry practice.” From the results in our paper, CAMBranch as well as GCNN in Gasse et al. (2019) obviously generalize well in homogeneous MILP instances. Therefore, no such concerns.

Reviewer comment (Evaluation):
CAMBranch does not perform well on relatively easy datasets. The improvement is not significant compared with GCNN baseline on many medium and hard datasets. Fow example, results in Table 8 do not reveal that CAMBranch consistently outperforms GCNN.

Author response:
Thanks for highlighting this. While CAMBranch might not consistently outperform GCNN in a small subset of easy and medium instances, it's important to note that the differences are marginal in these scenarios. However, the true strength of CAMBranch shines through in harder instances. 

For instance, in Table 5, which is the primary focus of this paper, the Combinatorial Auction Problem at the medium level reveals that CAMBranch performs with a mere 0.3s difference compared to GCNN(10%), specifically 12.68s versus 12.38s. This trend is consistent across most cases where CAMBranch doesn't outperform GCNN (10%), as similarly observed in Table 8. 

However, when tackling genuinely challenging instances, especially in Set Covering, Capacitated Facility Location, and Maximum Independent Set (problems significantly more complex than the hard Combinatorial Auction Problem, evident from the solving times and the number of instances successfully solved), CAMBranch distinctly demonstrates its superiority in solving these intricate problems. For instance, in Set Covering, CAMBranch has a solving time of 1427.02s compared to GCNN 10%'s 2385.23s. This exemplifies the practical utility of CAMBranch since real-world instances are rarely as simple as the Combinatorial Auction Problem; most instances pose challenges of much higher complexity, such as hard Set Covering problems.

Taking this into account, CAMBranch serves as a robust, plug-and-play solution, especially in scenarios where inefficient data collection hampers performance. It consistently exhibits high efficacy in MILP problem-solving while requiring fewer resources, making it a compelling choice for improving methodologies in such settings.

Reviewer comment (Novelty):
The augmentation method is tailored for the branching task, while not easy to be transfered to other algorithms in B & B solvers. Therefore, the application is limited.

Author response:
CAMBranch, its plug-and-play nature facilitates easy implementation, specifically enhancing the neural branching decision part in the B&B algorithm. While the primary focus of CAMBranch is on the branching part, it's worth noting that the core idea of generating expert samples sharing identical decisions from the original ones holds potential insights for other decision parts within B&B solvers, such as node selection and cutting planes. The fundamental idea of generating new labeled expert samples from original expert decisions holds promise for broader applicability within the B&B framework.

Reviewer comment (Experiments):
Which sample ratio, with CAMBranch, leads to a comparative results with the model trained with the full dataset? What if use the full training dataset with this augmentation? Will it stll bring improvement?

Author response:
The CAMBranch method is proposed to alleviate challenges arising from inefficient expert sampling.  Therefore, in such contexts, it is more meaningful to use CAMBranch when facing data-scarce issues. While CAMBranch is primarily tailored to address data scarcity issues, its adaptability allows users to combine this methodology with a full training dataset. This integration aids in the form of data augmentation, further fostering improvements in the training process.

Reviewer comment (Experiments):
How many instances does it need to generate 20k expert samples? The 10% means using 10% MILP instances or 10% expert samples? How many expert samples can one instance produce? Experiments should be conduct to investigate the effect of the number of MILP instances and the expert samples. Also, the accumulative time used in collecting the expert samples should be reported.

Author response:
Thanks for pointing these out. In the introduction part, we report that collecting 100k expert samples for four types of problems requires 26.65 hours, 12.48 hours, 84.79 hours, and 53.45 hours, respectively. Therefore, collecting 20k experts（20% of the whole data) needs about 20% of the former time periods. 

The 10% means 10% expert samples, not MILP instances. Generating MILP instances is trivial, but acquiring the expert samples is challenging since numerous NP-hard MILP instances need to be solved, making it a time-consuming endeavor.

Additionally, it's worth noting that Gasse et al. (2019) extensively explored the relationship between the number of solved MILP instances and expert samples in their Appendix Section 1. Our proposed methods have nearly identical observations to Gasse et al. (2019).

Thanks for your further feedback. We appreciate your insightful suggestions regarding the analysis of the relationship between performance and various factors such as the number of instances used for sample collection and the number of expert samples. Following your suggestions, we conducted the following experiments and present the findings from the table on Combinatorial Auction Problem:

(1) **Impact of Expert Sample Size:** From pairs (I, II), (III, IV), and (V, VI), we delve into the relationship between performance and the number of expert samples, holding the number of MILP instances constant. The results indicate that increasing the number of expert samples generally enhances performance, as evident in pairs (I, II) and (III, IV). However, in scenarios with a relatively small number of instances, such as (V, VI), augmenting expert sample size may not necessarily lead to a performance increase.

(2) **Influence of MILP Instance Count:** Examining pairs (I, III, IV) and (II, IV, VI), we further explore the relationship between performance and the number of MILP instances, maintaining a consistent expert sample size. The observations suggest a general trend of performance improvement with an increase in the number of MILP instances.

|Exp ID|	Model|	Time(Easy)|	Nodes(Easy)|	Time(Medium)|	Nodes(Medium)|	Time(Hard)|	Nodes(Hard)|
|---|---|---|---|---|---|---|---|
|I|	CAMBranch 362 instances 10k expert samples| 2.03|	91|	12.68|	758|	131.79|	9074|
|II|	CAMBranch 388 instances 20k expert samples|1.68|	91|	9.92|	762|	103.38|	9050|
|III|	CAMBranch 100 instances 10k expert samples |1.67|	94|	10.02|	757|	111.52|	9242|
|IV|	CAMBranch 100 instances 20k expert samples |1.64|	90|	10.03|	751|	109.58|	9195|
|V|	CAMBranch 50 instances 10k expert samples| 3.25|	98|	20.58|	796|	158.24|	9834|
|VI|	CAMBranch 50 instances 20k expert samples|3.18|	97|	20.59|	819|	166.02|	10892|

Based on these observations, to answer the question of when to use CAMBranch, our initial conclusion is that CAMBranch demonstrates more potential when there is sufficient number of MILP instances, but not necessarily too large scale (Using several hundreds of MILP instances can already achieve superior performance). Moreover, increasing the number of expert samples provides benefits for performance, as supported by our empirical findings. 

Furthermore, the number of augmented samples is determined by the epoch at which each CAMBranch model achieves its best performance on the validation set. During each training step, an equivalent quantity of augmented samples to those in each batch is generated simultaneously for contrastive learning. These augmented samples are used only once in each training step. Upon completion of the training process, the total number of augmented samples for the saved best model depends on the epoch at which the best model was achieved. In such contexts, here we report the number of one-time augmented instances for each model: I (670k), II (440k), III (550k), IV (720k), V (250k), and VI (380k).

Reviewer comment (Evaluation):
Sometimes GCNN (10%) even outperforms GCNN (e.g., MIS in Figure 2). Why?

Author response:
Yes, it is an interesting observation and we've thoroughly investigated this observation through multiple experiments, consistently confirming its occurrence. One potential reason is that Strong Branching is not always the optimal expert strategy for branching and sometimes may lead to sub-optimal decisions, even if it makes high-quality decisions most of the time. This aligns with similar observations outlined in recent studies by Scavuzzo et al. (2022), Dey et al. (2023), and Gamrath et al. (2020). Refer to our Appendix Discussion part, “Recent studies by Scavuzzo (2022), Dey (2023), and Gamrath (2020) have even suggested that Strong Branching may underperform in certain cases, falling short of problem-specific rules.” This intriguing observation inspires us to delve into this phenomenon, for more exploration in future work.

Reviewer comment (Presentation):
The presentation of the main result should be improved.

Author response:
Thanks for your valuable comments. We have revised our paper and uploaded it, according to your and Reviewer EfKd’s suggestions. For the result presentation, we finally chose to follow Gasse et al. (2019). Moreover, we added more analysis about measuring data collection efficiency and training overhead comparison in the main text, which are suggestions from Reviewer EfKd. Also, we added a part about evaluating CAMBranch on full datasets, which are from you and Reviewer mGEq. These two parts further verify the effectiveness and efficiency of our proposed method. 

We are continuously improving our paper. Thanks for your suggestions again!

Reviewer comment (Experiments):
CAMBranch (100%) result should be discussed. Despite this work aiming in data scarce situations, understanding how it behaves with the full dataset is critical information, considering the results with 10% of data is not outperforming the baseline using full data. Even if CAMBranch using the full data can't outperform the baseline using full data, it is still beneficial to share this limitation with the community if proper discussion is provided. However, without the result, I am negative about introducing this method to the community as a MILP solver because it is unknown whether it limits the potential of imitation learning-based MILP methods.

Author response:
Thanks for pointing this out. Your concern is that “*it is unknown whether it limits the potential of imitation learning-based MILP methods.*” While we understand that discussing CAMBranch (100%) compared to GCNN (100%) is one approach to alleviate this concern directly, we would like to propose an alternative way, i.e., to compare CAMBranch (10%) and GCNN (10%) since you can **assume the full datasets have 10k expert samples** which are also enough for training usually and the performance for GCNN (10%) is still acceptable in most of the cases. Therefore, after comparing GCNN (10%) and CAMBranch (10%), we can observe that CAMBranch (10%) shows high superiority. Thus, this observation suggests that CAMBranch will probably not limit the potential of imitation learning-based methods. Experiments for CAMBranch (100%) are also as the following. Here for combinatorial auction problem. The same observations are obtained.

Easy
|Model|Time|Wins|Nodes|
|---|---|---|---|
GCNN (100%)	|1.96	|4/100|	87
GCNN (10%)	|1.99	|2/100|	102
CAMBranch (10%)	|2.03|	1/100|	91
CAMBranch (100%)|	1.73|	93/100|	88

Medium
|Model|Time|Wins|Nodes|
|---|---|---|---|
GCNN (100%)	|11.30|	7/100|	695
GCNN (10%)|	12.38|	3/100|	787
CAMBranch (10%)|	12.68|	2/100|	758
CAMBranch (100%)|	10.04|	88/100|	690

Hard
|Model|Time|Wins|Nodes|
|---|---|---|---|
GCNN (100%)|	158.81|	4/94|	12089
GCNN (10%)|	144.40|	2/100|	10031
CAMBranch (10%)|	131.79|	11/100|	9074
CAMBranch (100%)|	109.96|	83/100|	8260

In fact, In practical scenarios, it is hard to measure the exact number of samples required to obtain an acceptable policy model. For some problems, maybe 10k is enough, while for others 100k, 200k, or more will lead to a better performance. In such contexts, CAMBranch is proposed to ensure that with any available dataset, one can use CAMBranch’s augmented approach to train a powerful policy model. Notably, CAMBranch initiates the process with **data augmentation**, enabling the generation of **Hundreds of Thousands of labeled samples** that are totally different data one another for neural networks. CAMBranch significantly mitigates the time-intensive data collection process by providing an abundance of augmented expert samples. The results demonstrate CAMBranch’s superiority and highlight its value.

Reviewer comment (Presentation):
The main results appear to be promising but could be presented better. Neither Figure 2-4 nor Table 5 looks optimal in their current forms.

Author response:
Table 5 follows the paper by Gasse et al. (2019), a format also employed by other papers within this domain. We have presented Figures 2-4 as a visual representation of the data in Table 5, as we believe it provides a clearer comparison of the gaps between different methods in different difficulty levels. We are open to exploring alternative methods to present the results and welcome any suggestions you may have. Thank you for your input.

Reviewer comment (Evaluation):
The result of Table 8 deserves to be in the main text, considering the goal of this work. Also, evaluation in another domain with a more dramatic performance gap could be better for the comparison. The presentation also has room for improvement; it is hard to compare the metrics of models using the same amount of data.

Author response:
Following your suggestions, we have conducted experiments on another domain with a more dramatic performance gap to improve Table 8. You can see the complete Table in the revised supplementary material Table 10.  Here we list the results on Hard level instances of Maximum Independent Set, which has an obvious performance gap.


|Model|	Time|	Wins|	Nodes|
|---|---|---|---|
|GCNN (5%)|	607.48|	2/39|	16850|
|GCNN (10%)|	695.96|	0/20|	17034|
|GCNN (20%)|	465.00|	16/43|	12998|
|CAMBranch (5%)|	592.81|	1/41|	15480|
|CAMBranch (10%)|	496.86|	1/40|	10828|
|CAMBranch (20%)|	**416.24**|	**20/49**|	**10455**|

The results demonstrate a consistent upward trend in performance across multiple metrics, including solving time, number of wins, and nodes, with increasing sample size. Notably, CAMBranch (20%) achieved the highest number of solved instances in the "Hard" level, 49 instances in total, underscoring its superiority. Moreover, the comparison between the two methods reveals that CAMBranch consistently outperforms GCNN across all three ratios, emphasizing the superior capabilities of our proposed framework.

This additional experiment further reinforces the effectiveness of our proposed methods. Thank you again for your valuable suggestions.

Reviewer comment (Evaluation):
What would be the intuition behind using shifted geometric mean for evaluation? And what is the $s$ value for each metric?

Author response:
Utilizing the shifted geometric mean for metric calculations stands as a widely accepted practice in MILP benchmarking within the community, which is also employed by Gasse et al. (2019) and Zarpellon et al. (2021), two representative papers in this domain. Shifted geometric means have the advantage of neither being compromised by very large outliers (in contrast to arithmetic means) nor by very small outliers (in contrast to geometric means). 

The solving time for different instances, even the same instance solved with different seeds, inherently has the potential for substantial variations due to the NP-hard nature. Therefore, the community use shifted geometric mean to facilitate a more equitable comparison. In such contexts, $s$ in our paper is set to 1 for time and 100 for Nodes, following the previous work.


**References**:

Gasse, M., Chételat, D., Ferroni, N., Charlin, L., & Lodi, A. (2019). Exact combinatorial optimization with graph convolutional neural networks, NIPS.

Zarpellon, G., Jo, J., Lodi, A., & Bengio, Y. (2021, May). Parameterizing branch-and-bound search trees to learn branching policies. AAAI.