Paper: Efficient Training of Multi-task Combinarotial Neural Solver with Multi-armed Bandits (ICLR-2024)

Reviewer comment (Novelty):
Novelty: given the specific structure of the problem, the paper could improvise more specifically tailored bandit algorithms for the problem; a simple example would be using a structured/contextualized bandit algorithm that takes into account the complexity and type of the tasks.

Author response:
The proposed method in this work is agnostic to any bandit algorithms. While it is true that employing more sophisticated bandit algorithms could enhance performance by leveraging pertinent prior knowledge, it's common to lack such priors. Therefore, we focus on the most general case, wherein the environment within the bandit context dynamically changes due to training neural solvers. We choose Exp3 to address potential adversarial effects under these circumstances.

Reviewer comment (Writing):
$\eta_t$ is never defined.

Author response:
We appreciate the reviewer's observation. $\eta_t$ represents the learning rate used in training neural networks. To enhance clarity, we have included a definition for $\eta_t$ in the revised Proposition 1.

Reviewer comment (Evaluation):
In Section 4.1, it looks like the STL model trained on the large instances perform on par with your model, based on the "Mean Results on *<COP>*" plots, except only for CVRP. Then why do you think "Our method can handle various types of COPs under the same number of training epochs, which is **impossible for STL** ... "?

Author response:
The comparison in Fig. 2 might be perceived as unfair due to the difference in training epochs. Our method demonstrates superiority by training a single model for 1000 epochs compared to 12 models trained under "COP-n," each for 1000 epochs (i.e., $1000$ epochs vs $12\times 1000$ epochs in total). Despite this discrepancy, our neural solver's performance on a specific COP is either better or competitive with the results from 3 models obtained by STL (trained for a total of 3000 epochs).

    The statement "...impossible for STL..." is grounded in the fact that STL exclusively focuses on one type of COP with a specific problem scale. Consequently, STL not only exhibits poor performance on different problem scales due to limited generalization ability but also fails to adapt to various COP types, given the presence of task-specific modules.

Reviewer comment (Writing):
The writing needs more improvement. For example, they never talk about the specific MAB algorithm they use till Appendix G and H (except one line in the experimental setting). It seems they are using Thompson Sampling, or Discounted Thompson Sampling, or Exp algorithms as base MABs. These choices need more justification.

Author response:
The proposed method affords flexibility in the selection of bandit algorithms, with the primary objective of introducing the concept of employing bandits to dynamically guide training. Due to space constraints, we defer an in-depth discussion of bandit algorithms to Appendix C. And we commit to providing enhanced justifications for the chosen algorithms in the revised version.

Reviewer comment (Experiments):
No analysis of budget allocation is done, even though this is fairly well understood in the multi-task setting (both online and offline).

Author response:
The analysis of budget allocation is detailed in Appendix G, supplemented with additional descriptions. In summary, Discounted Thompson Sampling, Exp3, and Exp3R, designed to handle changing and potentially adversarial environments, exhibit nearly identical budget allocations. In contrast, naive Thompson Sampling fails to adapt, leading to suboptimal performance.

Reviewer comment (Theory):
It is not clear to me how the four different COPS can be considered as different tasks, and how the difference is captured. There is no discussion on this. Moreover from Appendix F, it is not clear how the Gradient norm captures the difference between the four different COPS as well as the scales within a COP. Can the authors elaborate on this?

Author response:
In the ML4CO community, prior RL-based works have typically trained one model for a specific type of COP with a designated problem scale. Consequently, addressing various types of COPs with different problem scales can be conceptualized as a MTL problem for two reasons: **a)** The existence of task-specific modules that cannot be universally shared across different COP types. **b)**  Variations in the MDP transition steps occur for different problem scales within a COP.

    Concerning Appendix F, as acknowledged, differences among tasks cannot be effectively captured through the loss. This limitation is explicitly discusses in Section 3.1: "*...such measurement is invalid in our context because there are no significant differences among tasks.*" Consequently, we adopt gradient-based methods to distinguish tasks. In the discussion of Fig. 3(b), we illustrate that using the inner product between gradients as reward can be misleading due to varying gradient norms among tasks. To address this, we employ cosine similarity, focusing solely on the angles between gradients.

Reviewer comment (Theory):
You use the column sum of the influence matrix to calculate the average reward for each task and then select the next task based on that. How do you ensure that you conduct a sufficient exploration/exploitation of task? Don't you want to use an exploration bonus (like UCB) over the average reward \bar{r}^i_j?

Author response:
The average reward serves as the basis for updating the bandit algorithm, and the exploration-exploitation procedure is entirely governed by the bandit algorithm.

Reviewer comment (Experiments):
Why do you only use the column-sum of the rewards, and not the row-sum from the influence matrix? Note that the row-sum denotes the influence of all tasks on one task. Intuitively even that should facilitate learning for an individual task.

Author response:
As you rightly pointed out, the row-sum signifies the influence of all tasks on one specific task, while the column-sum represents the influence of one task on all tasks collectively, which aligns with the motivation of selecting a task that maximally benefits all tasks.

Reviewer comment (Evaluation):
This question is again related to the diversity of tasks. How is your neural solver affected when some tasks are more difficult than others? How does the budget get proportioned then? What happens to the influence matrix then? Can you elaborate on this?

Author response:
Our algorithm prefers tasks that provide maximal benefits to all tasks rather than solely focusing on the most challenging task. To provide further insight, we apply the proposed method on three tasks: TSP20, TSP50, and TSP100, with a clear difficulty level follows the order TSP20<TSP50<TSP100. The resulting budget allocation is distributed in the ratio of 0.326, 0.341, and 0.333, respectively, with a larger proportion of resources allocated to TSP50. This allocation strategy is justified both empirically and through the examination of the influence matrix:**a)** In the ML4CO community, it is well-known that training tasks of intermediate difficulty can benefit those on both sides of the difficulty spectrum.  **b)** The influence matrix, shown below, provides support for this allocation strategy: Training TSP50 has a higher similarity contribution of 0.126 to TSP20 and 0.137 to TSP100, compared to the contribution from TSP100 to TSP20 (0.031) and from TSP20 to TSP100 (0.025). 
|  |   |   | 
|-|-|-|
| 1. | 0.126 | 0.031 |
| 0.124 | 1. | 0.153 |
| 0.026 | 0.137 | 1. |
|  |   |   |

Reviewer comment (Experiments):
Some design choices in the experiment section need more explanation. For example why the balanced allocation is chosen in 1:2:3 ratio from small to large scale?

Author response:
The 1:2:3 allocation is deemed appropriate for categorizing tasks into easy-median-hard levels. We intend to convey the idea that better resource allocation yields superior results. As evidenced in Table 2, STL_bla outperforms STL_avg, highlighting the significance of proper resource allocation. While alternative allocation ratios could be explored, our focus lies in demonstrating the effects rather than determining the optimal allocation ratio. Thank you for pointing out this, we have revised the descriptions.

Reviewer comment (Writing):
There is a general lack of motivation for the proposed method throughout the paper.

Author response:
We aim to clarify our motivation in the following manner:
    - The task relationships among COP has been observed but lacks both experimental and theoretical validation. Our approach begins by empirically substantiating these observations through the proposed MTL algorithm. Furthermore, at the implementation level, training neural solvers for COPs typically demands more time compared to tasks in classical MTL benchmarks. Traditional MTL methods, involving simultaneous training of all tasks, prove to be time-consuming and memory-intensive. As evidenced by the comparisons in Table 3, our approach, which selectively trains one task at each time slot, demonstrates efficiency and effectiveness. This is particularly notable in terms of training flexibility and cost-effectiveness, as emphasized in the discussions on page 7 (Table 3) and Appendix D.

        We extend our method to other MTL scenarios, as detailed in Appendix I. The results indicate that there is no single MTL method consistently excelling across all scenarios, and in certain cases, they may even underperform compared to a naive MTL approach (refer to Table 7). However, in scenarios where traditional MTL methods show efficacy, our approach maintains competitive performance (refer to Table 6).

Reviewer comment (Novelty):
How is the technique proposed specific to solving COP? As far as I understand, it could be applied to MTL setting. Should it outperform other MTL baselines in a general scenario? If not, why is it performing well on the COP tasks?

Author response:
We extend our method to other MTL scenarios, as detailed in Appendix I. The results indicate that there is no single MTL method consistently excelling across all scenarios, and in certain cases, they may even underperform compared to a naive MTL approach (refer to Table 7). However, in scenarios where traditional MTL methods show efficacy, our approach maintains competitive performance (refer to Table 6).

Reviewer comment (Theory):
The paper went on to describe a heuristic reward design, but ultimately why is it better? There is neither theoretical guarantee nor complexity analysis, so the method is somewhat unconvincing to me.

Author response:
First of all, the heuristic reward design, grounded in the cosine similarity between gradients from different tasks, aligns with widely accepted practices established in influential prior works (Wang et al., 2020; Yu et al., 2020). Secondly, this design choice is default for us and is derived from fundamental loss decomposition (Proposition 1), providing a theoretical foundation. Experimentally, the results obtained with this reward align with observations in the field of ML4CO. Consequently, our reward design stands as both theoretically sound and empirically validated.

Reviewer comment (Evaluation):
Experiment results are generally promising, but error bar should be provided to quantify significance.

Author response:
We have made significant improvements to our presentation by incorporating error bars, showing the performance on 10,000 test instances from each COP, as shown in Figure 8 in Appendix J of the revised version.

Reviewer comment (Experiments):
It feels strange that the authors brought up (Fifty et al., 2021) in the intro as the inspiration for the proposed method, but did not compare to it. Is there a specific reason for this?

Author response:
The work by Fifty et al. (2021) focuses on MTL centered around task grouping. This approach necessitates an offline pre-training phase for constructing task relationships, followed by training the resulting groupings for evaluation. Conversely, our work and the considered baselines are all online MTL methods, involving training the model on-the-fly and conducting evaluations immediately post-training.

Reviewer comment (Experiments):
What is responsible for the efficiency of the proposed method? It is about 15 times faster than the closest method (Table 3) despite only changing the sampling scheme. Can the authors provide a breakdown of how other MTL techniques compute the influence matrix?

Author response:
The enhanced efficiency of our proposed method stems from the changing in the training paradigm. Unlike typical MTL methods that simultaneously train all tasks at each training time slot, our method adopts a more targeted approach by selecting and training only one task per slot. This streamlining significantly reduces computational load and accelerates the training process. Regarding the computation of the influence matrix, it's noteworthy that other MTL methods typically employ specific heuristics to balance the loss from each task, and they do not compute an influence matrix.

Reviewer comment (Experiments):
Can we apply a naive co-training scheme (with random sampling of training tasks) to achieve the same effect as the proposed method? I figure it would serve as a good ablation study.

Author response:
The naive co-training strategy, referred to as Naive-MTL, is presented in Table 2. In addition, we include the results of random sampling in the global response.

Reviewer comment (Theory):
How often does assumption 1 hold in practice?

Author response:
Ideally, Assumption 1 consistently holds as the cosine similarity of a gradient vector with itself is always 1. In our specific context, confirming Assumption 1 requires precise computation of $\nabla L_j^i\left(\Psi^i\left(t_1\right)\right)$, a domain less explored for neural networks, to the best of our knowledge. Nonetheless, even though the exact verification of this assumption is intractable, the empirical results align with widely accepted observations in ML4CO. Thus, we assert that it holds empirically.

Reviewer comment (Evaluation):
Why are some entries in the diagonal blocks negative (whereas many off-diagonal entries are positive). It feels unintuitive that tasks from the same group have weaker influence on one another than tasks from different groups.

Author response:
In Fig 3(a), the positive or negative signs merely signify the direction of influence on other tasks and it's reasonable to have negative transfers within the same COP types if their scales are distinct too much.

    As for the positive entries in the off-diagonals, Fig 3(b) provides a more detailed illustration, and we have discuss this comprehensively in Section 4.2: "*Notably, tasks belonging to the same type of COP are highly influential towards each other due to the large variance of their influence values. Conversely, influences between different types of COPs are negligible, evident from the influence values being concentrated around 0.*"

Reviewer comment (Experiments):
Can the method be generalized to account for unseen tasks?

Author response:
In this work, we concentrate on the tasks at hand, and no zero-shot transfers are contemplated. The generalization ability of ML4CO methods poses a persistent challenge, often yielding unsatisfactory results when applied to unseen scenarios without additional efforts. The model derived by our method can serve as a pre-trained model for potential integration with fine-tuning techniques. This avenue holds promise for future exploration but falls outside the scope of the current paper.

Reviewer comment (Novelty):
Although applying MTL to COPs is less studied, the methodologies presented in this paper (e.g. similarity measure based on gradient information, MAB algorithms) have been well developed. This paper likely attempts to combine them within the context of COPs. In fact, extracting similarity measures using gradient information has been considered in the literature (Wang et al., 2020; Yu et al., 2020). Applying bandit algorithms in MTL has also been studied before (Mao et al., 2021).

Author response:
This work represents a pioneering effort in addressing multiple COPs simultaneously using a unified model. It goes beyond mere amalgamation of existing methodologies within the COP context, and our distinct contributions can be elucidated as follows:
    1. About the similarity measure: **a)** While (Wang et al. (2020) and Yu et al.) heuristically propose a gradient-based similarity measure and discuss its rationale, our approach starts from a foundational perspective through loss decomposition (Proposition 1). This establishes a theoretical foundation for the heuristics employed by (Wang et al. and Yu et al.). **b)** The similarity measure serves as a pivotal assumption (Assumption 1) that facilitates the design of our proposed algorithm, circumventing the need for gradient estimation in the mean value theorem (Equation (8) in the revised paper): $\nabla^TL^i_j({\Psi}^i(t_1)) = \sum_{t=t_1}^{t_2}\eta_t\mathbb{1}(a_t=T^i_j) \nabla L^i_j(\Theta^i(t))$
    2. Concerning the application of bandit algorithms in MTL: Both theoretically and practically, our methodology and training paradigm differ from Bandit-MTL (Mao et al., 2021), which necessitates collecting loss information from all tasks and training them collectively at each time slot. In contrast, our approach involves selecting and training a single task using the bandit algorithm, resulting in superior performance (refer to Table 2) and enhanced efficiency (refer to Table 3).

Reviewer comment (Theory):
This is a pure experimental paper without theoretical guarantees. I would appreciate some performance guarantees based on the well-developed theoretical results in MAB problems.

Author response:
It is essential to acknowledge that bandit algorithms thrive under specific assumptions about rewards, environmental properties, and more. The theoretical guarantees sought by the reviewer can be extrapolated if our proposed gradient-based reward and the environment, as simulated by the training process of neural networks, adhere to the corresponding assumptions. Although investigating the correlation between parameter variations induced by training neural networks and the bandit setting's environmental aspects is an intriguing avenue, we believe it surpasses the scope of this work.

Reviewer comment (Experiments):
In terms of the performance of the proposed method shown in the experiments, I don't think it "achieves much higher overall performance, ..., compared to standard training schedules". First, in the comparison under the same training budget, each COP naively receives an equivalent budget of $B/4$. Yet, $STL_{avg}$ still achieves the best gap in many tasks (Table 2). Second, for comparison under the same training epochs, although I agree this is not a fair comparison, the performance is just equivalent to 100-200 epochs of STL, not significantly larger than the naive calculation $1000/12 \approx 83$ epochs. I would appreciate it if the authors could provide more evidence about the superiority of the proposed algorithm.

Author response:
For the experiments,
    1. When considering the same training budget, the primary focus should be on the overall performance across all tasks, indicated by the Avg. Gap in Table 2, where our method excels. Notably, the superior performance achieved by $\text{STL}{\text{avg.}}$ is confined to relatively straightforward tasks, such as COPs with smaller problem scales. However, this approach tends to disproportionately allocate resources to these easier tasks, resulting in inadequate training for more challenging ones. A comparison between $\text{STL}{\text{avg.}}$ and $\text{STL}_{\text{bal.}}$ reveals that the latter achieves better overall performance (Avg. Gap) through more judicious resource allocation.
    2. In the context of the same training epochs, it is crucial to emphasize that the comparison involves a single model generated by our method after 1000 epochs versus 12 models produced by the corresponding STL training scheme, each trained for 1000 epochs. Our objective is to underscore the remarkable generalization ability and efficiency of the neural solver obtained through our method, manifested in the following aspects: **a)** Our method yields a model capable of solving diverse COPs, a feat challenging for models trained on specific COPs due to disparate network structures and distinct state transition logic. **b)** Considering performance on the same kind of COPs, our method outperforms in CVRP, OP, and KP, as evidenced by the "Mean Results" in comparison with COP-small, COP-median, and COP-large. This advantage is attributed to the fact that models trained on one problem scale may perform badly on different scales. Even for TSP, considered the simplest among the four problems, our method maintains competitive performance. **c)** The training time for our method, averaging 0.56 minutes per epoch (Table 3), is significantly shorter than the most resource-intensive alternative (see Table 1), yet it effectively solves diverse COPs.

Reviewer comment (Evaluation):
It seems that the correlation of training the same COP with different scales might be negative, e.g. training TSP-20 pm TSP-100. This is very counterintuitive to me. Could the author explain why this happens?

Author response:
This observed phenomenon aligns with the considerations surrounding the generalization ability of neural solvers for COPs and has been documented in prior studies, as illustrated in Fig. 5 in (Kool et al., 2019)  and Fig. 3 in [1]. Our empirical validation in Fig. 3 within this paper further substantiates this observation. Within the ML4CO community, the challenge of effectively generalizing a model trained on one problem scale to another remains an open question. This knowledge gap motivated our exploration and proposal of MTL methods, specifically designed to train neural solvers capable of handling COPs across various problem scales.

[1] Learning the travelling salesperson problem requires rethinking generalization