Learning Generalizable Models for Vehicle Routing Problems via Knowledge Distillation
Abstract
Recent neural methods for vehicle routing problems always train and test the deep models on the same instance distribution (i.e., uniform).
To tackle the consequent cross-distribution generalization concerns, we bring the knowledge distillation to this field and propose an Adaptive Multi-Distribution Knowledge Distillation (AMDKD) scheme for learning more generalizable deep models.
Particularly, our
AMDKD leverages various knowledge from multiple teachers trained on exemplar distributions to yield a light-weight yet generalist student model.
Meanwhile, we equip AMDKD with an adaptive strategy that allows the student to concentrate on difficult distributions, so as to absorb hard-to-master knowledge more effectively.
Extensive experimental results show that, compared with the baseline neural methods, our AMDKD is able to achieve competitive results on both unseen in-distribution and out-of-distribution instances, which are either randomly synthesized or adopted from benchmark datasets (i.e., TSPLIB and CVRPLIB).
Notably, our AMDKD is generic, and consumes less computational resources for inference.
Introduction
The Vehicle Routing Problem (VRP) is a class of NP-hard combinatorial optimization problems with a wide variety of practical applications, such as freight delivery [1], last-mile logistics [2] and ridehailing [3].
For decades, the problem has been studied intensively in computer science and operations research, with numerous exact and (approximate) heuristic algorithms proposed [4–7].
Although the heuristic algorithms are usually preferred in practice given their relatively higher computational efficiency, they heavily rely on hand-crafted rules and domain knowledge, which may still leave room for improvement.
As a promising alternative, deep (reinforcement) learning could be used to automatically learn a heuristic (or policy) for VRPs in an end-to-end fashion, which has aroused widespread attention in recent years [8–16].
Compared to the traditional ones, the learned heuristics based on deep models could further reduce computational costs while ensuring desirable solutions.
However, existing deep models suffer from inferior generalization with respect to distributions of node coordinates.
To be concrete, they often train and test neural networks on instances of the same distribution, mostly the uniform distribution, where deep models are able to achieve competitive results more efficiently than the traditional heuristics (e.g., [12]).
Nevertheless, when the learned policy is applied to infer the out-of-distribution (OoD) instances, the solution quality is usually low.
This cross-distribution generalization issue inevitably hinders the applications of deep models, especially because the real-world VRP instances may follow various and even unknown distributions.
A number of preliminary attempts have been made to tackle this generalization issue for VRPs, which mainly leverage (automatic) data augmentation [17–20] and distributionally robust optimization [21], respectively.
However, they are not optimal in our view, as the former always starts with a specified single distribution which may limit the resulted performance, while the latter needs to manually define major and minor instances.
Different from them, in this paper, we aim to enhance the crossdistribution generalization by transferring various policies learned from respective distributions into one, where we exploit knowledge distillation to learn more generalizable models for VRPs.
Specifically, we propose a generic Adaptive Multi-Distribution Knowledge Distillation (AMDKD) scheme for training a light-weight model with favorable cross-distribution generalization performance.
To impart broad yet specialized knowledge, we exploit multiple teacher models that are (pre-)trained on respective exemplar distributions.
Our AMDKD then leverages those teachers to train a shared student model in turns, inspired by the learning paradigm of human-beings.
Meanwhile, we also equip the AMDKD with an adaptive strategy to track the real-time learning performance of the student model on every exemplar distribution, which allows the student to concentrate on absorbing those hard-to-master knowledge, so as to strengthen the effectiveness of the learning.
Accordingly, our contributions are summarized as follows: (1) We bring the knowledge distillation to the field of neural combinatorial optimization and aim at improving the cross-distribution generalization for solving VRPs, which offers a promising perspective to transfer knowledge/policy learned from multiple models into one; (2) We propose the Adaptive Multi-Distribution Knowledge
Distillation (AMDKD) scheme, where we distill diverse knowledge from multiple teachers trained on exemplar distributions to yield a light-weight yet generalist student model, and also present an adaptive strategy for the student to better assimilate hard-to-master knowledge from difficult distributions; (3) We apply our generic AMDKD to two representative deep models, i.e., AM [10] and POMO [12], respectively.
Results show that, while consuming less computational resources for inference, our AMDKD performs favorably against the backbone models as well as other existing generalization methods for deep models, especially on the benchmark dataset CVRPLIB [22].
By further coupling with the efficient active search (EAS) [23], our AMDKD achieves the new state-of-the-art performance.
We also conduct a series of analysis to verify our designs.
Related work
Recently, neural methods based on deep models for VRPs have aroused widespread interest.
These methods are categorized as neural construction models and neural improvement models in general.
Neural construction models.
They exploit deep (reinforcement) learning to autoregressively construct the solution in an end-to-end fashion.
Vinyals et al.
[8] introduced the first RNN-based Pointer
Network (Ptr-Net) to solve TSP based on supervised learning.
Bello et al.
[24] then exploited reinforcement learning to train the Ptr-Net for TSP.
In [9], the Ptr-Net was then extended to solve CVRP.
Different from the above RNN-based methods, Kool et al.
[10] proposed the well-known Attention
Model (AM) based on the Transformer architecture [25].
Subsequently, many studies extended AM for routing problems, such as [12, 13, 26–30].
As a representative, Kwon et al.
[12] proposed the
POMO (Policy Optimization with Multiple Optima) and achieved significantly better performance.
In a recent work, POMO was adapted to an efficient active search (EAS) framework to further boost the performance during inference [23].
Besides, graph neural networks (GNNs) are also utilized to effectively learn and identify the graph-structured features of the problem instance [1, 11, 31–33].
Neural improvement models.
They iteratively improve the initial solution by exploiting deep (reinforcement) learning to assist or control the local search.
Chen and Tian [34] proposed the
NeuRewriter that learned a policy to partially rewrite the current solution.
Hottung and Tierney [35, 36] leveraged deep networks to learn to perform the large neighborhood search.
Wu et al.
[37] and Costa et al.
[38] proposed to control the 2-opt operation [6].
Ma et al.
[14] further upgraded the deep model of Wu et al.
[37] to dual-aspect collaborative Transformer (DACT) with much superior performance.
The DACT method was further enhanced in [39] in order to learn a ruin-and-repair operation for pickup and delivery problems.
Generally, improvement methods consume much longer inference time than construction ones to achieve higher solution quality.
Cross-distribution generalization.
The above neural methods for VRPs often train and test the deep models on the same instance distribution.
Although the state-of-the-art deep models perform close to or even surpass the strong traditional heuristics, they are incapable to generalize the learned policy to other distributions, which seriously impairs their practical applications.
Therefore, the cross-distribution issue has gradually gained more attention [17–21].
Among the early attempts in this line of research, Xin et al.
[18] proposed a generative adversarial network (GAN) based framework to generate hard-to-solve instances for training the model.
Wang et al.
[20] leveraged the game theory and proposed the Policy Space Response Oracle (PSRO) framework to simultaneously learn a trainable solver and an instance generator.
Zhang et al.
[17] designed a hardness-adaptive TSP instance generator and adopted curriculum learning to train the model.
These methods primarily emphasized on data augmentation or are limited to TSP.
Different from them, Jiang et al.
[21] adopted CNN to acquire distribution-aware features and exploited group DRO (Distributionally
Robust Optimization) to enhance model robustness against distributions.
However, this method needs to manually define major and minor instances.
Note that we acknowledge that the generalization to different problem scales (or sizes) is also important, which we leave as the future research direction.
Preliminaries and notations
We first present the formulation of the studied VRPs and classic distributions.
Then we introduce how deep models are used to solve them, followed by the basic rationale of knowledge distillation.
3.1 VRPs and their distributions
We define VRPs over a complete graph G = {V , E} , where v i ∈ V represents the (customer) node, e ( v i , v j ) ∈
E represents the edge between two nodes, and C [ e ( v i , v j )] represents the cost (we use length in this paper) of the edge.
By referring tour τ (a.k.a. solution) to a permutation of nodes in V , the objective is usually to find the optimal tour τ ∗ with the least total cost(length) over a finite search space S containing all possible tours, which could be formulated as Eq.
(1) in general,
[Equation]
For different VRP variants, such objective may be subject to different problem-specific constraints [40, 41], where multiple sub-tours may exist in a valid tour τ .
Following the recent literature [10, 12, 23], we focus on two representative VRPs, i.e., TSP and CVRP, respectively.
A feasible tour for TSP considers a vehicle visiting each node in V once and only once.
As an extension from a vehicle to a fleet, CVRP considers an extra depot node v 0 and a capacity limit Q for any given vehicle, where each customer node v i ( i = 1 , ..., n ) is associated with a demand request δ i .
A feasible tour for CVRP consists of multiple sub-tours, each of which represents a vehicle in the fleet departing from the depot, serving a subset of nodes, and finally returning to the depot.
The total demand of each sub-tour must not exceed the capacity Q , and all nodes except for the depot must be visited once and only once.
In Figure 1, we visualize a number of VRP instances following various distributions from the literature [10, 21, 43], including the TSPLIB [42] and CVRPLIB [22] benchmark datasets.
As can be observed, the node coordinates of a VRP instance may follow complicated and even unknown distribution, which considerably intensifies the hardness for solving.
While the recent neural methods report superior performance to the traditional heuristics on some fixed distributions, it is unfortunate that they are more sensitive to distribution shift.
It is thus of great importance and interest to develop powerful deep models that can simultaneously handle as many diverse distributions as possible.
3.2 Tour construction by deep models
We focus on neural construction methods for VRPs, which usually exploit deep neural networks to sequentially construct the tour.
In light of this, the solving procedure is mostly modelled as a
Markov Decision Process (MDP), where the Transformer styled [10] architectures following the encoder-decoder structure are often adopted as the policy network.
Typically, the encoders project the nodes of the instance into node embeddings for feature extraction.
Afterwards, the decoder builds the
Expansion, l) Implosion, m) Explosion, (n) Grid, where instances (a)-(g) are from TSPLIB [42] and
CVRPLIB [22].
In this paper, we consider instances following distributions (h)-(j) for training and other unseen distributions (k)-(n), as well as unseen benchmark datasets for testing.
tour τ based on learned node embeddings and the partial tour a 1: t − 1 constructed previously.
At time step t of the MDP, the decoder picks an unvisited node as the action a t where invalid ones are masked for feasibility.
The procedure is repeated until the whole tour is completed, which is factorized as,
[Equation]
where p θ is the policy parameterized by θ , and ℓ denotes the final time step in MDP.
For TSP, ℓ = n ; for CVRP, ℓ ≥ n since the depot can be visited more than once 1 .
The total reward is defined as the negative of the tour length, i.e., − L ( τ |G ).
This is essentially consistent with the objective in Eq.
3.3 Knowledge distillation
Knowledge distillation [44] is a kind of teacher-student training paradigm that aims at transferring knowledge from a (group of) complex teacher model(s) θ T , to a succinct student model θ S .
The recent research findings reveal that knowledge distillation is not only able to effectively learn a lighter student network from larger teacher network(s) [45, 46], but also has potential to improve the generalization [47, 48] even over its teacher(s) [49–51].
Typically, the teacher models are pre-trained, and the student model compares the output of teacher model(s) with its own and considers it as a supervisory signal.
Formally, the student network is trained with the goal of minimizing a weighted combination of the distillation loss L KD and the original task loss
L Task as follows,
[Equation]
where α ∈ [0 , 1] .
The L KD with respect to one or more teachers ( N T ≥ 1 ) is generally formulated as,
[Equation]
where x is the training data from X and ϕ ( · ) measures the statistical distance between the teacher and the student, such as the Kullback-Leibler divergence KL ( p
[Equation]
= (cid:80)
i
[Equation]
(log
[Equation]
Methodology
Consider what happens in a classroom when students study multiple modules.
Typically, they (students) learn only one module (exemplar distribution) at a time from a professional teacher, and then go on to the next one until all modules are mastered.
When students do poorly on a quiz after class (validation dataset), they will be required to study that module more frequently in order
to
[Figure]
become generalists.
Motivated by this, we propose the Adaptive Multi-Distribution Knowledge
Distillation (AMDKD) scheme that could efficiently learn more generalizable models for VRPs.
Previously, deep models were trained on a single distribution, e.g., the Uniform in [12–14].
More recently, attempts have been made to expand the training data by augmenting atypical (minor) [21] or hard instances [17, 18].
Differently, AMDKD allows expert knowledge for tackling distinct distributions to be transferred into a single yet generalist model via distillation.
With a light network and high inference speed, the resulted model is supposed to perform favorably against existing methods.
Moreover, AMDKD is generic, and could be used to boost various deep models for VRPs.
4.1 Overall structure and teacher (pre)-training
The overview framework of our AMDKD is illustrated in Figure 2.
Given an existing deep model (e.g., AM [10]), AMDKD first performs teacher training (or directly use pre-trained ones) to obtain a teacher model for each distribution listed in a set of exemplar ones.
Then in each knowledge transfer epoch, AMDKD picks a specific distribution d and its teacher θ T d to train a student network.
To facilitate effective learning, the likelihood of picking each distribution is adaptively updated according to the current performance of the student on a validation dataset.
Finally, AMDKD performs on-policy distillation, allowing the student network θ S to sample RL trajectories (i.e., tours { τ θ S } ) for training, and calculates the two losses (i.e., KD loss and Task loss) for an update.
We consider Uniform, Cluster, and Mixed (mixture of uniform and cluster) as exemplar distributions to train teacher networks in this paper.
Consequently, a set of well-trained teachers with parameters θ T = { θ TU , θ TC , θ TM } are attained for each distribution.
Despite only limited distributions are exploited throughout training, the learned student network is expected to absorb generic and robust knowledge from teachers that may effectively generalize to unseen distributions such as Expansion, Implosion,
Explosion, and Grid (visualization in Figure 1).
Our motivation here is that each teacher only needs to master a specific distribution, and by working together via our distilling scheme, they can educate a generalist student.
Note that the exemplar distributions used here could also be substituted with others (see Section 5.3).
Furthermore, since our designed distillation scheme is model-agnostic, the teacher network can follow most of existing architectures.
In this paper, we assess our scheme by applying it to two representative construction methods, i.e., AM [10] and POMO [12], respectively.
4.2 Adaptive multi-distribution student training
Given the well-trained teachers on exemplar distributions, our AMDKD student selects one teacher and one distribution in each knowledge transfer epoch and gradually learns to make appropriate decisions.
Note that there is another line of works that argue for the simultaneous use of multiple teachers in the distillation process [52, 53].
In their strategy, all teachers are engaged to provide a weighted loss to train the student as aforementioned in Section 3.3.
However, such design may require teachers to be trained on a homogeneous task primarily with supervised learning, which does not suit our heterogeneous distributions with reinforcement learning (see Appendix C for results).
p
Student network.
We stipulate that the student shares a similar architecture with its teachers, but can reduce its network parameters as needed to speed up the inference.
Nevertheless, we find in
Section 5.3 that a larger student model usually leads to a better performance.
Therefore, there is a trade-off between solution quality and computational cost.
In this paper, we consider reducing the dimension of the node embeddings from 128 (teacher) to 64 (student), resulting in a reduction of 61.8% and 59.2% in the model parameters for the adaption with AM and POMO, respectively.
We also considered lowering the number of encoder layers, but the results were below the expectation.
The adaptive multi-distribution distilling strategy.
In each epoch, AMDKD selects one distribution and its corresponding teacher model.
At the beginning, it selects each distribution with an equal probability.
After epoch E ′ (a hyper-parameter), the probability would be adaptively adjusted according to the performance of the student on the given validation datasets Z U , Z C , Z M (each with 1,000 instances) for each exemplar distribution.
The likelihood p adaptive of selecting distribution d ∈ { U, C, M } is proportional to the exponent value of the gaps to the LKH solver [4] as follows,
[Equation]
where |D| is the number of exemplar distributions; { τ θ S }| Z d and { τ solver }| Z d refer to the tour set generated by the student θ S and the LKH solver on dataset Z d , respectively.
The validation sets are fixed, hence the LKH solver only needs to run once to attain { τ solver }| Z d for d ∈ { U, C, M } .
Meanwhile, we note that p adaptive may converge during distillation (see Appendix C), which means that it is possible to stop such evaluation early and reuse the stabilized one for an even faster training.
Loss function.
Following Eq.
(3), we leverage the task loss ( L Task ) and the KD loss ( L KD ) to jointly train the student with α =0 .
Pertaining to L Task , we define the task loss as follows,
[Equation]
where the training instances G are sampled following the selected distribution d , and the tours τ are constructed via the student network θ S according to Eq.
In AM and POMO, the REINFORCE algorithm [54] is used to estimate the gradients for the above loss function and we follow exactly the same way as per their original designs.
Pertaining to L KD , it is defined to encourage the student to imitate how a teacher network sequentially selects the nodes.
Specifically, given the instance G sampled from distribution d and a tour τ θ S constructed by the student θ S following its own policy, our
AMDKD leverages p θ T d ( τ θ S |G ) suggested by the teacher network θ T d , to compute L KD that measures the similarity of the probability distributions between teacher and student using the KL divergence,
[Equation]
We summarize our AMDKD in Algorithm 1.
Note that our AMDKD follows the on-policy scheme where the tours τ θ S are output by the student that currently performs learning.
As an alternative, such tours can also be output by the teacher (i.e., off-policy scheme).
However, it performs inferior to our on-policy one (see Section 5.3).
Finally, we note that the student learned by our AMDKD could also be coupled with the efficient active search (EAS) [23] during inference to further boost the performance.
Accordingly, the resulting AMDKD+EAS achieves a new state-of-the-art performance.
[Table]
U
[Table]
C
[Table]
Experiments
We conduct experiments on TSP and CVRP with n = 20, 50, and 100 nodes similar to [12, 14].
As aforementioned, we adopt Uniform, Cluster and Mixed (mixture of uniform and cluster) as exemplar distributions for training; Expansion, Implosion, Explosion, and Grid as the unseen distributions for testing.
For the above 7 distributions, we follow [10, 21, 43] to generate the respective instances (details are presented in Appendix A).
All experiments are conducted on a machine with NVIDIA
RTX 3090 GPU cards and Intel Xeon Silver 4216 CPU at 2.10GHz.
Our implementation in PyTorch are publicly available 2 .
Some additional analysis and discussions can be found in Appendix C.
Training and hyper-parameters.
We apply our AMDKD to two representative deep models, i.e.,
AM [10] and POMO [12], termed as AMDKD-AM and AMDKD-POMO, respectively.
For the teacher (pre)-training phase, we directly follow the RL algorithm, network architecture, and other hyper-parameters as suggested in the original backbone method.
For the student distillation phase, we use batch size B = 512 3 , and task-specific hyper-parameters T = 250, E ′ = 500 for AMDKD-AM and
T = 20, E ′ = 1 for AMDKD-POMO, respectively.
By default, the dimension of the node embeddings in our student networks AMDKD-AM and AMDKD-POMO is reduced from 128 (teacher) to(student) to enjoy a faster inference speed.
The total numbers of needed training epochs vary with the problem size.
Regarding AMDKD-AM, we use 2,000, 5,000, 10,000 for problem sizes 20, 50, 100, respectively (both TSP and CVRP); regarding AMDKD-POMO, we use 5,000 (TSP-20), 10, (CVRP-20, TSP-50, TSP-100), and 30,000 (CVRP-50, CVRP-100).
The Adam optimizer is used with learning rate 1e-4.
Training time also varies with the problem size.
Taking CVRP-100 as an example, one epoch takes about 4 minutes for AMDKD-AM and 1.4 minutes for AMDKD-POMO.
Inference.
For AMDKD-AM, it samples 1,280 solutions following AM [10]; and for AMDKDPOMO, we adopt the greedy rollout with × 8 augments following POMO [12].
All experiments are conducted on test datasets containing 10,000 instances per distribution.
Unless otherwise stated, the gaps are computed w.r.t.
the strong traditional solvers Gurobi [5] (for TSP) and LKH [4] (for CVRP).
5.1 Effectiveness analysis of AMDKD
We first study the effectiveness of our AMDKD when applied to backbone model AM and POMO for TSP and CVRP, respectively.
In Table 1, we display the gaps of the learned student and their respective teachers on unseen instances following three exemplar distributions (denoted as G U , G C , and G M ) and also report the parameter sizes of each model.
We can see that when the teacher model is trained on a specific distribution, it does not generalize well on the other ones (especially for AM), resulting in poor overall performance (if referring to the average gaps).
In contrast, our AMDKD could alleviate this issue effectively.
While the learned student model by AMDKD reduces the size of
[Table]
the teacher model from 0.68 to 0.26 M (a 61.8% reduction) for AM and from 1.20 to 0.49 M (a 59.2% reduction) for POMO, it still exhibits improved overall performance for both TSP and CVRP on all the three sizes.
Pertaining to AMDKD-AM, our AMDKD student not only significantly outperforms its three teachers for both TSP and CVRP, but also exhibits even lower gaps on the distribution where the teacher was previously trained in most cases.
Pertaining to AMDKD-POMO, although POMO itself presents a better generalization, our AMDKD can still boost its overall performance with a much lighter student network for both TSP and CVRP.
The above results validate that our AMDKD does not overfit to a single distribution, and successfully learns a lightweight yet generalist student model under the guidance of multiple teachers with expertise in different exemplar distributions.
5.2 Generalization analysis of AMDKD
We now assess the cross-distribution generalization of our AMDKD.
In addition to the two backbone models, we also compare with the following methods, i.e., 1) other deep models including- 1.a)
DACT [14], a neural improvement model that learns to perform local search; 1.b) LCP [13] (TSP only), a hybrid two-stage method that combines improvement and construction strategy; and 2) other methods specialized for generalization including- 2.a) HAC [17] (TSP only), a fine-tuning framework for AM by leveraging instances with different hardness; 2.b) DROP [21], a distributionally robust optimization based method to enhance POMO; 2.c) GANCO [18], a framework that enhances AM by learning a generative adversarial network to produce hard-to-solve training instances; and 2.d)
PSRO (a.k.a.
LIH)
[20], an improvement method with a game theory based policy space response oracle framework.
More implementation details of them are provided in Appendix B. Note that for DROP,
GANCO, and PSRO, we only compare the results on several benchmark instances reported in their original papers (see Table 3), since their codes (for re-training) are not publicly available.
Distribution mixture augmentation.
Given that baselines AM, POMO, DACT, and LCP were originally trained on the uniform distribution only (resulting in poor generalization), we re-train the above models (with # ) on a mixed dataset containing instances from the three exemplar distributions, to ensure a fair comparison.
We also apply this mixed dataset as the initial training instances for HAC (with # ).
Though such distribution mixture augmentation improves the generalization of construction methods POMO and AM on larger instances, we notice that it does not always happen so, especially for the methods that have an improvement component (e.g., DACT and LCP).
We present detailed results and discussions regarding this finding in Appendix B.
Meanwhile, for AM # and POMO # , we report their performance with network dimensions of both 128 and 64 (same size as our AMDKD student) for a more comprehensive comparison.
We assess the following metrics, 1) parameter size; 2)
[Table]
average gap on unseen in-distribution instances, i.e., G ID ; 3) average gap on unseen out-of-distribution instances, i.e., G OoD ; 4) average time for solving an individual instance and total time for solving 10,000 instances with batch parallelism allowed (one GPU) 4 .
As can be observed in Table 2, our AMDKD is able to achieve competitive results on both unseen in-distribution and out-of-distribution instances, and performs favourably against all baselines.
Pertaining to TSP, our AMDKD-AM (64) significantly outperforms the baseline AM # (128) and
AM # (64) on all sizes in terms of both G ID and G OoD with fewer parameters and higher inference speed.
Meanwhile, it also consistently beats the recent baseline HAC # (designed to boost the generalization of AM) on all sizes in terms of both two gaps and the time efficiency.
For the backbone model
POMO, which suffers from less generalization concern, the baseline POMO # (128) can already attain near-optimal solutions to TSP instances with a desirable generalization performance.
However, our AMDKD (64) is able to achieve competitive generalization results with nearly half of the model size and higher inference speed.
When we reduce the network dimension of POMO # to 64 (as our
AMDKD), POMO # (64) performs much inferior to our AMDKD-POMO (64).
Besides, both our
AMDKD-AM (64) and AMDKD-POMO (64) can significantly outperform improvement methods including DACT and LCP with lower gaps while exhibiting much faster inference.
Pertaining to
CVRP, similar patterns can be observed, where our AMDKD-AM (64) and AMDKD-POMO (64) consistently deliver favourable generalization performance against all the compared baselines.
Furthermore, we show that by coupling our AMDKD-POMO (64) model with the recent efficient active search (EAS) [23], the resulting AMDKD+EAS ( T =100) attains considerably superior performance that even surpasses LKH3 on CVRP-100 with much shorter time.
Given the advances of active search for a particular instance to be inferred, AMDKD+EAS allows the model learned by our AMDKD with strong generalization to continuously improve its performance during inference (longer T will yield even better performance), which leads to a new state-of-the-art hybrid solver.
We now evaluate the generalization of our AMDKD on the benchmark, i.e., TSPLIB and CVRPLIB, which contain various instances of unknown distributions and larger size.
As shown in Table 3, our
AMDKD-AM (64) significantly boosts the generalization of AM (128) and also outperforms the baseline AM # (128) as well as other generalization methods that take AM (128) as the backbone model (GANCO and HAC).
Similarly, our AMDKD-POMO (64) also significantly beats the other baseline methods, except for POMO # (128) on TSPLIB.
Nevertheless, regarding the harder CVRP instances from CVRPLIB, our AMDKD-POMO (64) could yield significantly better performance.
Finally, we note that the AMDKD+EAS achieves the smallest gaps for both TSP and CVRP among all baselines.
We refer to Appendix D for full results and discussions.
5.3 Further analysis of AMDKD
Effects of different components.
We conduct ablation studies on CVRP-50 and TSP-50 to verify the designs of our AMDKD, where we consider removing the proposed adaptive strategy, the KD loss L KD , and the task loss L Task , respectively.
As displayed in Table 4, the use of adaptive strategy further enhances the learning effectiveness, and the two losses play a prominent role in the phase of gradient update.
We also verify that when compared with our on-policy scheme, the off-policy variant (as mentioned in Section 4.2) saliently impairs the distillation performance.
Effects of student network architectures.
In Figure 3, we exhibit the boxplots of the averaged gaps when using different student network architectures (different dimensions of the entire network and different number of layers in its encoder) of AMDKD-POMO for solving CVRP-50.
As revealed, the larger the model, the better its performance.
Hence, we acknowledge that our AMDKD can be further boosted when the student (64) shares the same architecture with its teachers (128).
Specifically, for CVRP-50, our AMDKD-POMO (128) exhibits an average optimality gap of 0.81%, which is
[Table]
[Figure]
[Table]
lower than the 0.90% of AMDKD-POMO (64).
However, this improvement comes at the expense of increased computational costs in three aspects: 1) more training time - AMDKD-POMO (128) needs almost 1.7 times (12 days vs 7 days) more training time; 2) more parameters - AMDKD-POMO (128) has about 2.4 times (1.20 M vs 0.49 M) more parameters; 3) slower inference time - AMDKD-POMO (128) infers about 1.2 times (1.3 min vs 1.1 min) slower than AMDKD-POMO (64).
Hence, we note that there is a trade-off between solution quality and computational cost.
Effects of different exemplar distributions.
We now vary the distributions which were used as the exemplar ones to validate that AMDKD could still be effective.
We take the POMO and CVRP-50 as an example.
To distinguish from the original version (AMDKD-POMO), we term the model taking
Expansion, Implosion, Explosion, and Grid as the exemplar distributions as AMDKD-POMO*, and gathered the results in Table 5.
It shows that the overall performance of AMDKD-POMO* is still better than POMO trained on these four distributions respectively.
Hence, our AMDKD is able to consistently improve the cross-distribution generalization with different exemplar distributions.
Conclusions and future work
In this paper, we propose an Adaptive Multi-Distribution Knowledge Distillation (AMDKD) scheme to alleviate the cross-distribution generalization issue of deep models for VRPs.
Different from the existing generalization methods, we aim to enhance the distribution generalization by transferring various policies learned from exemplar distributions into one via an efficient knowledge distillation scheme.
To facilitate effective learning, we design an adaptive strategy to train a single yet generalist student network by leveraging multiple teachers in turns.
The experiment results exhibit competitive performance of our AMDKD in generalizing to other unseen out-of-distribution instances (randomly generated or from benchmarks), which also consumes less computational resources.
While our
AMDKD is generic, a potential limitation is that its boost is not guaranteed to be always significant across all unseen distributions for all backbone models.
For future work, we will investigate, 1) generalizing AMDKD for different/larger problem sizes; 2) considering the improvement models like DACT [14] as the backbone; 3) performing online distillation to jointly and efficiently train the teachers and the student models [55]; 4) assessing the impact of the quality of the validation dataset on the distillation; and 5) enhancing the interpretability of AMDKD [56].
Acknowledgments and Disclosure of Funding
This work is supported in part by the National Key R&D Program of China (2018AAA0101203), the
National Natural Science Foundation of China (62072483), and the Guangdong Basic and Applied
Basic Research Foundation (2022A1515011690, 2021A1515012298); in part by the Agency for Science Technology and Research Career Development Fund (C222812027), and the IEO Decentralised
GAP project of Continuous Last-Mile Logistics (CLML) at SIMTech (I22D1AG003).
A Details of the considered distributions
In this paper, we consider various distributions for the node coordinates in VRPs, followed which we randomly generate instances for both training and testing.
Below we present details on how to generate those instances.
Specifically, we follow the recent work [10] and benchmark dataset
TSPLIB [42] to generate instances of Uniform, Cluster and Mixed distributions, and follow the settings in [21, 43] for the Expansion, Implosion, Explosion, and Grid ones.
Uniform distribution.
It considers uniformly distributed nodes.
Following [10], we generate the two-dimensional coordinates ( x, y ) of each node by sampling from a uniform space U ([0, 1] 2 ).
An exemplary instance is displayed in Figure 1(h).
Cluster distribution.
It considers multiple ( n c ) clusters, where we set n c = 3 .
In specific, each cluster follows a normal distribution N ( µ , σ 2 ) 2 with the mean sampled uniformly, i.e., µ ∼ U ([0.2, 0.8] 2 ) and the standard deviation σ = 0.07.
According to the 3 σ rule, each node has a 99.7% probability of being generated in the [0, 1] 2 square region, and outliers will have their coordinates re-modified, where values less than 0 are changed to 0 and those greater than 1 are changed to 1, to ensure all coordinates are constrained to ([0, 1] 2 ).
An exemplary instance is displayed in Figure 1(i).
Mixed distribution.
It considers a mixture of the two distributions above, each with half of the nodes.
For the latter, we only consider n c =1 cluster.
An exemplary instance is displayed in Figure 1(j).
Expansion distribution.
It considers a linear function to mutate the nodes in Uniform distribution.
Gvien a randomly generated linear function y
=
ax + b , all nodes, orthogonal to the linear function within the distance r ( r = 0 . 3 ), are moved away from their original coordinates to a farther position, whose orthogonal distance is r + γ , where γ obeys an exponential distribution with the rate parameter λ = 10 , i.e. γ ∼ E ( λ ) .
Regarding the linear function, we first sample b (intercept) from [0, 1], then a (slope) is sampled uniformly from [0, 3] (if b < 0.5) or [-3, 0] (if b ≥ 0.5).
Finally, we normalize all node coordinates X as follows to ensure that they are constrained to [0, 1] 2 ,
[Equation]
where X ′ denotes the normalized coordinates.
An exemplary instance is displayed in Figure 1(k).
Implosion distribution.
It considers an implosion to mutate the nodes in Uniform distribution.
To simulate an implosion, it first samples a centroid ϵ i , then gathers all nodes within the circle of ϵ i (with the radius R ic = 0 . 3 ) together towards a new circle with the same centroid
ϵ i but a (randomly sampled) smaller radius ( R i ≤ 0 . 3 ).
An exemplary instance is displayed in Figure 1(l).
Explosion distribution.
It considers to mutate the nodes in Uniform distribution by imitating the particles affected by an explosion.
Similar to Implosion, it first randomly samples a centroid.
Then, instead of gathering all nodes towards the centroid in Implosion distribution, it moves away those nodes from the circle (radius R ec = 0 . 3 ) and explode them outside the circle, which follow the direction vector between the centroid ϵ e and the corresponding nodes.
The additive distance γ is randomly sampled from an exponential distribution with a rate parameter, i.e. γ ∼ E ( λ ) .
All nodes are them normalized using Eq.
An exemplary instance is displayed in Figure 1(m).
Grid distribution.
It considers to mutate the nodes in Uniform distribution by imposing a grid permutation.
We first generate the four vertex of a square with the width and height equal to
R g ( R g = 0 . 3) and then place it within the region [0, 1] 2 .
All the pre-generated nodes inside the box are re-arranged as a quadratic grid instead.
In this case, all nodes are constrained to [0, 1] 2 .
An exemplary instance is displayed in Figure 1(n).
The above distributions are considered in both TSP and CVRP.
For extra settings in CVRP, we follow the convention [10, 12, 14].
In specific, the demand δ i of each node is sampled uniformly from U (1,
2, · · · , 9) and the capacity Q of the vehicle varies with the problem scale, where we set Q 20 = 30,
Q 50 = 40 and Q 100 = 50.
For instances from CVRPLIB, we exactly follow their settings.
B Details of compared baselines
Implementation Details.
We compare our AMDKD with various types of baselines.
Regarding the neural baselines, we re-train LCP [13], HAC [17] and DACT [14] on our machine based on the code that are publicly available on Github.
By default, we follow their original settings and the suggestion on the hyper-parameters.
More details of the baselines are presented below.
• Gurobi [5]: we use Gurobi to obtain the optimal solutions to TSP instances, which are implemented under the default settings.
• LKH [4]: For CVRP, it is usually hard to obtain the optimal solutions.
Thus, we use the strong LKH solver to find near-optimal solutions.
Note that the LKH solver is also a widely used baseline to evaluate and compare the recent learning based methods for VRPs, where we run it following the conventions in [10, 12, 14, 23].
• LCP
[13]: LCP is a two-stage method, where a seeder generates diverse initial solutions and a reviser rewrites the current solutions partially.
We re-train the LCP on a mixed dataset containing instances from the three exemplar distributions.
For inference of TSP-50 and TSP-100, we employ the LCP* (the best version reported in [13]) with two revisers (the lengths of the tour for revision are set to ℓ r 1 =10 and ℓ r 2 =20) and a sampling strategy (1,280), and set the total number of revision iteration T r to 45 (i
e., T r 1 =25, T r 2 =20, respectively).
For TSP-20, we exploit one reviser ( ℓ r =10) since the revision length must be less than the problem size according to its design, and set the number of revision iteration T r to 10.
• HAC [17]: HAC designs a hardness-adaptive Gaussian instances generator to produce instances to fine-tune the given pre-trained AM model.
In its original design, the dataset used for fine-tuning contains half instances uniformly distributed and the other half produced by its own generator.
In this paper, we substitute the instances of uniform distribution with a mixed dataset containing instances from the three exemplar distributions.
• DACT [14]: DACT learns to guide the pairwise operator to perform local search.
We adopt the 2-opt version since it reports the best result for TSP and CVRP according to its original paper [14].
We re-train DACT on a mixed dataset containing instances from the three exemplar distributions, and set the iteration number to 1,280 for inference.
Effects of distribution mixture augmentation.
To ensure fair comparisons, we re-train the baselines on a mixed dataset containing instances from the three exemplar distributions (with # ).
However, we notice that this simple distribution augmentation does not always lead to a better generalization, espicially for the methods that have an improvement component.
For example, regarding the improvement method DACT, we find that DACT # performs even worse than DACT trained on Uniform distribution for CVRP; and regarding the hybrid method, LCP (U) performs slightly better than LCP #
[Table]
on TSP-100.
Nevertheless, this distribution augmentation improves the generalization of construction methods POMO [12] and AM [10] on larger instances, i.e., TSP-100 and CVRP-100.
Why AMDKD could be better than distribution mixture augmentation?
Recall that we conclude from Table 2 that our AMDKD usually achieves better cross-distribution generalization performance when compared to the above distribution mixture method (# models).
Though this still remains a open question, we list some possible intuitions on why AMDKD works better as follow:
• Different exemplar distributions may have different levels of difficulty for solving.
Without additional intervention, reinforcement learning tends to learn those easier things to get a higher reward.
This means that training on mixed data may possibly bias the learning towards distributions that are easier to solve (a "winner-take-all" issue).
On the contrary, our AMDKD selects a specific teacher model based on the weakness of the current student model (by our adaptive strategy), which encourages it to learn those hard-to-solve distributions.
And this adaptive strategy echoes how humans learn knowledge, where more time is always required for harder subjects.
• Directly training on a mixed dataset may not be efficient or stable.
On one hand, it would be hard for deep reinforcement learning to directly learn good patterns from mixed data that follow different distributions, due to the possibly limited representation capability of the neural networks for handling such hard optimization problems, even without the diversity in distributions.
On the other hand, the tasks for different distributions may have different reward ranges (such as the route length), which may cause instability in the RL training.
For example, the average total rewards for solving CVRP-100 instances following Uniform and Cluster distributions are around 16 and 7, respectively.
Our AMDKD tackles this issue in a way that only one distribution is leveraged in a training epoch.
Further empowered by the knowledge distillation, our AMDKD framework efficiently and effectively transfers useful knowledge (patterns) from various teacher models to a unified and light student model.
C Additional analysis of AMDKD C.1 Effects of multiple teacher co-training.
Recall that AMDKD selects only one distribution and its corresponding teacher in each epoch.
Different from ours, some existing multi-teacher knowledge distillation approaches exploit multiple teachers simultaneously in each epoch.
We term such strategy as MT, whose loss function follows
Eq.
In Figure 4, we draw the boxplot of the overall gaps (on all seven test distributions for
CVRP-50) of AMDKD-POMO and its MT version, and compare the results using the Wilcoxon test.
As clearly demonstrated, allowing unprofessional teachers to advise for distributions in which they are not specialized will interfere with the process of knowledge distillation, inducing significantly inferior performance compared to ours.
C.2 Effects of validation datasets.
Recall that the adaptive probability of teacher selection in Eq.
(5) is calculated by the real-time performance of student model on the validation datasets, we further investigate whether the size of the validation datasets V will largely influence AMDKD.
As displayed in Figure 5 and Figure 6, increasing the size of V (from 1,000 to 2,000) slightly improves the performance of AMDKD
(but with no statistical significance), whereas decreasing the size of V (from 1,000 to 500) slightly impairs the performance of AMDKD (also with no statistical significance), which indicates that the good performance of AMDKD may not rely on the size of the validation datasets.
Meanwhile, performing student model evaluation in each epoch inevitably introduces additional computation cost, where larger size of the validation datasets will cause longer training time.
However, we note that the increment of validation in training time is acceptable when we use V = 1,000.
Taking AMDKD-AM training on CVRP-100 as an example, the total evaluation time is 2.7s (0.9s per exemplar distribution on average) for each epoch, which is approximately 1% of the total training time (i.e., 4 min).
What’s more, we note that the likelihood of selecting different teacher models would eventually converge to a stable one, which means that we may stop such evaluation early to speed up the training further if needed.
For example, for training AMDKD-AM on CVRP-100 (see Figure 7), we may stop the student evaluation early at around 3,000 epochs.
[Figure]
epochs (AMDKD-AM on CVRP-100).
C.3 Effects of different hyper-parameters.
We further discuss the influence of the hyper-parameters on the performance of AMDKD.
• The starting epoch of the adaptive teacher selection strategy ( E ′ ):
it indicates the epoch to start our adaptive strategy for teacher selection.
We include this hyper-parameter because preliminary experiments revealed that there could be a point in the learning curve (i
e., the reward convergence curve) where the training curves without and with (starting at the first epoch E ′ =1) the adaptive strategy may meet.
This suggests that there might be a sweet spot
to implement the proposed adaptive strategy.
For AMDKD-AM, the sweet spot is around E ′ =500.
And for AMDKD-POMO, we do not observe such a pattern and thus we use E ′ )
=1.
In figure
we provide an example of how E ′ will affect the performance of AMDKD-AM.
• Number of steps per epoch ( T ): it indicates how long will the student model learn from the selected teacher before it possibly switches to a new one, which should not be too small or too large.
As for POMO, the original T is about 20, and we did not change it.
As for AM, the original T is about 2,500, and we empirically reduce it to 250 for a better trade-off.
• The total number of training epochs ( E ): it mainly follows the settings of the original backbone.
In this paper, we employ different training epochs for different sizes and tasks since the hardness of the task itself may grow with size, which may require more steps to converge.
Training curves of AMDKD-POMO (for CVRP) are depicted in Figure 9.
[Figure]
C.4 Stability studies of AMDKD.
To demonstrate the stability of our experiment results, we take CVRP-50 as an example and independently run our trained AMDKD-AM and AM # for 10 times, where we adopt different random seeds during the sampling process.
As shown in Figure 10, both AMDKD-AM and AM # exhibit extremely small fluctuations (even less than 0.001) when running with different seeds.
Based on the
Wilcoxon test, our AMDKD-AM significantly (with p -value < 0.001) outperforms AM # on every unseen in-distributions and OoD distributions.
As for AMDKD-POMO and POMO # , the greedy decoding strategy is adopted, hence the results are expected to be stable with almost no fluctuation.
[Figure]
D Detailed results on benchmark datasets
We present the detailed results of benchmark dataset in Table 8 (TSPLIB) and Table 9 (CVRPLIB), respectively.
As displayed, models that have only been trained on the uniform distribution, i.e.,
AM and POMO, perform extremely poorly when inferring instances that may follow unknown distributions from the benchmark.
The upgraded POMO # significantly outperforms POMO, which seemingly alleviates this issue through training on our exemplar distributions, however, this simple strategy does not work well with AM.
Regarding the prior cross-distribution generalization methods including GANCO, HAC, PSRO and DROP which show the potential to improve AM or POMO, their results are still far from satisfactory, where the performance of DROP are even inferior to POMO # trained on our exemplar distributions.
Nevertheless, our AMDKD not only outperforms all these baselines, but also brings a much more significant improvement over the backbone AM and POMO than those baselines do.
Meanwhile, our AMDKD-POMO exhibits a much better generalization over POMO # on CVRPLIB.
Finally, we note that AMDKD equipped with EAS performs the best among all baselines, even achieving the optimal solution on some instances (e.g., KroA100, KroD100, lin105 in TSPLIB and X-n110-k14 in CVRPLIB).
This leads to a new state-of-the-art performance for neural methods on these benchmark datasets.
Finally, we list the full generalization results of our
AMDKD in Table 10 (TSPLIB) and Table 11 (CVRPLIB), respectively.
[Table] [Table]
E Used assets and licenses
Table 12 lists the used assets in our work, which are all open-source for academic research.
For our code and used data (new assets), we are using the MIT License.
[Table]
[Table] [Table]