P ARETO S ET L EARNING FOR N EURAL M ULTI - OBJECTIVE C OMBINATORIAL O PTIMIZATION
A
BSTRACT
Multiobjective combinatorial optimization (MOCO) problems can be found in many real-world applications.
However, exactly solving these problems would be very challenging, particularly when they are NP-hard.
Many handcrafted heuristic methods have been proposed to tackle different MOCO problems over the past decades.
In this work, we generalize the idea of neural combinatorial optimization, and develop a learning-based approach to approximate the whole Pareto set for a given MOCO problem without further search procedure.
We propose a single preference-conditioned model to directly generate approximate Pareto solutions for any trade-off preference, and design an efﬁcient multiobjective reinforcement learning algorithm to train this model.
Our proposed method can be treated as a learning-based extension for the widely-used decomposition-based multiobjective evolutionary algorithm (MOEA/D).
It uses a single model to accommodate all the possible preferences, whereas other methods use a ﬁnite number of solutions to approximate the Pareto set.
Experimental results show that our proposed method signiﬁcantly outperforms some other methods on the multiobjective traveling salesman problem, multiobjective vehicle routing problem, and multiobjective knapsack problem in terms of solution quality, speed, and model efﬁciency.
I NTRODUCTION
Many real-world applications can be modeled as multiobjective combinatorial optimization (MOCO) problems (Ehrgott & Gandibleux, 2000).
Examples include the multiobjective traveling salesman problem (MOTSP) (Lust & Teghem, 2010a), the multiobjective vehicle routing problem (MOVRP) (Jozefowiez et al., 2008) and the multiobjective knapsack problem (MOKP) (Bazgan et al., 2009).
These problems have multiple objectives to optimize, and no single solution can optimize all the objectives at the same time.
Instead, there is a set of Pareto optimal solutions with different trade-offs among the objectives.
It is very challenging to ﬁnd all the exact Pareto optimal solutions for a MOCO problem.
Actually, ﬁnding one single Pareto optimal solution can be NP-hard for many problems (Ehrgott & Gandibleux, 2000), and the number of Pareto solutions could be exponentially large with regard to the problem size (Ehrgott, 2005; Herzel et al., 2021).
The decision-maker’s preference among different objectives is usually unknown in advance, making it very difﬁcult to reduce the problem into a single-objective one.
Over the past several decades, many methods have been developed to ﬁnd an approximate Pareto set for different MOCO problems within a reasonable computational time.
These methods often need carefully handcrafted and specialized heuristics for each problem.
It can be very labor-intensive in practice.
In many real-world applications, practitioners need to solve many different instances for the same particular problem, where the instances can be easily obtained or generated (Bengio et al., 2020).
It is desirable to learn the patterns behind these problem instances explicitly or implicitly to design efﬁcient algorithms (Cappart et al., 2021a).
Machine learning techniques can be naturally used for this purpose.
Some learning-based methods have been recently proposed for solving single-objective combinatorial optimization problems (Bengio et al., 2020; Vesselinova et al., 2020; Mazyavkina et al., 2021; Cappart et al., 2021a).
In this work, we extend the learning-based method to solve
MOCO problems in a principled way as shown in Figure 1.
Our main contributions include:
Neural MOCO Model
A
B
C D
E
A B
C D
E
A B
C D
E
A B
C D
E ...
Cost
C o s t(1,3) (2,3) (5,2) (2,6) (6,3) (1,8) (7,9) (4,2) (4,5)(6,4)
P 1 (10,18)
P 3 (20,15)
P 2 (16,17)
A
B
C D
E (
[Figure]
) Problem (b) Model (c) Solution Generation
Fast Forward Inference
Pref. on Obj1 P r e f .
on
O b j
P 4 (22,28) Preference:
We propose a novel neural multiobjective combinatorial optimization method to approximate the whole Pareto set via a single preference-conditioned model.
It allows decision makers to obtain any preferred trade-off solution without any search effort
• We develop an efﬁcient end-to-end reinforcement learning algorithm to train the single model for all different preferences simultaneously, and a simple yet powerful active adaption method to handle out-of-distribution problem instances.
• We conduct comprehensive experiments on MOTSP, MOVR and MOKP of different settings.
The results show that our proposed method can successfully approximate the Pareto sets for different problems in an efﬁcient way.
It also signiﬁcantly
outperforms other methods in terms of solution quality, speed, and model efﬁciency.
B ACKGROUND AND R ELATED W ORK
Multiobjective Combinatorial Optimization (MOCO).
MOCO has been attracting growing research efforts from different communities over the past several decades (Sawaragi et al., 1985; Wallenius et al., 2008; Herzel et al., 2021).
There are two main approaches to tackle the MOCO problems: the exact methods and the approximation methods (Ehrgott, 2005).
Exact methods could be prohibitively costly when, as it often happens, the MOCO problem is NP-hard and the problem size is very large (Florios & Mavrotas, 2014).
For this reason, many heuristics (Jaszkiewicz, 2002; Zhang & Li, 2007; Ehrgott & Gandibleux, 2008) and approximation methods (Papadimitriou & Yannakakis, 2000; Herzel et al., 2021) have been developed to ﬁnd a manageable number of approximated Pareto solutions with a reasonable computational budget.
However, these methods usually depend on carefully handcrafted designs for each speciﬁc problem (Ehrgott & Gandibleux, 2000), and the required effort is often nontrivial in real-world applications.
Machine Learning for Combinatorial Optimization.
As summarized in Bengio et al.
(2020), there are three main learning-based approaches for combinatorial optimization: learning to conﬁgure algorithms (Kruber et al., 2017; Bonami et al., 2018), learning alongside the algorithms (Lodi & Zarpellon, 2017;
Gasse et al., 2019; Chen & Tian, 2019), and learning to directly predict the solutions (Nowak et al., 2018; Emami & Ranka, 2018; Larsen et al., 2018).
Neural combinatorial optimization (NCO) belongs to the last category where the model directly produces a good solution for a given problem instance.
Vinyals et al.
(2015) proposed a pointer network to sequentially construct a solution for the TSP problem.
Bello et al.
(2017) made a critical improvement to use reinforcement learning to train the model, eliminating the impractical optimal solutions collection for
NP-hard problems.
Some other improvements on model structure and training procedure have been proposed in the past few years (Nazari et al., 2018;
Deudon et al., 2018; Kool et al., 2019; Veliˇckovi´c & Blundell, 2021), especially with graph neural networks (GNNs) (Dai et al., 2017;
Li et al., 2018;
Joshi et al., 2019; Dwivedi et al., 2020; Drori et al., 2020).
Recent efforts have been made on more efﬁcient learning strategies (Kwon et al., 2020; Karalias & Loukas, 2020;
Lisicki et al., 2020; Geisler et al., 2022), learning-based graph search (Cappart et al., 2021b; Kool et al., 2021; Fu et al., 2021;
Xin et al., 2021; Hudson et al., 2022), and iterative improvement methods (Wu et al., 2021; Ma et al., 2021; Li et al., 2021).
Neural MOCO.
Most of the existing learning-based methods are for single-objective combinatorial problems.
Recently, a few attempts have been made to solve MOCO problems (Li et al., 2020; Wu et al., 2020; Zhang et al., 2021a;b).
These methods adopt the MOEA/D framework (Zhang & Li, 2007) to decompose a MOCO problem into a number of single-objective subproblems, and then build a set of models to solve each subproblem separately.
However, since the number of Pareto solutions would be exponentially large (Ehrgott, 2005), the required number of models would be huge for ﬁnding the whole Pareto set.
In this work, we propose a single preference-conditioned model for solving MOCO problems, with which the decision makers can easily obtain any trade-off solutions.
The proposed single neural MOCO solver could be much easier to use in a real-world system (Veliˇckovi´c & Blundell, 2021), than those using a large set of different models.
P ROBLEM F ORMULATION 3. M ULTIOBJECTIVE C OMBINATORIAL O PTIMIZATION
A multiobjective combinatorial optimization (MOCO) problem can be deﬁned as follows:
[Equation]
) where X is a discrete search space, and F ( x ) =
( f 1 ( x ) , . . .
f m ( x )) is an m -objective vector.
Since the individual objectives conﬂict each other, no single solution can optimize all of them at the same time.
Therefore, practitioners are interested in Pareto optimal solutions, deﬁned as follows.
Deﬁnition 1 (Pareto Dominance).
Let x a , x b ∈ X , x a is said to dominate x b ( x a ≺ x b ) if and only if f i ( x a ) ≤
f i ( x b ) , ∀ i ∈ { 1 , ..., m } and f j ( x a ) <
f j ( x b ) , ∃ j ∈ { 1 , ..., m } .
Deﬁnition 2 (Pareto Optimality).
A solution x ∗ ∈
X is a Pareto optimal solution if there does not exist ˆ x ∈ X such that ˆ x ≺ x ∗ .
The set of all Pareto optimal solutions is called the Pareto set, and the image of the Pareto set in the objective space is called the Pareto front.
Each Pareto solution represents an optimal trade-off among the objectives, and it is impossible to further improve one of the objectives without deteriorating any other objectives.
3. D ECOMPOSITION AND P REFERENCE - BASED S CALARIZATION
Decomposition is a mainstream strategy for solving multiobjective optimization problem (Zhang &
Li, 2007).
It decomposes a multiobjective problem into a number of subproblems, each of which can be a single objective or multiobjective optimization problem.
MOEA/D (Zhang & Li, 2007) and its variants (Trivedi et al., 2016) solve these subproblems in a collaborative manner and generate a ﬁnite set of Pareto solutions to approximate the Pareto front.
The most widely used way for constructing a single objective subproblem is the preference-based scalarization (Ehrgott, 2005; Miettinen, 2012).
For an m -objective optimization problem, a preference vector for the objective functions can be deﬁned as λ ∈ R m that satisﬁes λ i ≥ [-
-]0 and (cid:80) mi[-=
-]
{+=
+} λ i = 1 .
Weighted-Sum Aggregation is the simplest approach.
It deﬁnes the aggregation function to minimize in the subproblem associated with λ as
[Equation]
However, this approach can only ﬁnd solutions on the convex hull of the Pareto front (Ehrgott, 2005).
Weighted-Tchebycheff (Weighted-TCH)
Aggregation is an alternative approach to minimize:
[Equation]
3) where z ∗ i < min x ∈X f i ( x ) is an ideal value for f i ( x ) .
Any Pareto optimal solution could be an optimal solution of problem (3) with a speciﬁc (but unknown) preference λ
(Choo & Atkins, 1983).
P r e f e r e n c e0.0 0.2 0.4 0.6 0.8 1.
P r e f e r e n c e0.0 0.2 0.
P r e f e r e n c e0.0 0.
(a) 105 Preferences
L1 10 15
L 10
L 1020(b) 105 Solutions
L1 10 15
L 10
L 1020(c) 1 , 035 Solutions
L1 10 15
L 10
L 1020(d) 10 , 011 Solutions
3. C URRENT D RAWBACKS AND O UR M ETHOD
Drawbacks of Existing Methods.
For many MOCO problems, the size of the Pareto set would be exponentially large with respect to the input size (e.g., nodes in MOTSP).
It is computationally impractical for existing methods to ﬁnd the whole Pareto set
(Herzel et al., 2021).
For this reason, all of the existing heuristic-based and learning-based methods are to ﬁnd a small subset of approximate
Pareto solutions.
Decision makers can only select solutions from this small set, which often does not contain their preferred solutions.
In addition, scalarization may also produce a complicated single objective subproblem.
For example, the Tchebycheff scalarized subproblem of MOTSP is not a classic TSP, and thus cannot be solved by the highly specialized TSP solvers such as LKH (Helsgaun, 2000) or Concorde (Applegate et al., 2007).
Our Method.
Instead of ﬁnding a set of ﬁnite solutions, we propose a novel way to approximate the whole Pareto set using a single model.
With our proposed model, decision makers can easily obtain any solution from the approximate Pareto set to satisfy their preferred trade-offs in real time as shown in Figure 2.
This is a clear advantage to support interactive decision making.
In addition, our proposed reinforcement learning based method can use a scalarization method to combine multiobjective rewards, and does not need to consider the problem-speciﬁc condition.
In this paper, we mainly consider learning the whole Pareto front.
It is possible to incorporate decision-maker’s preferences on speciﬁc regions for model building and inference as discussed in Appendix D.6.
We believe our proposed method is a new principled way to solve multiobjective combinatorial optimization problems.
T HE P ROPOSED M ODEL : P REFERENCE -C ONDITIONED N EURAL MOCO 4. P REFERENCE -C ONDITIONED S OLUTION C ONSTRUCTION
Decomposition and scalarization link preferences to their corresponding Pareto solutions.
This work builds a preference-conditioned model to accommodate all the preferences.
We use the MOTSP as an example to explain our model design.
In an MOTSP instance s , a fully connected graph of n nodes (cities) with m distance metrics on each edge is given.
A feasible solution is a tour that visits each city exactly once and returns to the starting city.
The i -th objective to minimize is the tour length (total cost) based on the i -th distance metric.
A tour can be represented as π = ( π 1 , · · · , π t , · · · , π n ) , π t ∈ { 1 , · · · , n } , a permutation of all the nodes deﬁning the order in which n cities is visited.
Our model deﬁnes a preference-conditioned stochastic policy p θ ( λ ) ( π | s ) parameterized by θ ( λ ) to construct a valid solution in sequence:
[Equation]
The goal is to learn an optimal preference-conditioned policy p θ ( λ ) ( π | s ) to construct tours with the lowest scalarized costs for each preference λ .
[Figure]
A B
C D
E
A
B
C D
E
A
B
C D
E
A B
C D
E
4. T HE P ROPOSED M ODEL
We propose to use an Attention Model (AM) (Kool et al., 2019) as our basic encoder-decoder model as shown in Figure 3.
For the MOCO problems considered in this work, a preference-agnostic encoder is capable to transfer problem instances into embeddings (e.g., embedding for all cities) used in the preference-conditioned decoder.
In our model, only the decoder’s parameters θ decoder ( λ ) are conditioned on the preference λ :
[Equation]
Preference-agnostic Encoder.
The encoder takes a problem instance s
(e.g., an MOTSP instance with n cities) as its input, and outputs a set of d -dimensional node embeddings { h 1 , · · · , h n } for each city.
For a given instance, the same embeddings can be used for different preferences.
Hence we only need a single forward pass for the dense encoder.
We use the attention-based encoder as in Kool et al.
(2019) for all preferences.
Preference-based Attention Decoder.
The decoder has the same structure as in the attention-based model (Kool et al., 2019), but with parameters θ decoder ( λ ) =
[ W Q ( λ ) , W K ( λ ) , W V ( λ ) , W MHA ( λ )] conditioned on the preference λ .
It takes the nodes embeddings for all cities as input, and sequentially selects the next node π t with probability p θ ( λ ) ( π t | s, π 1: t − 1 ) .
At time step t , the decoder ﬁrst constructs a context embedding ˆ h ( C ) =
[ h π 1 , h π t − 1 ] W Q ( λ ) from the ﬁrst selected node h π 1 , and the last selected node h π t − 1 .
The matrix W Q ( λ ) ∈ R 2 d × d projects the concatenated 2 d -dimensional vector to a d -dimensional vector.
Then we further aggregate the context embedding via a Multi-Head Attention (MHA) (Vaswani et al., 2017) with the embeddings for all cities { h 1 , · · · , h n } :
[Equation]
where Q, K, V are the query, key and value for MHA, respectively.
W MHA ( λ ) represents the MHA parameters.
The context embedding h ( C ) contains all information for the instance and the current partial tour at step t .
We can calculate the logit for selecting each city with its embedding h j :
[Equation]
j [Equation] otherwise
[Equation]
All already visited cities are masked with −∞ and will not be selected as the next city.
The logits of the rest cities are clipped into [ − C, C ] ( C = 10) as in the AM model (Kool et al., 2019).
The probability for choosing the j -th city at time step t can be calculated as p θ ( λ ) ( π t =
j | s, π 1: t − 1 ) =
e logit j / (cid:80) k
e logit k .
With this probability, the decoder can construct a feasible tour.
One remaining designing issue is how to generate the preference-conditioned parameters θ decoder ( λ ) .
Multiplicative interactions (Jayakumar et al., 2020) and hypernetwork (Schmidhuber, 1992; Ha et al., 2017) provide a powerful and efﬁcient way for conditional computation, which is widely used for transfer learning (von Oswald et al., 2020; Ehret et al., 2021; Lin et al., 2020; Navon et al., 2021).
We use a simple MLP hypernetwork θ decoder ( λ ) =
MLP ( λ | ψ ) to generate the decoder parameters conditioned on the preference.
The details of our proposed model can be found in Appendix B.
4: λ k ∼ SamplePreference (Λ) ∀ k ∈ { 1 , · · ·
K } 5: s i ∼ SampleInstance ( S ) ∀ i ∈ { 1 , · · ·
B } 6: π jki ∼ SampleTour ( p θ ( λ k ) ( ·| s i ))
∀ k, i ∀ j ∈ { 1 , · · · , N } 7: b ( s i | λ k ) ← 1 N (cid:80)
Nj =
1 L ( π jki | λ k , s i ) ∀ k ∈ { 1 , · · · , K } ∀ i ∈ { 1 , · · · , B } 8: ∇J ( θ ) ← 1 KBN (cid:80)
Kk =1 (cid:80)
Bi
=1 (cid:80) Nj =
[( L ( π jki | λ k , s i ) − b
( s i | λ k )) ∇ θ ( λ k ) log p θ ( λ k )
( π jki | s i )] 9: θ ← ADAM ( θ, ∇J ( θ )) 10: end for 11: Output: The model parameter θ
Instance Augmentation for MOCO.
Our proposed model only has a small extra computational and memory overhead to the original single-objective AM solver.
We keep our model as simple as possible, making it easy for our approach to use other models and other improvements developed for single-objective NCO.
These properties are crucially important for generalizing the NCO to multiobjective problems.
In this work, we simply extend the instance augmentation method (Kwon et al., 2020) to MOCO.
The details can be found in Appendix B.1.
P REFERENCE - CONDITIONED M ULTIOBJECTIVE P OLICY O PTIMIZATION 5. C OST F UNCTION
Our proposed node selection strategy guarantees that the model can always generate feasible solutions.
In this section, we develop an efﬁcient multiobjective policy optimization method to train the model for all the preferences simultaneously.
For an MOTSP problem, the objective functions are a vector of m different costs (i.e. lengths) for a tour L ( π ) =
[ L 1 ( π ) , · · · , L m ( π )] .
We can deﬁne a weighted-Tchebycheff scalarized cost for each preference λ :
[Equation]
where z ∗ i is an ideal cost for the i -th objective.
For a given instance s , our goal is to minimize the expected cost for all preferences:
[Equation]
where Λ is the uniform distribution over all valid preferences.
To train the model, we repeatedly sample different instances s ∼ S at each iteration.
We deﬁne the training loss as J ( θ ) =
E s ∼S J ( θ | s ) .
5. M ULTIOBJECTIVE REINFORCE
For a given instance s and a speciﬁc preference λ , we use the REINFORCE (Williams, 1992) to estimate the gradient for the preference-conditioned scalar cost:
[Equation]
where b ( s | λ ) is the baseline of expected cost to reduce the gradient variance.
This gradient can be estimated by Monte Carlo sampling.
At each update step, we randomly sample K preference { λ 1 , · · · , λ K } ∼ Λ , B instances { s 1 , · · · , s B } ∼ S , and N different tour { π 1 i , · · · , π Ni } ∼ p θ ( λ k ) ( ·| s i ) for each λ k - s i combination.
The approximated gradient is:
[Equation]
[Equation] We use the shared baseline b shared ( s i | λ k ) = 1 N (cid:80) Nj =1 L ( π jki | λ k , s i ) over N sampled tours for each λ k − s i combination.
The starting node for each tour π jki is chosen in random to force diverse rollouts as proposed in (Kwon et al., 2020).
The algorithm is shown in Algorithm 1 .
5. A CTIVE A DAPTION
We also propose a simple yet powerful active adaption approach to further adjust the whole model to approximate the Pareto front for a given test instance in Appendix B.3.
The proposed method does not depend on speciﬁc instance distribution S , and is suitable for out-of-distribution adaption.
E XPERIMENTS
Problems and Model Setting.
We consider MOTSP (Lust & Teghem, 2010a), MOCVRP (Lacomme et al., 2006) and MOKP (Bazgan et al., 2009) in our experimental studies, and use the same model settings for all problems with different task-speciﬁc input sizes and mask methods.
The main policy model encoder is the Attention Model (Kool et al., 2019) and the hypernetwork is an MLP.
We randomly generate 100 , 000 problem instances on the ﬂy for each epoch, and train the model for 200 epochs.
The optimizer is ADAM with learning rate η = 10 − 4 and weight decay 10 − 6 .
We train our models on a single RTX 2080 -Ti GPU, and it costs about 10 minutes for an epoch on
MOTSP100.
We give detailed model settings, problem formulations, and more experimental results in Appendix BCD.
The source code can be found in https://github.com/Xi-L/PMOCO .
Baseline.
We call our proposed preference-conditioned multiobjective combinatorial optimization as P-MOCO .
We compare it with three widely-used evolutionary algorithm frameworks for MOCO:
MOGLS (Jaszkiewicz, 2002) is a multiobjective genetic local search algorithm, NSGAII
(Deb et al., 2002) is a Pareto dominance-based multiobjective genetic algorithm, and MOEA/D (Zhang & Li, 2007) is a decomposition-based multiobjective evolutionary algorithm.
All these algorithm frameworks need problem-speciﬁc heuristics to generate and search feasible solutions for different problems.
We also compare P-MOCO with two other learning-based methods: DRL-MOA (Li et al., 2020) decomposes a MOCO with different preferences and builds a Pointer Network (Vinyals et al., 2015; Bello et al., 2017) to solve each subproblem, and AM-MOCO is a multi-models variant of our proposed model, which builds Attention Model (Kool et al., 2019) for each subproblem.
The
Weight-Sum scalarization of MOTSP and MOKP are their respective single-objective counterpart.
Therefore, we also compare our method with the approach that uses some state-of-the-art singleobjective solvers for each weight-sum subproblem.
[Table]
Model Information for the learning-based methods is shown in Table 1.
Our model supports ﬂexible preference assignment and only has 1 . 1% total parameters to the multi-model counterpart.
Inference and Metrics.
We report the results and run time for solving 200 random test instances for each problem, with normally 101 to 105 different trade-offed solutions, and up to 10 , 011 solutions for our proposed method.
In most cases, we report our model’s zero-shot generalization performance without any search and ﬁne-tune.
We use the hypervolume indicator (Zitzler et al., 2003) to measure the performance for each method.
For a set P ⊂ R m in the objective space, we can ﬁnd a reference point r ∗ that dominated by all solutions in P , and deﬁne the hypervolume HV ( P ) as volume for:
S = {
[Equation]
|
[Equation]
∈ P such that y ≺ r ≺ r ∗ } , (12) where HV ( P ) =
Vol ( S ) .
In general, the larger the hypervolume, the better the solution set tends to be.
The ground truth Pareto set always has the largest hypervolume.
We report the normalized hypervolume values in [0 , 1] with respect to the same r ∗ for all the methods, and also the ratios of hypervolume difference to our method.
A Wilcoxon rank-sum test with a signiﬁcance level 1% is conducted to compare the results for each experiment.
More details can be found in Appendix D.1.
6. R ESULTS AND A NALYSIS
MOTSP.
The results on two and three objective MOTSP are shown in Table 2 and Table 3 respectively.
MOGLS, NSGAII and MOEA/D all use 2 -opt local search heuristic (Jaszkiewicz, 2002)
to
[Table]
search for promising solutions.
We also include two weight-sum scalarization baselines with the state-of-the-art LKH solver (Helsgaun, 2000; Tin ´ os et al., 2018) and Google OR tools (Perron &
Furnon, 2019).
For the bi-objective problems [Table] our proposed method with a single model has similar performances compared with AM-MOCO on all problems.
It achieves the best performance with instance augmentation, which signiﬁcantly outperforms other methods but is beaten by the LKH solver.
For the three objective problems, our method can further improve its performance by generating much more trade-off solutions within a reasonable amount of time, which other methods cannot do.
As shown in Figure 2 and Figure 5, our method can successfully learn the mapping from preferences to the corresponding solutions, and can generate a good prediction to the whole Pareto front.
Decision makers can easily obtain any preferred trade-off solutions as they like.
This ﬂexibility could be desirable in many real-world applications.
More discussion on the connection between the preference and Pareto solution for three-objective TSP can be found in Appendix D.5 D.6 D.7.
MOCVRP.
In this problem, each node has a demand, and we need to construct multiple return routes for a vehicle with a ﬁxed capacity from the same depot to handle all demands.
The objectives we consider are to minimize the tour length for all routes and also the tour length for the longest routes (the makespan in scheduling)
(Lacomme et al., 2006).
All the non-learning algorithm frameworks use the problem-speciﬁc constructive heuristics and local search method proposed in
Lacomme et al.
(2006) to search feasible non-dominated solutions.
The results in Table 2 show that our method signiﬁcantly outperforms the non-learning heuristics in terms of both solution quality
[Table]
P-MOCO (1,035 pref.)
0.33 2.08% (19s) 0.36 4.03% (48s) 0.46 2.12% (2.9m)
P-MOCO (10,011 pref.)
0.34 0.00% (3m) 0.38 0.00% (9m) 0.47 0.00% (33m) and running time.
It also outperforms AM-MOCO with 100 individual models, which could be due to the asymmetric objective scales.
We provide further analysis in Appendix D.5.
MOKP.
The multiobjective 0 - 1 knapsack problem can be found in many real-world applications (Bazgan et al., 2009).
We consider the uni-dimension problem, where each item has multiple values and one weight.
The goal is to select a subset of items to maximize all obtained values with a weight constraint.
The non-learning methods use binary coding with a greedy transformation heuristic to maintain feasibility (Ishibuchi et al., 2014).
We also include weight-sum scalarization baselines with dynamic programming (DP) and a strong greedy search based on the value-weight ratio.
According to the results in Table 2, our method has the best performance on all problems.
The
DP method is also outperformed by our method since the weight-sum scalarization can only ﬁnd the convex hull of the Pareto front.
The Tchebycheff scalarization of MOKP is not a KP problem, while our method is more ﬂexible to use Tchebycheff scalarization on the reward function.
We also report the results on 10 objective MOKP100 and the generalization performance to problem with 500 items in Appendix D.8.
Out-of-Distribution Problems and Active Adaption.
We also validate the generalization performance of our method on 6 out-of-distribution (OOD) MOTSP problems from Fonseca et al.
Their ground truth Pareto fronts can be obtained by exhaustive search.
The results are shown in Appendix D.2 due to the page limit.
With active adaption, our method can achieve good performance ( 1% - 1 .
5% HV gap to the ground truth Pareto fronts) on these OOD problems.
C ONCLUSION AND F UTURE W ORK
Conclusion.
We have proposed a novel preference-conditioned method to approximate the whole
Pareto front for MOCO problems using a single model.
It allows decision makers to directly obtain any trade-off solutions without any search procedure.
Experiments on different problems have shown that our proposed method signiﬁcantly outperforms other methods in terms of performance, speed and model efﬁciency.
We believe the proposed method is a principled way for solving MOCO.
Future Work.
In a sense, our method can be regarded as a learning version of the decompositionbased algorithm (MOEA/D (Zhang & Li, 2007)) dealing with all the possible trade-off preferences.
Instead of maintaining a set of ﬁnite solutions as in other MOEA/D vaiants (Trivedi et al., 2016), we build a single learning-based model to solve the subproblems for all the preferences simultaneously in a collaborative manner.
We believe the single-model-for-all-preference approach is a promising alternative to the current default ﬁnite-population-based methods, and it could be an important research direction for multiobjective optimization.
Our method can be further improved with other advanced models and efﬁcient multiobjective training procedures.
In the future, we will study fundamental issues of multiobjective optimization (e.g., convergence v.s. diversity, exploitation v.s. exploration trade-off) for Pareto set learning methods.
Limitation.
It is very difﬁcult to give a convergence guarantee for learning-based MOCO, where each preference-based subproblem could be already NP-hard, and the number of Pareto solutions is exponentially large with respect to the input size.
See detailed discussion in Appendix A.
A CKNOWLEDGMENTS
We thank Prof. Hisao Ishibuchi for his valuable comments on an earlier version of this work.
This work was supported by the Hong Kong General Research Fund (11208121, CityU-9043148)
A P ARETO S ET L EARNING AND A PPROXIMATION A NALYSIS A. P ARETO S ET L EARNING AND C ONVERGENCE G UARANTEE
In this work, we have proposed a novel neural combinatorial optimization (NCO) method to approximate the whole Pareto set for MOCO problems with a single model.
The proposed learning-based
MOCO solver can directly generate arbitrary trade-off solutions without extra optimization.
We believe it is a principled way to solve MOCO problems.
However, the lack of an exact optimality guarantee is a limitation of the proposed method, which is also the case for previous work on single-objective neural combinatorial optimization (Vinyals et al., 2015; Bello et al., 2017; Kool et al., 2019).
This limitation is mainly due to the fact that many singleobjective combinatorial optimization (CO) problems are NP-hard, and the size of Pareto sets for a
MOCO problem would be exponentially huge, which makes it very difﬁcult to exactly solving the problems (Ehrgott, 2005;
Herzel et al., 2021).
In addition, the training for the parameterized policy (neural network model) cannot guarantee to ﬁt all training problems perfectly.
The generalization ability to problem instances with different patterns (out-of-distribution generalization) is another critical issue that makes it difﬁcult to give an exact optimality guarantee to the proposed learningbased algorithm.
On the other hand, our proposed model is an efﬁcient mapping from the preferences to the corresponding approximate set of the Pareto optimal solutions.
It provides a ﬂexible way for decision makers to obtain an approximate solution with their preferred trade-off directly.
The experimental results also show that our proposed method can generate good approximate Pareto sets for three different MOCO problems.
In the next subsection, we provide a thorough discussion on the approximation ability of our proposed method.
A. A PPROXIMATION A NALYSIS
For a MOCO problem, the number of Pareto solutions could be exponentially large with respect to its input size, which makes the problem intractable (Ehrgott, 2005;
Herzel et al., 2021).
The preference-based scalarization methods and decomposition methods (Choo & Atkins, 1983; Zhang & Li, 2007)
we used provides a principled way to link the Pareto solutions with preference, allowing us to tackle the problem in a systematic manner.
In this work, we propose to approximately solve the scalarized subproblem with all preferences via a single model.
We ﬁrst brieﬂy review the weighted scalarization method and its Pareto optimality guarantee as discussed in the main paper.
Then we provide further discussion on the approximation analysis.
Our proposed method decomposes a MOCO problem into preference-based subproblems with the weighted-Tchebycheff scalarization (Weighted-TCH):
[Equation]
(13) where z ∗ i is the ideal value for objective
f i ( x ) (e.g., the lower bound), and u ∗ i =
z ∗ i − ε is a utopia value with small positive component ε .
The preference vector λ ∈ R m satisﬁes λ
i ≥and (cid:80) mi =
λ i = 1 , where λ i is the preference for the i -th objective.
This approach has a desirable property:
Lemma 1 (Choo & Atkins (1983)).
A feasible solution x ∈ X is Pareto optimal if and only if there is a weight vector λ > 0 such that x is an optimal solution to the problem (13).
According to Lemma 1 , we can obtain any Pareto solution by solving the Weighted-TCH subproblem with a speciﬁc weight.
However, the weight for each Pareto solution depends on its objective values, which are not known in advance (Sawaragi et al., 1985; Ehrgott, 2005).
The decision-maker still needs to solve multiple subproblems with different preferences to ﬁnd a desirable solution.
To ﬁnd the whole Pareto set, it needs to solve an exponentially huge number of subproblems.
Given a problem instance s , our proposed model provides a single mapping function x λ = h ( λ ) from any preference λ to its corresponding solution x λ , which is constructed by the preferencebased policy p θ ( λ ) ( x | s ) .
In the ideal case, if all generated solutions x λ are the optimal solutions x ∗ λ of problem (13) with preference λ , according to Lemma 1 , our proposed model can generate the whole Pareto set (all Pareto optimal solutions) for the original MOCO problem.
In practice, we are interested in the proposed method’s approximation ability.
We ﬁnd that its performance strongly depends on the approximation ability of the parameterized policy (neural network model) on the single-objective scalarized subproblem.
We ﬁrst give an informal claim on our method’s approximation ability, then provide detailed explanations and discussions.
(Informal) Claim 1.
If the proposed method can approximately solve the subproblem (13) with any preference λ , it can generate a good approximation to the whole Pareto set for the MOCO problem.
To support this claim, we follow the traditional ε -Pareto approximate method for MOCO problems (Papadimitriou & Yannakakis, 2000; Herzel et al., 2021).
First, an ε -Pareto domination relation between two individual solutions can be deﬁned as:
Deﬁnition 3 ( ε -Pareto Domination).
For a MOCO problem and an ε > 0 , let x a , x b ∈ X , x a is said to ε -dominate x b ( x a ≺ ε x b ) if f i ( x a ) ≤ (1 + ε ) f i ( x b ) , ∀ i ∈ { 1 , · · · , m } .
This deﬁnition is a natural generalization from the (1 + ε ) -approximation for single-objective optimization.
With this concept, an ε -approximate Pareto set (Papadimitriou & Yannakakis, 2000) can be deﬁned as:
Deﬁnition 4 ( ε -Approximate Pareto Set).
For an ε > 0 , a set P ε ⊂ X is an ε -approximate Pareto set, if for any feasible solution x ∈ X , there exists a solution x (cid:48) ∈ P ε such that x (cid:48) ≺ ε x .
In other words, all feasible solutions of the MOCO problem can be almost dominated by some solutions in P ε (Papadimitriou & Yannakakis, 2000).
When the Pareto set is intractable and hard to ﬁnd, the ε -approximate Pareto set would be a reasonable choice to achieve in practice.
Each MOCO problem has a unique Pareto set, but can have different ε -approximate Pareto sets.
The ability of our proposed method to ﬁnd an ε -approximate Pareto set strongly depends on its performance on each single-objective preference-based subproblem.
Theorem 1.
Let x ∗ λ denotes the optimal solution of the problem (13) with preference
λ , if the proposed method can generate an approximate solution x λ ≺ ε x ∗ λ for any preference λ , it is able to generate an ε -approximate Pareto set P ε to the MOCO problem.
Proof.
Let P be the Pareto set for a MOCO problem, for any x Pareto ∈ P , according to Lemma 1 , there is a weight vector λ > 0 such that x = x ∗ λ is the optimal solution for subproblem (13) with a speciﬁc preference λ .
Therefore, our proposed method can generate an approximated solution x λ ≺ ε x ∗ λ = x Pareto .
By generating approximate solutions for all x Pareto ∈ P , our proposed method is able to generate an ε -approximate Pareto set P ε to the MOCO problem.
A. L IMITATION
Strong Assumption on (Approximately) Solving all Subproblems: The approximation guarantee in Theorem 1 heavily depends on the ability to (approximately) solve each weighted subproblem.
Due to the NP-harness, it is indeed non-trivial to give a convergence guarantee to generate ε -dominate solutions for any preference with a small enough ε .
This limitation also applies for other end-to-end learning-based (e.g, neural combinatorial optimization) and heuristic-based methods.
We are aware that some efforts have been made to combine the learning-based method with dynamic programming to achieve asymptotically optimal solution solution for speciﬁc single-objective problem in recent works (Cappart et al., 2021b; Kool et al., 2021).
These methods provide a controllable trade-off between the solution quality and the computational cost for solving NP-hard problems.
However, their generalization to the multi-objective problem is not straightforward, since the scalarized subproblem for each preference is not necessary the same as its single-objective counterpart.
For example, a Tchebycheff scalarized MOTSP is not a single-objective TSP as discussed at the end of Section 3.2.
In addition, according to Bengio et al.
(2020), these methods belong to the class of learning alongside the algorithms, while our proposed approach is learning to directly produce the solutions (neural combinatorial optimization).
Therefore, the idea for learning enhanced multiobjective combinatorial algorithm could be an important research topic in future, but out of the scope for the current work.
Dense Approximation for the Whole Pareto Set: Another concern would be the required number of solutions in the ε -approximate Pareto set P ε .
If the required number is exponential to the input size, the approximation itself is also intractable.
In their seminal work, Papadimitriou & Yannakakis (2000) establish a promising result:
Theorem 2 (Papadimitriou & Yannakakis (2000)).
For any multiobjective optimization problem and any ε , there is an ε -approximate Pareto set P ε of which the size is polynomial in the number of solutions and 1 ε (but exponential in the number of objectives).
However, the existence of such a set still does not mean that it can be easily found (Papadimitriou & Yannakakis, 2000; Herzel et al., 2021).
The computability (whether P ε can be constructed in polynomial time) would be hard to justify for a real-world problem.
For a new unseen problem instance in practice, our proposed method might still need to generate an exponentially large number of solutions to construct an ε -approximate Pareto set P ε .
It is also unclear how to properly select a set of preferences in advance.
Many research efforts have been made on developing approximation methods for solving MOCO problems in the past decades (Herzel et al., 2021;
Hansen, 1980; Papadimitriou & Yannakakis, 2000; Vassilvitskii & Yannakakis, 2005; Koltun & Papadimitriou, 2005;
Bazgan et al., 2017).
In future work, it is important to better leverage the current advanced approximation strategies to design more efﬁcient preference-based methods.
In the learning-based optimization scenario we consider, it is also possible to learn the suitable approximation method and/or preference distribution directly from the data (problem instances).
B D ETAILS ON THE P ROPOSED M ODEL B
M ODEL S ETTING
[Table]
We use the same model for all MOCO problems while tuning the input size and mask method for each problem.
Table 4 shows the number of parameters of a standard single-objective attention model (Kool et al., 2019) and our proposed preference-based multiobjective attention model.
Our model supports ﬂexible preference assignment at the inference time with a small overhead, while the other neural MOCO methods all require training multiple AM models for different preferences.
We build the single-preference attention models as well as our model following the implementation in Kwon et al.
Attention Encoder.
The encoder we use is the standard attention encoder as in Kool et al.
and it is shared by all preferences.
The encoder has 6 attention layers, and 128 -dimensional node embedding for the input nodes.
Each attention layer has a multi-head attention (MHA) with eight 16 -dimensional heads, and a fully connected layer (FC) with one 512 -dimension hidden sublayer.
The encoder also includes skip-connection and batch normalization for each attention layer.
We use the same model for all MOCO problems (MOTSP, MOCVRP, MOKP) but with different input dimensions for each problem, which will be introduced in the next section.
Preference-Conditioned Decoder.
The decoder’s main model structure is the same as the AM decoder (Kool et al., 2019).
It has one multi-head attention layer with eight 16 -dimensional heads similar to the encoder, but without skip-connection and batch normalization.
The decoder uses a single 128 -dimensional attention head to calculate the probabilities of selecting different nodes at each step.
Different problems have different masking methods for probability calculation.
We use a simple MLP model to generate the preference-conditioned parameters for the decoder.
For all MOCO problems, the MLP model has two 128 -dimensional hidden layers with ReLu activation.
The input is an m -dimensional preference vector λ which satisﬁes λ i ≥ 0 and (cid:80) mi =
λ i = 1 , where m is the number of objectives and λ
i is the preference for the i -th objective.
We adopt the parameter compression approach in Ha et al.
(2017) to control the model size.
The MLP model ﬁrst generates a hidden embedding e ( λ ) =
MLP ( λ | ψ ) , then maps the hidden embedding to the decoder parameters via linear projection θ decoder =
We ( λ ) + b .
The learnable parameters are ψ for the MLP model
MLP ( λ | ψ ) and the parameter matrices W and b for the decoder.
Training Procedure.
For all problems, we train our proposed model for 200 epochs, with 100 ,problem instances randomly generated on the ﬂy at each epoch.
At each iteration step, we need to sample K preferences, B problem instances, and N tours to calculate the policy gradient.
We set
K × B = 64 to make the batch of 64 instances for training a single AM model, and let N equal to the problem size (e.g., the number of nodes) as in Kwon et al.
We ﬁnd the model performance is equally good for setting K = 1 , 2 and 4 , and keep using K = 1 for all problems.
In other words, we randomly generate a preference λ that satisﬁes λ i ≥ 0 and (cid:80) mi =1 λ i = 1 at each training step.
For the AM-MOCO baseline, we adapt the transfer training approach in Li et al.
(2020) to train multiple AM models for different preferences.
We ﬁrst train a single AM model with a single preference on one objective from scratch with 200 epochs, then transfer its parameter to the model for neighbor subproblem with similar preference, and ﬁne-tune the new model with 5 epochs.
With sequentially transfer and ﬁne-tune, we can obtain a set of trained models for different preferences.
In most experiments, we set the number of preferences as 101 .
Therefore, we need to build 101 AM models with total 700 training epochs.
Instance Augmentation for MOCO.
Due to the design choice of minimal essential change (e.g., the preference-conditioned decoder), our method can also enjoy the current improvements that were originally proposed for the single objective NCO.
Here, we generalize the instance augmentation method proposed in Kwon et al.
(2020) to the MOCO version.
The key idea of instance augmentation for NCO is to ﬁnd multiple efﬁcient transformations for the original problem such that they share the same optimal solution.
Then, we can use an NCO method to solve all problems and select the best solution among all obtained (potentially different) solutions.
In this way, we have a more robust result similar to the test-time augmentation for computer vision (Szegedy et al., 2016).
For the single-objective euclidean TSP and CVRP, there is a set of straightforward transformations, which simply ﬂips or rotates the coordinate for all the 2D locations in a problem instance (Kwon et al., 2020).
For a location ( x, y ) , there is eight different transformation, namely, { ( x, y ) , ( y, x ) , ( x, 1 − y ) , ( y, 1 − x ) , (1 − x, y ) , (1 − y, x ) , (1 − x, 1 − y ) , (1 − y, 1 − x ) } .
For an m -objective euclidean MOTSP problem, the concrete location representations are independent for each objective.
Therefore, we can independently apply different transformations for each objective.
Consider the above eight different transformations for each objective, we can have 8 m different problem transformations for an MOTSP instance.
We have ﬁxed 8 transformations for
MOCVRP since it only has one 2D coordinate, and no transformation for MOKP.
The details for each problem can be found in the next section.
B. T RAINING E FFICIENCY
We use the same amount of samples to train our proposed preference-based model as the other single-objective solvers need (Kool et al., 2019; Kwon et al., 2020).
Indeed, our proposed model requires signiﬁcantly fewer samples and training epochs, compared to the other MOCO methods that need to build multiple models for different preferences.
Concorde
LKH
OR Tools
[Table]
We compare our model’s performance on one of the objective (e.g., with preference (1 , 0) ) with the other SOTA single-objective solver and learning-based solver, the results are shown in Table 5.
The results of Concorde/LKH/OR Tools are from Kwon et al.
and we run the learning-based solver by ourselves.
We report the average performance over 10 , 000 test instances.
AM is the single-objective solver (one model in AM-MOCO), P-MOCO (single preference) is our proposed model but only training on a single ﬁxed preference (1 , 0) , and P-MOCO (all preferences) is our proposed model with the reported result on the preference (1 , 0) .
With the same amount of training samples, our model has similar single-objective performance with learning-based single-objective solver, while it can additionally approximate the whole Pareto front.
The learning-based solver’s performance can be further improved by sampling or active search.
These results indicate that we can use a single encoder to efﬁciently learn a shared representation for all trade-offs among different objectives, and there is a positive knowledge transfer among preferences during the learning procedure.
In addition, it also conﬁrms the assumption that similar preferences should have similar corresponding (approximate) Pareto solutions for the multiobjective problems we consider in this paper.
These ﬁndings could be useful to design more powerful learning-based models for MOCO in the future.
B. A CTIVE A DAPTION
After end-to-end training, our proposed method can directly generate different trade-off solutions to a given problem without further search procedure.
However, similar to single-objective neural combinatorial optimization, this approach could still have a gap to the Pareto front, especially for problems out of the training distribution S (e.g., with different sizes and patterns)
(Lisicki et al., 2020).
Iterative search methods, such as sampling and beam search, can further improve the performance for a single solution or single preference (Veliˇckovi´c & Blundell, 2021).
However, these approaches can not ﬁnd a better approximation to the whole Pareto set for a MOCO problem.
1: Input: model parameter θ , instance s , preference distribution Λ , number of adaption steps T , number of preferences per iteration K , number of tours N 2: for t = 1 to T do 3: λ k ∼ SamplePreference (Λ) ∀ k ∈ { 1 , · · · , K } 4: π jk ∼ SampleTour ( p θ ( λ k ) ( ·| s ))
∀ k ∈ { 1 , · · · , K } ∀ j ∈ { 1 , · · · , N } 5: b ( s | λ k ) ← 1 N (cid:80) Nj =
1 L ( π jk | λ k , s ) ∀ k ∈ { 1 , · · · , K } 6: ∇J ( θ ) ← 1 KN (cid:80)
Kk =1 (cid:80) Nj =
[( L ( π jk | λ k , s ) − b ( s | λ k )) ∇ θ ( λ k ) log p θ ( λ k )
( π jk | s )] 7: θ ← ADAM ( θ, ∇J ( θ )) 8: end for 9: Output: The model parameter θ
We propose a simple yet powerful active adaption approach as shown in Algorithm 2 .
It iteratively adapts the model parameter θ ( λ ) to a given instance s (or a batch of instances) with all preferences from the distribution Λ rather than searching for a speciﬁc solution.
This method is similar to the active search in Bello et al.
(2017) which actively reﬁnes the single-objective model for efﬁcient candidate solutions searching.
Our approach focuses on adapting the whole model for a better Pareto front approximation.
Since this method is distribution-agnostic (not depend on speciﬁc instance distribution S ), it is suitable for out-of-distribution adaption.
C D ETAILS OF THE MOCO P ROBLEMS
This section introduces the detailed problem formulation for the MOTSP, MOCVRP and MOKP we used in this work.
We also provide the model conﬁguration (e.g., input size, masks) for each problem.
C. MOTSP
We consider the Euclidean multiobjective traveling salesman problem (Euclidean MOTSP), which is widely used in the MOCO community (Lust & Teghem, 2010b; Florios & Mavrotas, 2014).
Its single objective counterpart, 2D Euclidean TSP, has also been studied in single-objective neural combinatorial optimization (NCO) (Vinyals et al., 2015; Bello et al., 2017; Kool et al., 2019).
A general m -objective MOTSP instance s with n nodes has m n × n cost matrices { C i = ( c ijk ) , i = 1 , · · · , m } for m different costs.
The problem is to ﬁnd a tour (cyclic permutation π ) to minimize all the costs:
[Equation]
where L i ( π | s )
=
c
i π ( n ) π (1) + n − 1 (cid:88)
j = c i π ( j ) π ( j +1) .
In a Euclidean MOTSP, the cost information is stored in the nodes rather than the edges.
The j -th node has a 2 m -dimensional vector
[ x 1 j , x 2 j , · · · , x mj ] where x ij ∈ R 2 is a 2D coordinate for the i -th objective.
The i -th cost c ijk = || x
ij − x ik || 2 is the Euclidean distance for moving from node j to k .
If we only have one objective m = 1 , it reduces to the single-objective 2D Euclidean TSP:
[Equation]
The single-objective TSP is already NP-hard, so does the MOTSP.
In addition, the Pareto set of
MOTSP has an exponential cardinality with respect to its input size (e.g., number of nodes), so it is intractable even for the 2-objective case (Ehrgott & Gandibleux, 2003).
Problem Instance.
Similar to the previous work on single-objective NCO (Lust & Teghem, 2010b;
Florios & Mavrotas, 2014), we randomly sample all n nodes with uniform distribution on the 2 mdimensional unit hyper-square (e.g., [0 , 1] 2 m ) for all problem instances.
Model Details.
In m -objective MOTSP, each node has a 2 m -dimensional vector to store all cost information, so the input size is 2 m for the encoder.
To calculate the probability for selecting the next node, the decoder needs to mask all already visited nodes as unavailable.
We have a valid tour when all node is selected (we assume the end node will connect to the start node).
C. MOCVRP
The vehicle routing problem (VRP) is a classical generalization of TSP, which has been studied for several decades.
This work studies the capacitated vehicle routing problem (CVRP).
In this problem, in addition to the location, each node (city) has a demand δ i needed to be satisﬁed.
There is an extra depot node and a vehicle with a ﬁxed capacity D > δ i , ∀ i to handle all the demands.
The vehicle will always start from the depot node, then goes to different cities to satisfy multiple demands (cid:80)
δ i ≤ D , and turns back to the depot node.
A solution to this problem is a set of routes that satisﬁes the demands for all cities.
In the multiobjective problem, we consider two objectives to optimize.
The ﬁrst one is the total tour length as in the single-objective CVRP, and the other one is the tour length for the longest route (which is also called makespan in scheduling theory).
This problem has been studied in the MOCO community (Lacomme et al., 2006).
Problem Instance.
Similar to the TSP problem, the location of n nodes are uniformly sampled from the unit square.
For the demand, similar to the previous work on the single-objective counterpart (Kool et al., 2019; Kwon et al., 2020), we randomly sample discrete δ i from the set { 1 , · · · , 9 } .
For problem with size n = 20 , 50 , 100 , we set the capacity as D 20 = 30 , D 50 =
and D 100 = 50 , respectively.
Without loss of generality, we normalize the demands ˆ δ i
= δ i D and capacity
ˆ D = DD
= 1 as in the previous work (Kool et al., 2019; Kwon et al., 2020).
Split delivery is not allowed in this problem.
Model Details.
In the MOCVRP, the depot node has a 2 -dimensional location vector, and the other nodes all have 3 -dimensional vectors to store their locations and demands.
We use different parameter matrices to project the nodes into the input embedding with the same dimension d h = 128 .
For node selection, the model records the current capacity of the vehicle and the rest demands for all nodes.
If a node has been already visited or has demand larger than the vehicle’s current capacity, it will be masked as unavailable for the vehicle to visit.
If no node is available to visit, the vehicle will go back to the depot.
Once all nodes have 0 demands, the node selection is ﬁnished and we have a valid solution to the problem.
C. MOKP
Knapsack problem (KP) is also a widely studied combinatorial optimization problem.
In this work, we consider the 0 - 1 multiobjective knapsack problem (MOKP) with m objectives and n items:
[Equation]
where f i ( x ) =
(cid:80) nj =1 v ij x j , subject to (cid:80) nj =1 w j x j ≤ W, x j ∈ { 0 , 1 }
[Equation]
(16) where each item has a weight w j and m different values { v ij , i = 1 , · · · , m } .
The problem (e.g., knapsack) has a maximum weight capacity W , and the goal is to select a set of items within the weight capacity to maximize the sum values for each objective.
To make this problem nontrivial, we further assume all values v ij , ∀ i, j , weights w j ∀ j and the total capacity are non-negative real value.
The total weight of all items is larger than the capacity (cid:80) w i > W , while each single weight is smaller than the capacity w i < W, ∀ i
= 1 , · · · , n .
The single-objective knapsack problem is
NP-hard, so does the MOKP problem (Ehrgott & Gandibleux, 2003).
Problem Instance.
We randomly generate the values and weight for each item both uniformly in [0 , 1] .
We consider problems with n = 50 , 100 , 200 nodes, and the weight capacities are W 50 = 12 .
5 , W 100 =
W 200 = 25 as in the previous work (Bello et al., 2017; Kwon et al., 2020).
Model Details.
In an m -objective MOKP, each item has m values and 1 weight, so the input dimension is 3 for the encoder.
For node selection at each step, we mask all already selected nodes and nodes with weights larger than the remained capacity as unavailable.
We terminate the selection when all nodes are labeled as unavailable.
D
A DDITIONAL
E XPERIMENTAL R ESULTS
D.
H YPERVOLUME
I NDICATOR
[Figure]
To solve a MOCO problem, the result for each method is a set of approximate Pareto solutions.
Since the ground truth Pareto set is usually unknown, we use the hypervolume (HV) indicator (Zitzler et al., 2007) to numerically compare the performance for each method.
The hypervolume indicator is widely used in the MOCO community for algorithm comparison.
The hypervolume of a set is the volume in the objective space it dominates.
For a set P ⊂ R m in the objective space, we can ﬁnd a reference point r ∗ that dominated by all solutions in P , and deﬁne the hypervolume HV ( P ) as the volume of the set:
[Equation]
where HV ( P ) =
Vol ( S ) .
An illustration example is shown in Figure 4.
The grey area is the set
S dominated by the solutions in set P = { p 1 , p 2 , p 3 , p 4 } with the reference point r ∗ .
In this 2dimensional case, the hypervolume HV ( P ) is the size of the grey area.
The hypervolume indicator has two important advantages for measuring the approximate set quality with respect to Pareto optimality (Zitzler et al., 2007).
First, if an approximate set A dominates another approximate set B , it will have a strictly better hypervolume HV ( A ) > HV ( B ) .
In addition, if an approximate set C contains all Pareto optimal solutions, it is guaranteed to have the maximum hypervolume value.
In comparison, an approximate set has better performance if it has a larger hypervolume.
With different objective scales, the hypervolume value will vary signiﬁcantly among different problems.
We report the normalized hypervolume values ˆ H ( P ) =
HV ( P ) / (cid:81) mi r ∗ i for all methods and also their performance gaps to our method.
For each experiment, all methods share the same reference point r ∗ , which contains the largest value achieved for each objective.
Since all problems we consider have positive objective values, we have 0 ≤ ˆ H ( P ) ≤ 1 for all solution sets.
The ground truth Pareto set P ∗ usually has ˆ H ( P ∗ ) < 1 , unless the zero vector 0 ∈ R m is feasible and in the
Pareto set.
D. O UT - OF -D ISTRIBUTION P ROBLEM WITH E XACT P ARETO F RONT
[Table]
We conduct experiments on 6 two-objective MOTSP100 instance (L1-L6) in Florios & Mavrotas (2014) of which the exact Pareto fronts are available.
In these problems, the objective functions have different ranges
[Table]
and the cities are not uniformly located, so they are out of our method’s training distribution.
The results can be found in the Table
[Table]
In addition to hypervolume
[Table]
we also report the Inverted Generational Distance (IGD)
(Fonseca et al., 2006) to measure the average Euclidean distance between the set of approximated Pareto solutions to the exact Pareto front.
A smaller IGD value means the approximated set is closer to the exact Pareto front.
According to the results, our method, with the instance augmentation and/or active search (10 min budget), can have a good performance on these out-of-distribution (OOD) instances with a 1% − 1 .
5% hypervolume gap.
The proposed method also signiﬁcantly outperforms the weight-sum
OR tools baseline.
There is still a gap to the strong weight-sum LKH baseline.
As discussed in the paper, robust OOD generalization is an important research direction for the learning-based solver.
P r e f e r e n c e0.0 0.2 0.4 0.
P r e f e r e n c e0.0 0. P r e f e r e n c e 0.0 0.
[Table] [Table]
[Figure]
P r e f e r e n c e0.0 0.2 0.4 0.
P r e f e r e n c e0.0 0. P r e f e r e n c e 0.0 0.
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure] [Figure]
[Figure] [Figure]
L 4 5 68 9
L 4 57
[Figure] [Figure]
[Figure] [Figure]
D. F LEXIBLE P REFERENCE - BASED A PPROXIMATION
With our model, it is ﬂexible to generate different number of solutions to approximate the Pareto front.
We present an example on the three-objective TSP in Figure 5.
We use the structured weight assignment approach from Das & Dennis (1998) to give the sets of weights for different instances.
This method can generate n = C m + p − 1 p evenly distributed weights with an identical distance to their nearest neighbor on the unit simplex (e.g., (cid:80) mi
=
λ i = 1 with λ i ≥ 0 , ∀ i ), where m is the number of objectives and p is a parameter to control the number of weights.
For the three objective TSP problems ( m = 3 ), we assign p = 13 , 44 and 140 to generate n = 105 , 1035 and 10011 weights respectively.
We also show the corresponding generated solutions for MOTSP instances with 20 , 50 and 100 cities.
According to the results in Figure 5, our model can generate well-distributed solutions with a small number of preferences, and generate a dense approximation with more preferences.
The ability to generate a dense approximation to the whole Pareto set also allows the decision-maker to generate arbitrary preferred solutions on the approximate front.
D. P REFERENCE -S OLUTION C ONNECTION
[Figure]
o n
[Figure] [Figure]
r
[Figure]
r
[Figure]
c
[Figure]
n
[Figure]
b j
[Figure]
c t i v
[Figure]
We further analyze the connection between the preference and its corresponding solution on the uniform and non-uniform Pareto front.
Figure 6 shows the connections in our model with different numbers of preferences for the MOTSP100 instance.
Since the two objectives (costs) in MOTSP have the same scale, this problem has a uniform connection between the preferences and the (approximate) Pareto front.
By increasing the number of preferences, we have three sparse to dense generated Pareto front approximations.
We are more interested in MOCVRP, which has a non-uniform Pareto front.
In this problem, we consider two different objectives to optimize, namely, the total tour length (objective 1) and the tour length for the longest route (objective 2).
These two objectives are in quite different scales, where the ﬁrst objective is signiﬁcantly larger than the second one.
In Figure 7, we show different connections for the MOCVRP100 instance.
For MA-MOCO, we report the connections for all 101 models.
For our proposed model, we report the connections with different numbers of uniform preferences.
In this problem, 101 models or our model with 101 uniform preferences are not enough to generate a dense approximate Pareto front.
The obtained solutions are biased to the area that objective 1 has a much better relative performance.
By increasing the number of preferences, our proposed method can generate more solutions that have relatively better performance for objective 2, which leads to a better Pareto front approximation with higher hypervolume.
In this work, we always use a straightforward uniform sampling method to select the preferences.
It is interesting to design a learning-based approach to select the preferences for a given problem instance.
Preference adjustment and model adaption with awareness on the shape of Pareto front are also worthy to investigate.
We left them to the future work.
In the MOCVRP instance, we also ﬁnd the 101 -model MA-MOCO has a worse performance compared to our method with 101 preferences.
The reason would be the mismatch between the uniform transfer training and the non-uniform Pareto front.
Increasing the training steps for ﬁne-tuning each model might ﬁx this issue, but will lead to an even larger computational overhead, given the current training already require 700 epochs.
The ﬁxed preferences assignment is another issue for MAMOCO.
It requires a ﬁxed set of preferences for each model at the start of the training procedure when the decision makers might have no knowledge on the problem.
When the training procedure is done, it dose not allow any preference adjustment without retraining the models.
D. C ONNECTION BETWEEN P REFERENCES AND S OLUTIONS
In the previous sections, we use the weighted Tchebycheff aggregation to connect the preference to its corresponding solution for two-objective optimization problems: [Equation] (18) where z ∗ i < min x ∈X f i ( x ) is an ideal value for f i ( x ) .
There are also many other aggregation function we can use to build the connection.
For example, a modiﬁed version of weighted Tchebycheff aggregation can be deﬁend as:
[Equation]
where the only difference is the weight vector 1 λ i .
The penalty-based boundary intersection (PBI) is another widely-used aggregation function for decomposition-based multiobjective optimization (Zhang & Li, 2007):
[Equation]
[Equation]
where θ is the penalty parameter, F ( x )
=
( f 1 ( x ) , . . .
, f m ( x )) and z ∗ =
( z ∗ i , . . .
, z ∗ i ) are the objective vector and ideal vector respectively.
An inverted version of PBI (IPBI) aggregation function (Sato, 2014) can be deﬁned as:
[Equation]
(21) where z N is the nadir vector that contain each objective’s worst value among all Pareto solutions.
For a two-objective optimization problem, when we can ﬁnd a dense set of corresponding solutions to cover the Pareto front for each aggregation function, their performance could be similar to each
[Figure]
other.
However, different aggregation functions would have quite different performances on the problems with three or more objective functions (called many-objective optimization problems).
The performances will heavily depend on the shape of Pareto front (Ishibuchi et al., 2016), especially with a limited number of approximate solutions.
We compare the performance of our proposed method with different aggregation functions on
MOTSP50 with 105 , 1035 and 10011 preferences respectively in Fig. 8.
According to the results, the
IPBI method can generate the most uniformly distributed solutions for the MOTSP problem with an inverted triangular shape of Pareto front, of which the shape is similar to the weight vector distribution (e [Figure]
g., see Fig 5).
This observation is consistent with the ﬁndings and analysis in Ishibuchi et al.
According to these results, we use the Tchebycheff aggregation for all two-objective optimization problems and IPBI aggregation for all problems with more than two objective functions in this work.
[-L [Figure] L 5 
 L 13 (c) 1035 Solutions-]Since the shape of Pareto front tends to be irregular for real-world applications (Ishibuchi et al., 2019), how to properly choose the aggregation function and assign the preference distribution could be an important future work.
[-L 10 
 [Figure] L 5 
 L 13 (d-]D. T HREE -O BJECTIVE MOTSP WITH A SYMMETRIC P ARETO F RONT
[Figure]
e f e r e n c e
[Figure]
[-L 5 
 [Figure] [Figure]-] In this subsection, we conduct experiments on the three-objective MOTSP100 instances with asymmetric Pareto fronts.
The deﬁnition of irregular MOTSP instance is almost the same as in Section C.1, except the coordinates for the three objectives and randomly sampled from [0 , 1] 2 , [0 , 0 .
5]and [0 , 0 . 1] 2 respectively, rather than uniformly from [0 , 1] 6 .
In this way, the objective values for the
MOTSP instance will be in quite different scales, thus leading to an irregular Pareto front (the axes in Figure 9 are in different scales).
A well-known drawback of the scalarization-based approach is that it cannot evenly explore the irregular Pareto front with a set of uniform weights, which can also be observed in Figure 9(a)-(d).
Our proposed approach allows the user to generate arbitrary trade-off Pareto solutions on the inference time, therefore they can directly generate a dense approximation and then select the preferred solutions as in Figure 9(d).
This ﬂexibility can partially address the unevenly distributed issues caused by a (small) set of ﬁxed weights in the traditional scalarization-based approach.
If we know the approximate range of different objectives in advance, we can ﬁrst normalize them into [0 , 1] to encourage a more symmetric Pareto front.
Otherwise, on the inference time, we can use a (prior knowledge-based) biased and non-uniform weight assignment to generate uniformly distributed solutions.
In Figure 9(e)-(h), we ﬁrst multiple the three-dimensional weights by (1 , 2 , 10) and then normalize them back to [0 , 1] 3 which leads to a set of non-uniform weights as shown in
Figure 9(e).
With this weight assignment, we have a a set of more evenly distributed Pareto solutions as shown in Figure 9(f)-(h).
D. P REFERENCE - BASED I NFERENCE
Even without any prior knowledge, our proposed approach allows the user to adaptively adjust the weights in real-time to search for the most suitable solutions in their preferred region(s).
Some examples of selected weights and their corresponding solutions are shown in Figure 10 for symmetric
Pareto front and Figure 11 for asymmetric Pareto front.
If we have prior knowledge of the preference
(e.g., the decision-makers will only care about a speciﬁc region of the Pareto front), we can modify the training preference distribution Λ accordingly to enhance the training efﬁciency.
For the problem with a truly irregular Pareto front, it is also possible to adaptively adjust the given weights to make them evenly explore the Pareto front during the learning/searching process.
One potential direction could be to consider the connection between scalarization and hypervolume maximization as in Zhang & Golovin (2020).
We believe this could be an important research topic for the learning-based scalarization approach in future work.
P r e f e r e n c e0.0 0.2 0.4 0.6 0.8 1. P r e f e r e n c e 2 0.0 0.2 0.4 0. P r e f e r e n c e0.0 0.2 0.
[Table]
P r e f e r e n c e0.0 0.2 0.4 0.6 0.8 1. P r e f e r e n c e 2 0.0 0.2 0.4 0. P r e f e r e n c e0.0 0.2 0. [Table]
[Table]
P r e f e r e n c e0.0 0.2 0.4 0.6 0.8 1. P r e f e r {+e n c e 2 0.0 0.2 0.4 0. P r e f e r e n c e0.
+}
[Table]
[Table] [Table]
[Table]
P r e f e r e n {+c e0.0 0.2 0.
 [Table] [Table]+}
[Table]
P r e f e r e n c e 2 0.
[Figure]
P r e f e r e n c e
[Figure]
P r e f e r e n c e
[Figure]
P r e f e r e n c
[Figure]
P
[Figure]
e r
[Figure]
P r e f e r e n c e
[Figure]
P r e f e r e n
[Figure]
P r e f e r e n c e
[Figure]
P r e f e r e n c e
[Figure]
P r e f e r e n c e
[Figure]
P r e f e r e n c
[Figure]
D. P ROBLEM WITH M ORE O BJECTIVES
[Figure] [Figure]
D,
NSGA-II and our proposed P-MOCO.
Our model is trained on 10 -objective MOKP100.
Finally, we test the performance of our proposed method on the 10 -objective knapsack problems.
We train a new model for the 10 objective MOKP with 100 items with uniform 10 -dimension preferences.
The obtained value path plots on the 10 -objective MOKP100 are shown in Figure 12.
For problems with more objectives, we need a large number of solutions to approximate the Pareto set.
Training a large number of neural network models would have a huge computational and storage overhead, which is also not desirable in practice.
Therefore, we do not compare with the AMMOCO and MOA-DRL methods on this problem.
{+For inference, to approximate the Pareto set, we use a set of 715 ﬁxed preferences following the weight assignment approach from (Das & Dennis, 1998) (with m = 10 , p = 4 , hence n = C 
+}
+4 − 1= 715 ). The model generates different trade-off solution for each preference, so there are 715 different value paths (lines) on each plot.
In MOKP, we want to maximize the values for all objectives under the capacity limitation.
A set of good approximate solutions should have relatively high overall values.
According to the results, our proposed method has the best performance.
We also test the performance of our method on a larger problem with 500 items.
The results shown in Figure 13 conﬁrm that our trained model generalizes well to problems with a larger size.