A Deep Reinforcement Learning Framework for Column Generation
Abstract
Column Generation (CG) is an iterative algorithm for solving linear programs (LPs) with an extremely large number of variables (columns).
CG is the workhorse for tackling large-scale integer linear programs, which rely on CG to solve LP relaxations within a branch and price algorithm.
Two canonical applications are the Cutting Stock Problem (CSP) and Vehicle Routing Problem with Time Windows (VRPTW).
In VRPTW, for example, each binary variable represents the decision to include or exclude a route , of which there are exponentially many; CG incrementally grows the subset of columns being used, ultimately converging to an optimal solution.
We propose RLCG , the first Reinforcement Learning (RL) approach for CG.
Unlike typical column selection rules which myopically select a column based on local information at each iteration, we treat CG as a sequential decision-making problem: the column selected in a given iteration affects subsequent column selections.
This perspective lends itself to a Deep Reinforcement Learning approach that uses Graph Neural Networks (GNNs) to represent the variable-constraint structure in the LP of interest.
We perform an extensive set of experiments using the publicly available BPPLIB benchmark for CSP and Solomon benchmark for VRPTW.
RLCG converges faster and reduces the number of CG iterations by 22.4% for CSP and 40.9% for VRPTW on average compared to a commonly used greedy policy.
Our code is available at .
Introduction
Machine Learning (ML) for Mathematical Optimization (MO) is a growing field with a wide range of recent studies that enhance optimization algorithms by embedding ML in them to replace humandesigned heuristics [Bengio et al., 2021].
Previous work has overwhelmingly focused on combinatorial or integer programming problems/algorithms for which all decision variables are explicitly given in advance.
For example, in a knapsack problem, each binary variable represents the decision to include or exclude an item in the knapsack; even with hundreds of thousands of items, modern integer programming solvers or heuristics can assign values to some or all variables simultaneously without any memory issues.
However, there are many optimization problems in which there are more decision variables than one could ever explicitly deal with.
In VRPTW, for example, each binary variable represents the decision to include or exclude a route , of which there are exponentially many.
Column generation (CG) is an algorithm for solving linear programs (LPs) with a prohibitively large number of variables (columns)
[Desaulniers et al., 2006].
It leverages the insight that in an optimal solution to LP, only a few variables will be active.
In each iteration of CG, a column with negative reduced cost is added to a Restricted Master Problem (RMP, where “restricted" refers to the use of only a subset of columns) until no more columns with negative reduced cost are found.
When that occurs, CG will have provably converged to an optimal solution to the LP.
To solve an integer linear program, CG can be used as an LP relaxation solver within the so-called branch and price algorithm.
In this work, we will focus on CG as an algorithm for solving large-scale linear programs; our conclusion section discusses the straightforward application of our method to the integer case.
A commonly used heuristic rule is to greedily select the column with the most negative reduced cost in each iteration.
However, is this always the “optimal" column to add if one is interested in converging in as few iterations as possible?
Could ML-guided column selection, based on the structure of the current instance being solved, make better selection decisions that speed up the convergence of CG?
These are the questions we tackle in this work.
We adopt the standard view that one is interested in solving problem instances that have the same mathematical formulation but whose data (objective function coefficients, constraint coefficients, num. of vehicles, etc.) differ and are drawn from the same distribution.
In VRPTW, for example, the geographical locations of customers that must be served by the vehicles may vary, as do the corresponding service time windows.
To tackle these questions, we propose a ML framework to accelerate the convergence of the CG algorithm.
In particular, we utilize RL to select columns for LPs with many variables, where the state of the CG algorithm and the structure of the LP instance are encoded using GNNs that operate on a bipartite graph representation of the variable-constraint interactions, similar to Gasse et al.
Q-learning is used to derive a column selection policy that minimizes the total number of iterations through an appropriately designed reward function.
Our premise is that by directly optimizing for convergence speed using RL on similar LP instances from the same problem (e.g., CSP, VRPTW), one can outperform traditional heuristics such as the greedy rule.
Our contributions can be summarized as follows:
CG as a sequential task: We bring forth the first integration of RL and CG which appropriately captures the sequential decision-making nature of CG algorithm.
Our RL agent, RLCG , learns a Q-function which takes the future rewards (namely, total number of iterations) into consideration and therefore can make better column selections at each step.
In contrast with prior work on RL for mathematical optimization, ours is, to our knowledge, the first to tackle the very widely applicable class of problems with exponentially many variables.
Curricula for learning over a diverse set of instances: In practice, instances of the same optimization problem may vary not only in their data, but also their size/complexity.
For example, in the CSP, instances may vary in the number of rolls that must be cut.
To enable efficient RL over a widely varying instance distribution, we show how a curriculum can be designed for a given dataset of instances with a minimal amount of domain knowledge.
Evidence of substantial improvements over existing myopic heuristics: We evaluate RLCG on the CSP, which is known as the representative problem in this domain [Ben Amor and Valerio de Carvalho, 2004, Desaulniers et al., 2006], and the VRPTW, another widely studied and applied problem.
We compare RLCG with the commonly used greedy column selection strategy and an expensive, integer programming based one-step lookahead strategy described by Morabit et al.
Our algorithm converges faster than both of these in terms of the number of iterations and total time.
Our results show the value of considering CG as a sequential decision-making problem and optimizing the entire solving trajectory through RL.
Related Work
Work on the use of RL to guide iterative algorithms can be traced back to Zhang and Dietterich [1995] who used RL for a scheduling problem.
More recently, Dai et al.
[2017] and Bello et al. [2016] proposed deep RL for constructing solutions to graph optimization problems; the survey by Mazyavkina et al.
[2021] summarizes subsequent advances in this space.
Dai et al.
[2017] were the first to use GNNs in this setting, a line of work that has also grown substantially (e.g., Cappart et al.
[2021]) including in integer programming [Gasse et al., 2019].
Relatedly, Tang et al.
[2020] apply RL to the cutting plane algorithm for integer linear programming, where a policy that selects “good" Gomory cuts is derived using evolutionary strategies.
Within the framework of Constraint
Programming, Cappart et al.
[2020] present a deep RL approach for branching variable selection within an exact algorithm.
Nonetheless, to our knowledge, the sequential decision-making perspective has not been leveraged in the context of column generation or integer/linear programming with many variables, a setting which is relevant to many practical applications such as resource allocation (e.g.,
CSP), routing problems (e.g., VRPTW), and airline crew scheduling [Barnhart et al., 2003].
The closest work to ours is that of Morabit et al.
They formulate the column selection in each
CG iteration as a column classification task where the label of each column (select or not) is given by an “expert".
This “expert" performs a one-step lookahead to identify the column which maximally improves the LP value of the Restricted Master Problem (RMP) of the next iteration.
This is done with an extremely time-consuming mixed-integer linear program (MILP).
The RMPs are encoded using bipartite graphs with columns nodes ( v ) and constraint nodes ( c ), where an edge between v and c in the graph indicates the contribution of a column v to a constraint c .
Each node of the graph is then annotated with additional useful information for column selection stored as node features.
Morabit et al.
[2021] then use a GNN to imitate the node selection of the expert using supervision, similarly to what was done for branching by Gasse et al.
In contrast, our work treats the column generation as a sequential decision-making problem and utilizes RL to select a column at each iteration of CG.
Our GNN acts as a Q-function approximator that maximizes the total future expected reward.
As such, our work focuses on directly reducing the total number of CG iterations, whereas Morabit et al.
[2021] derive a classifier that does not consider the interdependencies across iterations, treating them as independent.
One approach to accelerate
CG further is to add multiple columns per CG iteration Desaulniers et al.
[1999]; we discuss this extension in the Conclusion.
Preliminaries: Column Generation
We will use the canonical Cutting Stock Problem (CSP) to describe the CG method as is typically done in textbooks on the topic [Desaulniers et al., 2006].
CSP is a general resource allocation problem where the objective is to subdivide a given quantity of a resource into a number of predetermined allocations to meet certain demands so that the total usage of the resources (e.g., total number of sizefixed paper rolls) is minimized.
Such minimization is achieved by finding a set of optimal divisions of each resource, or in other words, using a set of optimal cutting patterns to divide resources.
Due to the CSP’s combinatorial nature and its exponentially large set of possible patterns (variables), CG is used to solve the LP relaxation of CSP iteratively without explicitly enumerating all possible patterns.
The CSP formulation and column generation algorithm we use is a common modification of Gilmore and Gomory [1961].
The set of all feasible patterns P that can be cut from a roll is defined as:
[Equation]
where each pattern p ∈ P is represented using a vector x k ∈ N n .
With a i being a possible cut length from a roll with length L , each element of x k specifies how many such cuts with length a i are included in pattern p .
For instance, assume the length of the resource roll L is 4m, so all possible a i s are 1m, 2m, 3m and 4m, and then one possible cutting pattern p is represented by x k = (0 , 2 , 0 , 0) .
Let λ p be the number of times pattern p is used.
The formulation with λ p being a decision variable is:
[Equation]
where the objective function minimizes the total number of patterns used, which is equivalent to minimizing the number of rolls used.
The constraints ensure demand is met, while enforcing the integrality restriction on λ p .
This problem has an extremely large number of decision variables as P is exponentially large.
Therefore, the problem is decomposed into the Restricted Master Problem (RMP) and the SubProblem (SP).
The RMP is obtained by relaxing the integrality restrictions on λ p with an initial set ˜ P where ˜ P ⊂ P .
The RMP formulation of the cutting stock problem is defined as follows:
[Equation]
The SP formulation of the provided cutting stock problem is defined as follows:
[Equation]
where π i is the dual value associated with the demand constraints.
The SP is used to generate a pattern p represented by vector x ∈ N n with the most negative reduced cost, then adding that p to ˜ P in the next iteration.
Below, we provide an overview of the column generation algorithm, noting that our method will intervene in Step 4 to make a potentially non-greedy column selection decision:
Solve the RMP to obtain λ ⋆ and ¯ π ; 2.
Update the SP objective function using ¯ π ; 3.
Solve SP to obtain x ⋆i ; 4.
if 1 − (cid:80) {+n i =
+}
¯ π i x ⋆ i ≤ 0 , add the column associated with x ⋆ i and return to step 1, else Stop.
The RLCG Framework
At a high level, our RL-aided column selection strategy, RLCG , works as follows.
We assume that the sub-problem (SP) is solved at each iteration and a set of near-optimal column candidates G is returned, which is a general feature of optimization solvers such as Gurobi.
While the greedy CG algorithm, as described in Section 3, adds the single column with the most negative reduced cost from G to the RMP of the next iteration, RLCG selects columns from G according to the Q-function learned by the RL agent.
The RL agent is fused within the CG loop and actively selects the column to be added to the next iteration using the information extracted from the current RMP and SP.
An illustration comparing CG and RLCG is provided in Figure 1.
4.1 Formulating CG as MDP
We formulate CG as a Markov decision process (MDP).
As is customary, we use ( S , A , T , r , γ ) to denote our MDP, where S is the state space,
A the action space, T : S × S ×
A → [0 , 1] , ( s ′ , s, a ) (cid:55)→ P ( s ′ | s, a ) the transition function, r : S ×A×A → R the reward function, and γ ∈ (0 , 1) the discount factor.
We train the RL agent with experience replay [Mnih et al., 2015].
4.1.1 State space S
The state represents the information that the agent has about the environment.
In RLCG , the environment is the CG solution process corresponding to a given problem instance.
As shown in Figure 1, the information passed to the RL agent is the bipartite graph of the current CG iteration from the RMP and the candidate columns from the SP.
At each iteration, the RMP is an LP as shown in Section 3.
As introduced in Gasse et al.
[2019], such an LP is encoded using a bipartite graph with two node types: column nodes V and constraint nodes C .
An edge ( v, c ) exists between a node v ∈ V and a node c ∈ C if column v contributes to constraint c .
An example bipartite graph representation of the state is shown in left of Figure 2 with column nodes shown on the left hand side, the constraint nodes on the right hand side, and the action nodes, which are the candidate columns returned from the SP, are shown in green (e.g., v 6 , v 7 ).
To incorporate richer information about the current CG iteration, we include node features for both column nodes and constraint nodes ( vf and cf next to the nodes).
We designed 9 column node features and 2 constraint node features in our environment based on our previous experience with CG and inspiration from Morabit et al.
These node features are described in Appendix G.
As such, the state space S is the space of bipartite graphs representing all possible RMPs from the problem instances drawn from the distribution D with the node features given above.
The bipartite graph shown in the left of Figure 2 is a particular state s in S .
As states are bipartite graphs with node features, it is natural to use a Graph Neural Network (GNN) as the Q-function approximator in our
DQN agent.
This bipartite graph representation not only encodes variable and constraint information in the RMP, but also interactions between variables and constraints through its edges.
4.1.2 Actions, transition, and reward
Action space A .
As shown in Figure 2, the RL agent selects one column to add to the RMP for the next iteration from the candidate set G returned from current iteration SP.
Therefore, the action space A contains all possible candidate columns that can be generated from SPs; for example, the green nodes v 6 and v 7 in Figure 2.
As the action space is discrete and the state space is continuous, the GNN (Q-network) performs the operation of returning action values in the current state, i.e., ˆ q ( s, a 1 ; w ) , . .
, ˆ q ( s, a m ; w ) .
Transition function T .
Transitions are deterministic.
After selecting an action from the current candidate set G , the selected column enters the basis in the next RMP iteration.
We then delete the action nodes that were not selected from the bipartite graph (current state), turn the selected action node into a column node, solve the new CG’s RMP and SP, update all the features, and augment all the action nodes returned from the next SP iteration into the left-hand side of the graph, which results in a new bipartite graph state.
Take the bipartite graph state shown in the left of Figure 2 as an example, and assume action v 6 is selected at this iteration.
The transition occurs as follows: v 6 (in grey) becomes a column node, and candidate columns v 8 and v 9 returned by SP (in green) are added.
Reward function R .
The reward function consists of two components: (1) the change in the RMP objective value, where a bigger decrease in value is preferred; (2) a unit penalty for each additional iteration.
Together, they incentivize the RL agent to converge faster.
The reward at time step t is defined as:
[Equation]
where obj 0 is the objective value of the RMP in the first CG iteration and is used to normalize ( obj t − 1 − obj t ) across instances of various sizes; α is a non-negative hyperparameter that weighs the normalized objective value change in the reward.
4. RLCG training and execution
Algorithm 1 shows how a trained RLCG agent is applied to solve a problem instance.
The MDP components ( S , A , T , r, γ ) used in this section are defined in Section 4.1.
Before starting the iterative optimization, the initialization steps 1–3 build the initial bipartite graph with computed node features and add an initial set of columns into basis (for instance, in CSP, we first add simple cutting patterns into the basis to initialize).
Inside the while loop, a column is selected from the candidate set G based on the best Q-value computed by the RL agent.
Then, the RMP and the SP are updated in the same way as the traditional CG algorithm.
The MDP model corresponds to extracting the MDP components from the current updated RMP and SP, which are discussed in Section 4.1.1 and Section 4.1.2.
Steps 7, 8 correspond to the deterministic state transition T described in Section 4.1.2, and
S t +1 is the resulting state due to action a ∗ t . [Table]
Training.
The DQN algorithm with experience replay is used [Mnih et al., 2015] with the typical mean squared loss between Q ( s 0 , a ) and Q target ( s 0 , a ) ∀ a (all green action nodes) in bipartite graph state s 0 .
Q target ( s 0 , a ) is defined as r + γ · max a Q ( s 1 , a ) .
A GNN is used as Q-function approximator.
Training instances are sequenced based on a domain-specific curriculum described for each of CSP and VRPTW in Section 5.
Experimental Results
Baseline strategies and evaluation metrics.
To assess the performance of RLCG , we consider two baseline methods: the greedy column selection strategy (most negative reduced cost) and the MILP expert column selection of Morabit et al.
[2021] as described in Section 2.
This expert provides an upper bound on the performance of the supervised learning approach of Morabit et al.
[2021], as their ML model is approximately imitating the expert, i.e., it will never select better columns than the expert.
We consider two standard evaluation metrics: (1) Number of iterations for CG to converge; (2) Time in seconds.
For the latter, this includes GNN inference time and feature computations for RLCG .
Our computing environment is described in Appendix Section C.
5.1 Cutting Stock Problem (CSP)
Dataset.
We use BPPLIB Delorme et al.
[2018], a widely used collection of benchmark instances for binary packing and cutting stock problems, which includes a number of instances proven to be difficult to solve [Delorme and Iori, 2020, Wei et al., 2020, Martinovic et al., 2020].
BPPLIB contains instances of different sizes with the roll length n varying from 50 to 1000 and the number of orders m varying from 20 to 500.
Our training set has instances with n = 50 , 100 , 200 .
The remaining instances with n = 50 , 100 , 200 , 750 are split into a validation set and a testing set, with hard instances n = 750 only appearing in the testing set, as it is very expensive to solve such large instances during training.
The detailed statistics of these three sets are listed in Table 4 in Appendix E.
Note that our test set is more challenging than that used in training, allowing us to assess the agent’s generalization ability.
Curriculum Design.
In the RL context, curriculum learning Narvekar et al.
[2020] serves to sort the instances that a RL agent observes during the training process from easy to hard.
In our experiments, we noticed that adopting the curriculum learning paradigm improves the learning of the RL agent and results in better column selection and faster convergence.
We train our RLCG agent by feeding the instances in order of increasing difficulty.
For CSP, instances are sorted based on their roll length n and number of orders m to build a training curriculum (Table 5 in Appendix C).
The detailed comparison of training with and without the curriculum is deferred to Appendix A.
In short, the former is crucial to convergence when the training set contains instances of varying difficulties. The overall training process
Hyperparameter tuning.
We briefly describe the outcome of a large hyperparameter tuning effort which is described at length in Appendix B.
We tune the main hyperparameters, α in the reward function (1), ϵ the exploration probability in DQN, γ the discount factor, and the learning rate lr in gradient descent.
A grid with 81 possible configurations is explored by sampling 31 configurations, training the agent, and evaluating them on a validation set.
The majority of the configurations resulted in agents that outperform the greedy strategy (see Appendix B Figure 10).
The best configuration is: α = 300 , ϵ = 0 .
05 , γ = 0 .
9 and lr = 0 .
For all the experiments conducted below, we use this configuration, both for CSP and VRPTW. [Figure] [Figure] [Figure]
[Figure] Performance comparison.
Figure 3 shows pair-wise comparisons between RLCG and the two baselines in terms of the number of CG iterations (top row) and solving time (bottom row) on the testing instances.
Each point in a plot corresponds to a test instance.
Across all n and for both baselines in Figure 3, the majority of the points are above the diagonal line, which indicates that RLCG outperforms the competing method w.r [Figure]
t. the evaluation metric.
Such a tendency becomes more pronounced as the CSP instances become larger (left to right).
RLCG performs better than the greedy column selection for all the roll lengths n and the performance improvement becomes larger as n grows.
For CSP instances with n = 50 and n = 200 , the MILP expert is able to maintain similar performance compared to RL.
However, solving the MILP expert requires solving a MILP at each iteration with only one-step lookahead which is time consuming.
For CSP instances with n = 750 , RLCG begins to outperform the MILP expert due to its ability to take future rewards into consideration
[Figure]
Notice that even though we did not train RLCG using CSP instances with n = 750 , it was able to perform well on such instances, indicating strong generalization to harder problems.
We also generate box plots to compare statistically the three methods shown in Figure 4.
Detailed statistics can be found in Table 6.
As shown by all the subplots in Figure 4, especially for large instances in the subplots (c) and (f), the proposed RLCG achieves statistically meaningful improvements on the CG convergence speed compared to both the greedy and the expert column selection methods [Figure]
Convergence plots.
In Figure 5, we visualize the CG solving trajectories for all test instances for n = 50 , 200 , 750 .
We record the objective values of the RMP at each CG iteration for given method, then take the average over all instances.
Note that we normalized the objective values to be in [0,1] before taking the average among instances.
Since the MILP expert is by far the slowest, we restricted our attention to the two less expensive methods: RLCG and Greedy.
Looking at Figure 7, it is clear that not only does RLCG terminate in fewer iterations and less time, but it also dominates Greedy throughout the CG iterations.
In other words, if one had to terminate CG earlier, RLCG would result in a better (smaller) objective value as compared to what Greedy would achieve.
5.2 Vehicle Routing Problem with Time Windows (VRPTW)
The VRPTW seeks a set of possible routes for a given number of vehicles to deliver goods to a group of geographically dispersed customers while minimizing the total travel costs.
In the language of CG, each route is one column and there are exponentially many.
Constraints of VRPTW include: vehicle capacity; a vehicle has to start from a depot and return to it; all customers should be served exactly once during their specified time windows.
The detailed formulation of VRPTW is given by Cordeau et al.
[2002]; implementation details are in the appendix.
Dataset.
We use the well-known Solomon benchmark [Solomon, 1987].
This dataset contains six different problem types (C1, C2, R1, R2, RC1, RC2), each of which has 8–12 instances withcustomers.
“C" refers to customers which are geographically clustered, “R" to randomly placed customers, “RC" to a mixture.
The “1" and “2" labels refer to narrow time windows/small vehicle capacity and large time windows/large vehicle capacity, respectively.
The difficulty levels of these sets are in order of C, R, RC.
There are 56 instances in total in Solomon’s dataset, and from each original Solomon instance, we can generate smaller instances by considering only the first n <customers.
We use instances from types C1, R1, RC1 for training.
For the training set, we generated 80 smaller instances per type from the original Solomon’s instances by only considering the first n customers where n is randomly sampled from 5–8, for a total of 240 training instances.
For testing, two sets of instances are considered: (1) 60 small-size instances (number of customers n within same range as training instances); (2) 37 large-size instances (number of customers n from 15–30).
All the test instances are either generated from held-out Solomon instances (e.g., sets C2, R2, RC2) or generated from some training instance but with a much different n .
Curriculum Design.
As in CSP, we designed a curriculum for VRPTW that sequences instances in order of difficulty: C1, R1, then RC1.
This order is based on the fact that clustered instances have more structure that enables “compact" routes for neighboring customers, whereas random instances require more complex routes.
The mixed RC1 instances require reasoning about both types of customers simultaneously [Desrochers, 1992]. [Figure]
Performance comparison.
The hyperparemeters of the RL agent are chosen to be the same as the best set of hyperparameters found for CSP.
Figure 6 shows similar trends to those seen for CSP: RLCG converges in fewer iterations and less time than Greedy on most instances.
This effect is more pronounced for the large VRPTW instances, a fact that can also be observed in the convergence plot of Figure 7 (b): RLCG converges in roughly 100 iterations compared toiterations for Greedy, on average.
Additional box plots are deferred to Appendix J. Note that we do not compare to the Expert here given that it is much too slow and practically intractable. [Figure] [Figure]
[Figure]
Conclusions & Discussion of Limitations
RLCG shows superior performance and better convergence both in terms of the number of iterations and time compared to the greedy column selection and the MILP expert strategy.
In addition, our curriculum learning enables the agent to generalize well when facing harder test instances.
Our experiments on two important large-scale LP families show that there is value in modeling CG as a sequential decision-making problem: taking the future impact of adding a column into account helps convergence.
However, our current work is restricted to adding only one column per CG iteration.
Adding multiple columns per iteration can speed up convergence.
However, this makes the RL action space exponential and thus finding the action with the largest Q-value becomes a combinatorial optimization problem.
Recent work by Delarue et al.
[2020] addresses this problem, and thus our RL formulation can be expanded using the results of this paper to allow for multiple columns.
Alternatively, policy gradient methods could be used instead of Q-learning.
We believe this is an exciting next step but one that stretches beyond the limits of our paper, which is the first ever on RL for CG.
A trained RLCG agent can also be embedded within the branch and price algorithm for solving the integer-constrained versions of CSP/VRPTW and invoked to solve each LP relaxation.
The speed-ups demonstrated herein would transfer to that setting, assuming that an appropriate dataset of training and validation instances can be collected.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Dataset and code link in appendix.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] As discussed in Section 1 and 2, this work is the first application of RL on CG, and the benefits of using RLCG to enhance {+CG is demonstrated in Section 5 and shown in Figure 3 
+}
[-demonstrated in Section 5 and shown in Figure 2 
-]
7 6.Detailed improvement statistics can be verified in Table [-
-]7 6.
(b) Did you describe the limitations of your work?
[Yes] Limitation is discussed at the end of Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code link is provided in Abstract.
Data we use, training and testing procedure are discussed in detail in corresponding sections in 5 and Appendix A C E. Hyperparameter settings for experiments in Appendix B; (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Datasets used, training testing split, training schedule generation are described set described in Section 5.1 for CSP and Section 5.2 for VRPTW.
We conduct complete hyperparameter tuning for CSP in Appendix B and use the best hyperparameters setting for both CSP and VRPTW experiments.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes
The solving process is deterministic so test on each instance once in the testing sets, however, we did normalize and average over all instance results and generate box plots in Figure and plot standard deviation bars in Figure
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Computing environment in Appendix C; 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[Figure] Figure 8 above compares the training trajectories between training with 400 CSP instances following the sequence provided in Table 5 and training with the same 400 CSP instances randomly ordered. In (a), as every 40 instances we increase the CSP difficulty.
Although there is an upward trend in the training curve, however, within each instance difficulty setting (fixed n and m), there is a downward trend showing a sign of learning.
In contrast, there is no clear sign of learning in (b).
Therefore, for all the experiments shown in this paper, the RLCG model is trained using a curriculum. We also visualize the scheduled training process of RLCG for CSP using a validation set. For every 20 training episodes, we stop the training process and validate the current model with the validation set. The result is shown in Figure 9.
[Figure]
g e n
[Figure]
RLCGGreedy
B Hyperparameter tuning
Table 1 shows RL related parameters.
In Table 2, we provide GNN parameters that we used.
We conduct hyperparameters tuning and sensitivity analysis using a validation set over: the parameter α used in equation 1 to weight the change in the normalized objective value used in our reward function, the exploration parameter of the RL agent ϵ , the discount factor γ , and the learning rate lr .
All other hyperparameters and their values are listed in Table 1 and Table 2.
The values we consider for each hyperparameter are the following: α ∈ {0, 100, 300}, ϵ ∈ {0.01, 0.05, 0.2}, γ ∈
[Table]
[Table]
{0.9, 0.95, 0.99} and lr ∈ {0.01, 1e-3, 3e-4}.
We choose the value for α ∈ {0, 100, 300} because when α = 0 , we place no weight on decreasing the objective value, otherwise, α = 100 , 300 will bring the normalized change in the objective values into similar scale as the step penalty.
Therefore, the search space for hyperparameters is defined as the Cartesian product between all these sets of different hyperparameters possible values, which gives us 81 configurations, and we randomly select 31 configurations out of them.
Then we train 31 RLCG models corresponding to selectedconfigurations, and we evaluate these models using our validation set.
The validation metric or reward is defined as the ratio of the total number of iterations RLCG takes to solve each CSP instance divided by the total number of iterations greedy takes.
For each model, we compute such ratio for all the instances in validation set, and we generate the following box plot shown in Figure 10 showing the validation metric for all the models.
Among the 31 configurations we tested, the majority of the models were able to outperform greedy strategy on the instances in the validation set.
The best configuration is model 3: α = 300 , ϵ = 0 .
05 , γ = 0 .
9 and lr = 0 .
For all the results reported in this paper, we use this configuration.
This includes VRPTW, although no direct hyperparameter tuning has been done for this problem class.
To assess the sensitivity of the RL training with respect to randomness such as the GNN initialization and the exploration in RL, we compare the average validation reward relative to greedy for the selected model across five random seeds.
The average varies between 1.23 and 1.25, indicating little to no sensitivity.
In Table 3, we provide detailed configurations of each model index for our analysis of hyperparameters as well as their detailed validation results.
Validation reward relative to Greedy
H y pe r pa
r
a m e t e r c on ﬁ gu r a t i on
The detailed hyperparameter configurations each index refers to are listed in Appendix B Table 10.
The horizontal axis shows the ratio of RLCG model solving iterations to greedy column selection solving iterations.
The blue vertical line shows the ratio threshold indicating same performance as greedy
The best to worst models are ordered
C Computing environment
To implement GNN, we use Tensorflow 2.7.0.
To solve both RMP and SP optimization problems in
CG, we use Gurobi 9.5.0.
For the training using 400 instances schedule for CSP and 240 instances schedule for VRPTW, the training takes around 8-10 hours CPU time using the following CPU settings: Intel 2.30 Ghz, 2 CPU Cores, Intel (R) Xeon(R), Haswell CPU family.
D Graph neural network for bipartite graph
Graph Neural Network (GNN) has been successfully applied to many different machine learning tasks with graph structured data.
GNN includes a message passing method where the features of each node for each node pass to other neighbouring nodes through learned transformations to generate aggregated information, and such information can be used for node classification, edge selection so on so forth.
Due to the effectiveness of GNN to utilize graphical structure of the data, catch node-level dependencies, and has permutation invariant properties, GNN is an appropriate method for performing node(column) selection task in the present column generation problem.
For this study we are encoding the information for each iteration of CG using a bipartite graph
G =
( E, V ) as the state, where each column and constraint is represented by node v ∈ V and there is an edge e ∈ E between columns and constraints only if the column contributes to the constraint.
Detailed encoding for state is discussed in Section 4.1.1.
As we are using a bipartite graph, the GNN we use should be able to achieve convolution on a bipartite graph with two types of nodes (variable nodes and constraint nodes), and we utilize the similar bipartite GNN as in the study conducted by (Morabit et al.
[2021]) with modification of the task it performs (from binary classification to Q value regression).
Here we give a brief overview of how convolution or the features update is achieved in this bipartite GNN.
The features update is done in two phase: phase 1 updates the constraint features and phase 2 updates the column features.
In phase 1, the constraint features in next iteration is obtained by applying a non-linear transformation of previous constraint node features and aggregate information of its neighbouring nodes, which are column nodes that are connected to this constraint node.
This can be treated as information passing from variable nodes to constraint nodes.
Similarly,the second phase can be seen as message passing from constraint nodes to variable nodes.
Once the column node features have been updated for several iterations for all the column nodes, these features are fed into a fully connected layer, which results in Q values for each column node.
[Table]
Model(0, 0.2, 0.99, 0.001) 0.
Model(300, 0.2, 0.95, 0.001) 1.
Model(300, 0.2, 0.9, 0.0003) 1.
Model(0, 0.05, 0.9, 0.0003) 0.
Model(0, 0.2, 0.9, 0.001) 0.
Model(300, 0.05, 0.99, 0.0003) 1.
Model(300, 0.2, 0.99, 0.001) 1.
Model(100, 0.2, 0.9, 0.01) 1.
Model(300, 0.05, 0.9, 0.001) 1.
Model(100, 0.2, 0.95, 0.001) 1.
Model(100, 0.01, 0.9, 0.001) 1.
Model(0, 0.2, 0.9, 0.001) 1.
Model(0, 0.2, 0.9, 0.001) 0.
Model(0, 0.05, 0.95, 0.001) 0.
Model(100, 0.05, 0.95, 0.0003) 1.
E Training, Validation, Testing set
Table 4 below shows the information of the instances contained in the training set, validation set and testing set for CSP.
Column lists total number of instances in each dataset, while other columns list the number of instances with specific roll length n in that dataset.
The division of VRPTW dataset can be found in Section 5.2.
[Table]
F Curriculum Learning design
In Table 5, we provide data characteristic we considered, for the sake of curriculum learning of the RL agent.
In this paper the RL agent is trained on instances that are ordered according to their difficulty level.
To accomplish this instances are divided into three categories of easy, normal and hard according to the stock length for CSP.
Easy instances have stock length of 50, normal instances have stock length of 100 and hard instances have stock length of 200.
There are 40 instances for each instance type.
Details of training curriculum of different instance types are shown in table 5.
Figure 8 displays the number of steps to convergence vs. instance number for training instances.
It is clear that for each instance type the steps taken for convergence decreases as the model is trained on more of the same instance type.
This shows that the
RL agent successfully learns to select columns to enter basis.
However, when instances are ordered randomly (Figure 8b) there are no specific trend on the steps taken to converge.
This highlights the necessity of curriculum learning.
Curriculum for VRPTW can be found in Section 5.2.
[Table]
G Node Features
Node features used for CSP :
Column node features : Feature (a) and (c) relate to solving the RMP problem as they are all information about decision variables in RMP, and each column node corresponds to one decision variable.
Feature (b) and (d) are determined by the problem formulation of each cutting stock instance, while feature (e) - (i) corresponds to the dynamical information of each column entering and leaving the basis.
(a) Reduced cost : Reduced cost is a quantity associated with each variable indicating how much the objective function coefficient on the corresponding variable must be improved before the solution value of the decision variable will be positive in the optimal solution (the cutting pattern will be used in optimal set of cutting patterns).
The reduced cost value is only non-zero when the optimal value of a variable is zero.
(b) Connectivity of column node : Total number of constraint nodes each column node connects to.
As each constraint is a particular demand, this node feature indicates the ability of a column node (a pattern) to satisfy demands.
It also indicates the connectivity of each column node in the bipartite graph representing the state.
(c) Solution value : The solution value of each decision variable corresponding to each column node after solving the RMP in the current iteration.
For each column node, this feature is continuous number greater than or equal to 0.
The candidate column nodes have this feature set to be 0.
(d) Waste : A feature recording the remaining length of a roll if we were to cut the current pattern from the roll.
Again, each column node corresponds to one decision variable in RMP, which also represents one particular cut pattern.
(e) Number of iterations that each column node stays in the basis : If the column node stays in the basis for a long time, it is most likely that the pattern corresponds to this column node is really good.
(f) Number of iterations that each column node stays out of the basis : if the column node stays out of the basis for a long time, it is most likely never enters the basis and being used in optimal solution in future iterations.
(g)
If the column left basis on the last iteration or not : This is a binary feature recording the dynamics of each column node.
(h)
If the column enter basis on the last iteration or not : Similar binary feature as (f).
(i) Action node or not : A binary feature indicating whether a column node is a candidate (a newly added action) or not.
If the column node is a candidate node (column) that is generated at the current iteration by SP, then this binary feature is 1 otherwise 0.
Constraint node features : Each constraint node corresponds to one constraint in RMP, so the
number
of
constraint nodes
are fixed for each cutting stock problem instance.
(a) Dual value : Dual value or shadow price is the coefficient of each dual variable in sub-problem objective function, and as each constraint node corresponds to one dual variable, we record dual value as one feature for constraint node.
(b) Connectivity of constraint node : Total number of column nodes each constraint node connects to, which also indicates the connectivity of each constraint node in the bipartite graph representing the state.
VRPTW node features used are quite similar to CSP:
Column node features : Reduced cost, connectivity of column node, solution value, route cost, Number of iterations that each column node stays in the basis, Number of iterations that each column node stays in the basis, If the column left basis on the last iteration or no, If the column enter basis on the last iteration or no. 2.
Constraint node features : Dual value, connectivity of constraint node
H Detailed statistics of testing results
In Table 6, we provide statistics obtained from our experimental results for CSP.
We report both the average and standard deviation of number of iterations, solution time measured in seconds, and the objective function values.
Note that the resulting objective function values between column selection policies might differ due to the early-stopping criteria we adopted.
Note that for the sake table spacing, we use Objval to denote objective function value, and µ, σ for mean and standard deviation, respectively.
We observe the clear dominance of RLCG over the potential of RLCG in solving challenging CG problems in practice.
[Table]
In Table
[Table]
we provide the same statistics obtained from our experimental results for VRPTW.
[Table]
I Per instance results CSP
[Table]
rc103.txt, n =  2135.
rc107.txt, n =  766.
rc105.txt, n =  578.
rc101.txt, n =  106.
rc102.txt, n =  999.
rc108.txt, n =  5449.
r102.txt, n =  960.
c101.txt, n =  88.
r105.txt, n =  3877.
c107.txt, n =  5609.
c105.txt, n =  3874.
rc201.txt, n =  1752.
rc106.txt, n =  991.
r101.txt, n =  100.
r205.txt, n =  1498.
[Table]
[Table]
[Table]
[Table]
J VRPTW box plots
Similar to CSP, we see the upward generalization of RLCG in VRPTW: we only train our model with small sized instances, and tested on large instances with more customers, and we observe that RLCG was able to perform well using VRPTW testing instances with large size.
These results, again indicate that our proposed RLCG can be preferable in solving challenging column generation problems because of its ability to generalize well.
Besides, compare large testing results with small testing results for
VRPTW (also compare n=750 results with n=50,200 for CSP), we observe that the more challenging the problem, the larger gap exist between RLCG and benchmark method.
This indicates that for CG problem we considered, the harder the problem, the more important considering the future effect of
CG becomes, thus taking future into consideration would drastically accelerate the solving process of
CG.
[Figure]
[Figure] [Table] K per instance results CSP
[Table] [Table] [Table]
[Table]
[Table]