Learning Elimination Ordering for Tree Decomposition Problem
Abstract
We propose a Reinforcement Learning-based approach to approximately solve the Tree Decomposition problem.
Recently, it was shown that learned heuristics could successfully solve combinatorial problems.
We establish that our approach successfully generalizes from small graphs, where an optimal Tree Decomposition can be found by exact algorithms, to large instances of practical interest, while still having very low time-to-solution.
On the other hand, the agent-based approach surpasses all classical greedy heuristics by the quality of the solution
Introduction
At the core of many practical tasks, such as probabilistic inference, decision making, planning, and other problems, lies a combinatorial (NP-complete) optimization problem.
The solution of large NP-problems is often possible only with the help of heuristics.
These heuristics are designed manually, which is a complicated and time-consuming process.
The resulting algorithm is also typically domain-speciﬁc and can not be reused.
Recently, Reinforcement Learning (RL) application to design heuristics gained signiﬁcant attention [2, 19, 15].
RL is a natural framework for the automatic design of approximation algorithms for problems with an inherent cost function and large search space, which is the essence of combinatorial optimization.
In this work, we consider the
Tree Decomposition (TD) [6] problem.
The TD is central to the analysis of the complexity and the topological structure of graphs.
An integer parameter treewidth characterizes the solution of the TD problem.
The treewidth quantiﬁes the complexity of many NP-problems; the computational cost of solving these problems is exponential in the treewidth, but only polynomial in the problem’s graph size.
Tree Decomposition emerges as a core step in various contexts, such as probabilistic inference [14] or shortest path search [9].
The TD problem is usually solved on non-Euclidean graphs, as opposed to the traveling salesman problem (TSP), which is the most common target of recent studies of trainable heuristics [19, 2].
Several exact [13, 22, 7] and approximate [5] algorithms exist to solve the TD problem.
We propose a Reinforcement Learning-based approach for TD.
In this work, we demonstrate that the resulting policy can successfully generalize to problems with different graph structures and sizes.
Our ﬁndings show that the agent can be trained even on a single graph .
The quality of the solution of our agent-based procedure is superior compared to all simple greedy heuristics, and the time-to-solution is much lower compared to advanced algorithms
Graph Convolutional Policy
We start with a formulation of Tree Decomposition as a linear ordering problem of a graph’s vertices.
Then we shortly deﬁne a graph agent, which ﬁnds an ordering to calculate a TD.
During tree decomposition, an arbitrary input graph is mapped to a tree graph.
A full deﬁnition of
Tree Decomposition can be found in the original work [20] or in a classical review [6].
Informally, the Tree Decomposition measures how close a given graph resembles a tree.
We provide a more formal deﬁnition in Appendix A.
The quality of TD is measured by treewidth , which should be minimized across all possible trees.
It can be shown that ﬁnding a tree with the lowest treewidth is
NP-hard [3].
Instead of building the tree directly, in this work, we will search for TD by using its relation to the ordering of vertices [4].
The procedure to build a ﬁnal tree given a permutation of vertices is described in Appendix A.
A permutation π ( u ) : u ∈ U → [1 . . .
| U | ] of vertices of a graph G =
( U, E ) is called an elimination order.
Consequently, π − 1 ( t ) : t ∈ [1 . . .
| U | ]
→ U is an inverse function of the order.
Given a number π − 1 ( t ) returns a node.
The elimination order of the graph can be used in the following procedure for a graph G :
For t ∈ [1 . . .
| U | ] , take the t -th node u = π − 1 ( t ) .
Connect all neighbors of u into a clique (fully connected subgraph) and record the size of the resulting clique.
Remove u .
This results in a new graph G t .
Repeat the procedure until G t is empty.
If G has treewidth at most k , then there is an elimination order π of G, such that each vertex has at most k + 1 neighbors in the elimination procedure with respect to π [1].
We deﬁne the maximal number of neighbors associated with a permutation π as c π .
The treewidth is a minimum of c π −across all possible permutations, i.e. tw ( G ) =
min π c π − 1 .
If the treewidth of a graph is small, then it is tree-like.
In particular, a tree has treewidth 1.
We deﬁne our problem as follows: given an undirected graph G , ﬁnd an elimination order, i.e., a permutation of the vertices, such that the number of neighbors in the elimination procedure along π is minimal across all permutations.
The example of the elimination procedure is shown in Appendix A.
Graph Agent. At every step t of the elimination process we represent the graph G t with n nodes as the pair ( A t , H t ) , where A t is the adjacency matrix and H t ∈ R n × d is the node feature matrix, assuming that the graph has d features.
Input features H t 0 is a constant matrix initialized with ones.
H t 0 is passed through a graph neural network, which parameterizes our Graph Convolution Policy (GCP), and is transformed into H t .
We experiment with different graph neural architectures, which consist of three types of functions: message passing, aggregation, and update function.
Our approach to feature extraction is similar to the one used in the recent work [25].
The details of the architecture are listed in Appendix B.1.
To formulate the TD problem in the RL framework, we should deﬁne a Markov decision process (MDP) for the agent.
At each time step t , the agent selects a node of the graph G t based on the observed information, represented by the state features.
The node is then eliminated from G t , and the graph G t +1 is produced.
We deﬁne the MDP as a tuple ( S , A , P, r, γ ) , where the A is a set of actions, S is a set of states and r is the cost for the sequential combinatorial problem.
A consists of nodes u ∈ U t ⊂ U , which have not been eliminated at the current step.
For the TD problem, r is the size of a maximal clique in the graph obtained during the elimination process.
Each state s t ∈ S directly deﬁned as the embedding matrix H ( l ) and the structure of the graph G t .
We use original GCN [17] to extract features on every step, and conduct experiments with its extensions [26, 24].
Solving an MDP means ﬁnding an optimal policy Π , a mapping which outputs a distribution of actions.
We apply an Actor-Critic [18] algorithm to solve this problem.
Experiments
In this section, we demonstrate that the agent can successfully learn a heuristic.
We compare our results with common human-designed heuristics and show that our neural heuristic trained on a single graph can generalize on the graphs with many vertices.
3.1 Data
We use three datasets with different structures for the experiments.
The ﬁrst dataset consists of random Erd˝os–R´enyi (ER) graphs [11] with edge probability 5 /n , where n is the number of nodes.
We experimentally found that this choice of the edge probability leads to hard instances of TD problem (the ER graphs contain many cycles and are not too dense).
For the validation, we use a ﬁxed set
D of 100 Erd˝os–R´enyi graphs with 10 to 1000 nodes and the set D small as ﬁrst 50 graphs from D .
The second dataset is taken from the PACE2017 competition [10] on designing TD algorithms. The
[Figure] third dataset comprises graphs that emerge during the simulation of random quantum circuits [8], a common framework in the study of quantum computing supremacy.
The main reason for using the
ER graphs is to simplify the reproducibility of the experiments.
Our main results are obtained on the PACE2017 dataset, explicitly created to test TD algorithms.
The third dataset is selected as a practical TD application example.
3.2 Baseline and Evaluation Details
Before starting to experiment with our model, we should introduce other algorithms used in this work.
We use two greedy heuristics, two specialized TD solvers and adapt the S2V-DQN [15] method to the TD problem.
The greedy heuristics are minimal degree and minimal ﬁll-in [6].
They work fast enough and are accurate in practice [23].
The minimal degree heuristic selects nodes with a minimal number of neighbors.
The minimal ﬁll-in algorithm selects the nodes such that the number of introduced edges at each step is minimized.
Two specialized solvers are based on very different approaches, and both will produce an exact solution if provided enough running time (exponential in the graph size).
The ﬁrst solver by Tamaki et al. [22] employs the connection of TD and vertex separators.
This method searches for optimal TD directly in the space of tree graphs without referring to vertices’ ordering.
It is a winner algorithm on the PACE2017 competition.
Another powerful solver is QuickBB [13], which is based on the branch and bound algorithm.
We restrict both solvers’ runtime to 30 minutes per graph instance as in the PACE2017 competition [10].
We also compare with another Reinforcement Learning approach called S2V-DQN [15].
This algorithm was previously used to solve sequential optimization problems on graphs, such as Minimum
Vertex Cover, and we adapt it to our tasks.
We train the S2V-DQN model on a single graph to have faster convergence.
To produce a solution using our RL-based heuristic, we sample 10 trajectories and take the one with the lowest treewidth.
We compare the performance of different solvers with respect to the Tamaki solver [22].
As a performance metric an Approximation Ratio (AR), AR = tw method ( G ) tw tamaki ( G ) is used.
The
AR metric is the standard in the literature on approximation algorithms.
The treewidth is calculated from the elimination order π produced by the solvers.
Training Graph.
Surprisingly, we ﬁnd that the agent’s score does not signiﬁcantly depend on the source of the training graph, despite the graphs which we use have quite different structures.
Also, the training set’s size does not signiﬁcantly affect ﬁnal accuracy but rather resulted in longer training times.
More details are provided in Appendix B.2. This fact may be attributed to the agent’s inefﬁciency or the uniform structure of the TD problem already for average-sized graphs (with 50 nodes).
We admit that this intriguing fact may need additional investigation.
To simplify the reproducibility of our experiments, we choose Erd˝os–R´enyi graphs for further tests.
As can be seen from Figure 4a in Appendix B.3, the accuracy of the GCN-agent depends on the number of nodes only slightly.
For further experiments, we select a graph with 70 nodes.
[Table]
3.3 Comparison with Other Solvers
In this section
[Table]
we compare our agent’s performance to other methods on the graphs of the different sizes and structures.
The agent is trained on the ER graph with 70 nodes.
The results of all experiments are summarized in Figure 1.
The learned solver usually does not reach the accuracy of specialized exact algorithms but outperforms greedy heuristics.
It is interesting to see that one can learn a relatively useful heuristic using only a single graph.
The quality of the agent’s solution deteriorates as the test graphs’ size grows, but usually slower than the quality of the solution found by greedy heuristics.
The PACE2017 dataset results, which we consider the most representative
[Table]
are summarized in Table 1.
We note that in addition to the high accuracy comparing to greedy heuristics (both on average and in the worst case), the RL-based heuristic has very competitive time-to-solution, which is essential for solving NP problems.
Furthermore, this time can be signiﬁcantly decreased, as sampling can be trivially performed in parallel.
One explanation of why the agent works this way is the excellent ability of graph neural networks to approximate dynamic programming algorithms, which is demonstrated in [25].
Conclusion
This work formulates the elimination process for Tree Decomposition as a task for RL.
We propose a model that can directly learn how to solve this combinatorial optimization problem using a single graph for training and simple GCN.
The training procedure also can be performed using a large set of graphs, but this work aims to show the generalization ability of a simple agent trained on a single graph.
We provide experiments that a learnable heuristic can surpass greedy, manually designed ones.
Our method can generalize to large problem instances without signiﬁcantly sacriﬁcing the solution’s quality and without a large increase in computational time.
Our preliminary results suggest that this approach is a good starting point for learning heuristics for combinatorial problems on graphs.
Outperforming specialized algorithms is still a challenge in this setting.
The generalization ability of our agent-based approach could also be fruitful for algorithmic reasoning tasks.
The perspective direction of research is a combination of RL-based heuristics with local-search methods and an extension of this method for tensor contraction task.
APPENDIX A Elimination Ordering
Here we provide examples of the elimination procedure and discuss its connection to tree decomposition.
Consider two elimination procedures of the same graph in Fig.
In the ﬁrst case (upper part), the maximal clique size is 3 (and hence the treewidth of the associated TD is 2).
In the second case, the treewidth is 1, and the order is optimal (since the graph is a tree).
Note that the optimal order may not be unique: nodes number 1 and 2 can be swapped, for example, but the same treewidth is achieved. [Figure]
Let us formally introduce Tree Decomposition.
A Tree Decomposition is a mapping of the initial graph G =
( U, E ) into a tree graph F =
( B, T ) .
Here U is the set of nodes, and E is the set of edges of the initial graph; B is the set of bags (each bag b ∈ B is a subset of nodes of the graph G , b ⊂ U ) and T is the set of the edges of the tree graph.
A Tree Decomposition has to fulﬁll three criteria to be valid:
Every node of G is in some bag, i.e., ∪ b ∈ B b = U .
For every edge ( u, v ) ∈
E there must be a bag such that both endpoints are in that bag, i.e., ∃ b : u ∈ b, v ∈ b . 3.
For every node u of G , the subgraph of the tree F , induced by all bags that contain u , is a connected tree.
Tree decomposition is inherently related to the elimination procedure and can be built provided some order of node eliminations is chosen.
To build a TD, one has to add all cliques which emerge during the elimination procedure as the vertices B .
A minimum spanning tree T over B , which is consistent with the criteria mentioned in the previous paragraph, is the TD.
Pseudocode for building a TD given an elimination order can be found, for example, in [21].
The treewidth is the size of the maximal bag minus 1.
Conversely, given a TD, multiple equivalent elimination orders (in the sense of the maximal clique/treewidth) can be found.
This procedure can also be found in [21].
B Additional Experiments
This section contains details of additional experiments.
All of them are aimed at choosing the agent training procedure and the analysis of generalization.
B.1 Training Details
Our code is based on the PyTorch Geometric [12].
We train all models with Adam [16] with parameters β 1 = 0 .
9 , β 2 = 0 .
999 and learning rate lr = 0 .
We also set the parameters of the RL algorithm as follows: the discount factor γ = 0 .
999 , the GAE weight λ = 0 .
85 , the weight of the value loss β value = 1 . 0 , and the multiplier of the entropy regularization β entropy = 0 .
The GCN subnetwork contains three layers, and the hidden feature size is 64, as in [15].
A 2-layer perceptron parameterizes policy and value heads with a hidden dimension equal to 64.
All hyperparameters are selected with grid search on the hold-out validation set.
We train our model on the NVIDIA 1080ti with one thread for sampling.
[Figure]
B.2 Choice of the Training Graph
In this series of experiments [Figure] we would like to check how the choice of training dataset affects the resulting agent performance.
In deep RL, it is common to train the agent on a huge number of problem instances and expect an increase in the model quality from a larger number of training examples.
We check this intuition in the following experiment.
First, we train the agent on a single graph.
We choose one ER graph with 68 nodes and one graph from the PACE dataset (”he064”).
Next [Figure] we train on different ER and PACE graphs with up to 200 nodes.
We randomly select a graph for every training step.
The performance of the agents is compared to the same validation set of
Erd˝os–R´enyi graphs.
The results are shown in Figure 3.
It turns out it is sufﬁcient to train our agent for 10 epochs (around 20 minutes)
[Figure]
t i o a b c d e f g 0.
a b c d f g a b c d f 0.
a b c d b c d b d d
B.3 Training Graph size
In this set of experiments, we consider training on graphs of different sizes.
It is known that if the action space is large, the RL methods may have problems with convergence due to the signiﬁcant variance of the gradients.
It is hence desirable to keep the training graph size small (for example, less than 100 nodes); however, training on larger instances may produce agents with better performance.
To check the inﬂuence of the training graph size on the learned heuristic accuracy, we train separate agents on ER graphs with 10 to 210 nodes.
The dependence of the AR is shown in Figure 4a.
It can be noted that better accuracy is obtained for the graphs with 70 and 130 nodes, which is approximately the GCN feature size or twice feature size.
The ﬁnal result on the validation set slightly depends on the number of nodes on the training graph.
B.4 Agent Decision Making
In this section, we analyze the agent’s decision-making process in order to get insights into the structure of the TD problem.
As mentioned earlier, in all experiments, we use sampling to get candidate solutions and select one with the best score.
We use a small sample of 10 trajectories, which is a stronger result compared to other works on neural-based heuristics, where samples of size 1000 are common [19].
The efﬁciency with a small sample size suggests that the agent learns a distribution close to the ”true” distribution of solutions.
Another evidence in support of the high quality of the learned distribution is the behavior of the agent on complete graphs.
A fully connected graph with n nodes has treewidth n − 1 , and all elimination orders on it are equivalent.
The policy learned by our agent produces a uniform distribution on the complete graphs (after they appeared during the elimination process), as shown in Figure 4b.