O N R EPRESENTING L INEAR P ROGRAMS BY G RAPH N EURAL N ETWORKS
A BSTRACT
Learning to optimize is a rapidly growing area that aims to solve optimization problems or improve existing optimization algorithms using machine learning (ML).
In particular, the graph neural network (GNN) is considered a suitable ML model for optimization problems whose variables and constraints are permutation–invariant, for example, the linear program (LP).
While the literature has reported encouraging numerical results, this paper establishes the theoretical foundation of applying GNNs to solving LPs.
Given any size limit of LPs, we construct a GNN that maps different LPs to different outputs.
We show that properly built GNNs can reliably predict feasibility, boundedness, and an optimal solution for each LP in a broad class.
Our proofs are based upon the recently–discovered connections between the Weisfeiler–Lehman isomorphism test and the GNN.
To validate our results, we train a simple GNN and present its accuracy in mapping
LPs to their feasibilities and solutions.
I NTRODUCTION
Applying machine learning (ML) techniques to accelerate optimization, also known as Learning to Optimize (L2O) , is attracting increasing attention.
It has been reported that L2O shows great potentials on both continuous optimization (Monga et al., 2021;
Chen et al., 2021; Amos, 2022) and combinatorial optimization (Bengio et al., 2021; Mazyavkina et al., 2021).
Many of the L2O works train a parameterized model that takes the optimization problem as input and outputs information useful to classic algorithms, such as a good initial solution and branching decisions (Nair et al., 2020), and some even directly generate an approximate optimal solution (Gregor & LeCun, 2010).
In these works, one is building an ML model to approximate the mapping from an explicit optimization instance either to its key properties or directly to its solution.
The ability to achieve accurate approximation is called the representation power or expressive power of the model.
When the approximation is accurate, the model can solve the problem or provide useful information to guide an optimization algorithm.
This paper tries to address a fundamental but open theoretical problem for linear programming (
[Equation]
Which neural network can represent LP and predict its key properties and solution?
[Equation]
To clarify, by solution we mean the optimal solution.
Let us also remark that this question is not only of theoretical interest.
Although currently neural network models may not be powerful enough to replace those mathematical-grounded LP solvers and obtain an exact LP solution, they are still useful in helping LP solvers from several perspectives, including warm-start and configuration.
It requires that neural networks have sufficient power to recognize key characteristics of LPs.
Some very recent papers (Deka & Misra, 2019; Pan et al., 2020; Chen et al., 2022) on DC optimal power flow (DC-OPF), an important type of LP, experimentally show the possibility of fast approximating
LP solutions with deep neural networks.
Practitioners may initialize an LP solver with those approximated solutions.
We hope the answer to (P0) paves the way toward answering this question for other optimization types.
Linear Programming (LP).
LP is an important type of optimization problem with a wide range of applications, such as scheduling (Hanssmann & Hess, 1960), signal processing (Candes & Tao, 2005), machine learning (Dedieu et al., 2022), etc.
A general LP problem is defined as:
[Equation]
where A ∈ R m × n , c ∈ R n , b ∈ R m , l ∈ ( R ∪ {−∞} ) n , u ∈ ( R ∪ { + ∞} ) n , and ◦ ∈ {≤ , = , ≥} m .
Any LP problems must follow one of the following three cases (Bertsimas & Tsitsiklis):
• Infeasible.
The feasible set X F := { x ∈ R n : Ax ◦ b, l ≤ x ≤ u } is
empty
In another word, there is no point in R n that satisfies the constraints in LP (1.1).
• Unbounded.
The feasible set is non-empty, but the objective value can be arbitrarily good, i.e., unbounded from below.
For any R > 0 , there exists an x ∈ X F such that c ⊤ x < − R .
• Feasible and bounded.
There exists x ∗ ∈ X F such that c ⊤ x ∗ ≤ c ⊤ x for all x ∈ X F .
Such x ∗ is named as an optimal solution, and c ⊤ x ∗ is the optimal objective value.
Thus, considering (P0), an ideal ML model is expected to be able to predict the three key characteristics of LP: feasibility, boundedness, and one of its optimal solutions (if exists), by taking the
LP features ( A, b, c, l, u, ◦ ) as input.
Actually, such input has a strong mathematical structure.
If we swap the positions of the i, j -th variable in (1.1), elements in vectors b, c, l, u, ◦ and columns of matrix A will be reordered.
The reordered features ( ˆ A, b, ˆ c, ˆ l, ˆ u, ˆ ◦ ) actually represent an exactly equivalent LP problem with the original one ( A, b, c, l, u, ◦ ) .
Such property is named as permutation invariance .
If we do not explicitly restrict ML models with a permutation invariant structure, the models may overfit to the variable/constraint orders of instances in the training set.
Motivated by this point, we adopt Graph Neural Networks (GNNs) that are permutation invariant naturally.
GNN in L2O. GNN is a type of neural networks defined on graphs and widely applied in many areas, for example, recommender systems, traffic, chemistry, etc (Wu et al., 2020;
Zhou et al., 2020).
Accelerating optimization solvers with GNNs attracts rising interest recently (Peng et al., 2021; Cappart et al., 2021).
Many graph-related optimization problems, like minimum vertex cover, traveling salesman, vehicle routing, can be represented and solved approximately with GNNs due to their problem structures (Khalil et al., 2017; Kool et al., 2019; Joshi et al., 2019;
Drori et al., 2020).
Besides that, one may solve a general LP or mixed-integer linear programming (MILP) with the help of
GNNs.
Gasse et al. (2019) proposed to represent an MILP with a bipartite graph and apply a GNN on this graph to guide an MILP solver.
Ding et al. (2020) proposed a tripartite graph to represent
MILP.
Since that, many approaches have been proposed to guide MILP or LP solvers with GNNs (Nair et al., 2020; Gupta et al., 2020; 2022; Shen et al., 2021; Khalil et al., 2022; Liu et al., 2022;
Paulus et al., 2022; Qu et al., 2022; Li et al., 2022).
Although encouraging empirical results have been observed, theoretical foundations are still lack for this approach.
Specifying (P0), we ask:
Are there GNNs that can predict the feasibility, boundedness and an optimal solution of LP?
(P1)
Related works and contributions.
To answer (P1), one needs the theory of separation power and representation power .
Separation power of a neural network (NN) means its ability to distinguish two different inputs.
In our settings, a NN with strong separation power means that it can outputs different results when it is applied on any two different LPs.
Representation power of NN means its ability to approximate functions of interest.
The theory of representation power is established upon the separation power.
Only functions with strong enough separation power may possess strong representation power.
The power of GNN has been studied in the literature (see Sato (2020); Jegelka (2022); Li & Leskovec (2022) for comprehensive surveys), and some theoretical efforts have been made to represent some graph-related optimization problems with GNN (Sato et al., 2019; Loukas, 2020).
However, there are still gaps to answer question (P1) since the relationships between characteristics of LP and properties of graphs are not well established.
Our contributions are listed below:
• (Separation Power).
In the literature, it has
been shown that
the separation power of GNN is equal to the WL test (Xu et al., 2019; Azizian & Lelarge, 2021; Geerts & Reutter, 2022).
However, there exist many pairs of LPs that cannot be distinguished by the WL test.
We show that those puzzling LP pairs share the same feasibility, boundedness, and even an optimal solution if exists.
Thus, GNN has strong enough separation power.
• (Representation Power).
To the best of our knowledge, we established the first complete proof that GNN can universally represent a broad class of LPs.
More precisely, we prove that, there exist GNNs that can be arbitrarily close to the following three mappings: LP → feasibility, LP → optimal objective value ( −∞ if unbounded and ∞ if infeasible), and LP → an optimal solution (if exists), although they are not continuous functions and cannot be covered by the literature (Keriven & Peyr´e, 2019; Chen et al., 2019; Maron et al., 2019a;b; Keriven et al., 2021).
• (Experimental Validation).
We design and conduct experiments that demonstrate the power of GNN on representing LP.
The rest of this paper is organized as follows.
In Section 2, we provide preliminaries, including related notions, definitions and concepts.
In Section 3, we present our main theoretical results.
The sketches of proofs are provided in Section4.
We validate our results with numerical experiments in
Section 5 and we conclude this paper with Section 6.
P RELIMINARIES
In this section, we present concepts and definitions that will be used throughout this paper.
We first describe how to represent an LP with a weighted bipartite graph, then we define GNN on those
LP-induced graphs, and finally we further clarify question (P1) with strict mathematical definitions.
2. LP REPRESENTED AS WEIGHTED BIPARTITE GRAPH
Before representing LPs with graphs, we first define the graph that we will adopt in this paper: weighted bipartite graph .
A weighted bipartite graph G =
( V ∪ W, E ) consists of a vertex set
V ∪ W that are divided into two groups V and W with V ∩ W = ∅ , and a collection E of weighted edges, where each edge connects exactly one vertex in V and one vertex in W .
Note that there is no edge connecting vertices in the same vertex group.
E can also be viewed as a function E :
V
× W
→ R .
We use G m,n to denote the collection of all weighted bipartite graphs G =
( V ∪ W, E ) with | V | = m and | W | = n .
We always write V = { v 1 , v 2 , . .
. , v m } , W = { w 1 , w 2 , . . . , w n } , and
E i,j = E ( v i , w j ) , for i ∈ { 1 , 2 , . . .
, m } , j ∈ { 1 , 2 , . . .
, n } .
One can equip each vertex with a feature vector.
Throughout this paper, we denote h Vi ∈ H V as the feature vector of vertex v i ∈ V and denote h W j ∈ H W as the feature vector of vertex w j ∈ W , where
H V , H W are feature spaces.
Then we define H Vm :
=
( H V ) m ,
H Wn :
=
( H W ) n
and concatenate all the vertex features together as H
=
( h V 1 , h V 2 , . .
, h Vm , h W 1 , h W 2 , . . .
, h Wn ) ∈
H Vm × H Wn .
Finally, a weighted bipartite graph with vertex features is defined as a tuple ( G, H ) ∈ G m,n × H Vm
× H Wn .
With the concepts described above, one can represent an LP (1.1) as a bipartite graph (Gasse et al., 2019): Each vertex in W represents a variable in LP and each vertex in V represents a constraint.
The graph topology is defined with the matrix A in the linear constraint.
More specifically, let us set
• Vertex v i represents the i -th constraint in Ax ◦ b , and vertex w j represents the j -th
variable x j . • Information of constraints is involved in the feature of v i :
h Vi = ( b i , ◦ i ) .
• The space of constraint features is defined as H V :
= R × {≤ , = , ≥} .
• Information of variables is involved in the feature of w i :
h Wj = ( c j , l j , u j ) .
• The space of variable features is defined as H W :
= R ×
( R ∪ {−∞} ) × ( R ∪ { + ∞} ) .
• The edge connecting v i and w j has weight E i,
j = A i,j .
Then an LP is represented as a graph ( G, H ) ∈ G m,n ×H Vm ×H Wn .
In the rest of this paper, we coin such graphs as LP-induced Graphs or LP-Graphs for simplicity.
We present an LP instance and its corresponding LP-graph in Figure 1.
[Equation]
vvwwh V
[Equation]
h V
[Equation]
=
[Equation]
h W 1 = (1 , 0 , + ∞ ) h W 2 = (2 , − 1 , + ∞ )
2. G RAPH NEURAL NETWORKS FOR LP
The GNNs in this paper always take an LP-Graph as input and the output has two cases:
• The output is a single real number.
In this case, GNN is a function G m,n × H Vm
× H
Wn
→ R and usually used to predict the properties of the whole graph.
• Each vertex in W has an output.
Consequently, GNN is a function G m,n × H Vm
× H
Wn
→ R n .
Since W represents variables in LP, GNN is used to predict properties of each variable in this case.
Now we define the GNN structure precisely.
First we encode the input features into the embedding space with learnable functions f V in : H V
→ R d 0 and f W in : H W → R d 0 :
[Equation]
[Equation]
(2.1) where h 0 ,Vi , h 0 ,Wj ∈ R d 0 are initial embedded vertex features and d 0 is their dimension.
Then we choose learnable functions
f Vl , f Wl : R d l − 1 → R d l and g Vl , g Wl : R d l − 1 × R d l → R d l and update the hidden states with 1 :
[Equation]
j f Wl ( h l − 1 ,W j ) (cid:19) , i = 1 , 2 , . . .
, m, (2.2) h l,Wj = g Wl (cid:18)
h l − 1 ,W j , m (cid:88) i =
E i,j f Vl ( h l − 1 ,V i ) (cid:19) , j = 1 , 2 , . . .
, n, (2.3) where h l,Vi , h l,Wj ∈ R d l are vertex features at layer l (1 ≤ l ≤ L ) and their dimensions are d 1 , · · · , d L respectively.
The output layer of the single-output GNN is defined with a learnable function f out : R d L × R d L → R : y out = f out (cid:18) m (cid:88) i =
h L,Vi , n (cid:88)
j = h L,Wj (cid:19) .
The output of the vertex-output GNN is defined with f W out :
R d L × R d L × R d L → R : y out ( w j ) =
f W out (cid:18) m (cid:88) i =
h L,Vi , n (cid:88)
j = h L,Wj , h L,Wj (cid:19) , j = 1 , 2 , · · · , n.
We denote collections of single-output GNNs and vertex-output GNNs with F GNN and F W GNN , respectively:
F
[Equation] [Equation]
→ R | F yields (2.1) , (2.2) , (2.3) , (2.4) } ,
F
[Equation]
F
[Equation]
→
[Equation]
n | F yields (2.1) , (2.2) , (2.3) , (2.5) } .
In practice, all the learnable functions in GNN f V in , f W in , f out , f W out , { f Vl , f Wl , g Vl , g Wl } Ll =0 are usually parameterized with multi-linear perceptrons (MLPs).
In our theoretical analysis, we assume for simplicity that those functions may take all continuous functions on given domains, following the settings in Azizian & Lelarge (2021, Section C.1).
Thanks to the universal approximation properties of MLP (Hornik et al., 1989; Cybenko, 1989), one can extend our theoretical results by taking those learnable functions as large enough MLPs.
2. R EVISITING QUESTION (P1)
With the definitions of LP-Graph and GNN above, we revisit the question (P1) and provide its precise mathematical description here.
First we define three mappings that respectively describe the feasibility, optimal objective value and an optimal solution of an LP (if exists).
Feasibility mapping
The
feasibility
mapping is a classification function
[Equation]
where Φ feas ( G, H )
= 1 if the LP associated with ( G, H ) is feasible and Φ feas ( G, H )
=
otherwise.
Optimal objective value mapping Denote
[Equation]
as the optimal objective value mapping, i.e., for any ( G, H ) ∈ G m,n × H Vm
×
H
Wn , Φ obj ( G, H ) is the optimal objective value of the LP problem associated with ( G, H ) .
Remark 2.1.
The optimal objective value of a LP problem can be a real number or ∞ / − ∞ .
The “ ∞ ” case corresponds to infeasible problems, while the “ −∞ ” case consists of LP problems whose objective function is unbounded from below in the feasible region.
The preimage of all finite real numbers under Φ obj , Φ − 1 obj ( R ) , actually describes all LPs with finite optimal objective value.
Remark 2.2.
In the case that a LP problem has finite optimal objective value, it is possible that the problem admits multiple optimal solutions.
However, the optimal solution with the smallest ℓ 2 -norm must be unique.
In fact, if x ̸ = x ′ are two different solutions with ∥ x ∥ =
∥ x ′ ∥ , where ∥ · ∥ denotes the ℓ 2 -norm throughout this paper.
Then 12 ( x + x ′ ) is also an optimal solution due to the convexity of LPs, and it holds that ∥ 12 ( x + x ′ )
∥ 2 < 12 ∥ x ∥ 2 + 12 ∥ x ′ ∥ 2 = ∥ x ∥ 2 = ∥ x ′ ∥ 2 , i.e., ∥ 12 ( x + x ′ )
∥ <
∥ x ∥ = ∥ x ′ ∥ , where the inequality is strict since x ̸ = x ′ .
Therefore, x and x ′ cannot be optimal solutions with the smallest ℓ 2 -norm.
Optimal solution mapping For any ( G, H ) ∈ Φ −
[Equation] [Equation]
, we have remarked before that the LP problem associated with ( G, H ) has a unique optimal solution with the smallest ℓ 2 -norm.
Let
[Equation]
be the mapping that maps (
G, H ) ∈
Φ − 1 obj ( R ) to the optimal solution with the smallest ℓ 2 -norm.
Invariance and Equivariance
We denote S m , S n as the group consisting of all permutations on vertex groups V, W respectively.
In another word, S m involves all permutations on the constraints of LP and S n involves all permutations on the variables.
In this paper, we say a function F :
G m,n × H Vm
× H
Wn
→ R is invariant if it satisfies
[Equation]
and a function F W : G m,n × H Vm
× H Wn
→ R n is equivariant if it satisfies
[Equation]
S
[Equation]
where ( σ V , σ W ) ∗ ( G, H ) is the permuted graph obtained from reordering indices in ( G, H ) using ( σ V , σ W ) , which is the group action of S m × S n on G m,n × H Vm
× H Wn .
One can check that Φ feas , Φ obj , and any F ∈ F GNN are invariant, and that Φ solu and any F W ∈ F W GNN are equivariant.
Question (P1) actually asks: Does there exist F ∈ F GNN that well approximates Φ feas or Φ obj ?
And does there exist F W ∈ F W GNN that well approximates Φ solu ?
M AIN R ESULTS
This section presents our main theorems that answer question (P1).
As we state in the introduction, representation power is built upon separation power in our paper.
We first present with the following theorem that GNN has strong enough separation power to represent LP.
Theorem 3.1.
Given any two LP instances ( G, H ) , ( ˆ G, ˆ H ) ∈ G m,n × H Vm × H Wn , if F ( G, H ) =
F ( ˆ G, ˆ H ) for all F ∈ F GNN , then they share some common characteristics:
(i) Both LP problems are feasible or both are infeasible, i.e., Φ feas ( G, H )
=
Φ feas ( ˆ G, ˆ H ) .
(ii)
The two LP problems have the same optimal objective value,
i
e., Φ obj ( G, H ) =
Φ obj ( ˆ G, ˆ H ) .
(iii)
If both problems are feasible and bounded, they have the same optimal solution with the smallest ℓ 2 -norm up to a permutation, i.e., Φ solu ( G, H ) =
σ W (Φ solu ( ˆ G, ˆ H )) for some
σ
W ∈ S n .
Furthermore, if F W ( G, H ) = F W ( ˆ G, ˆ H ) , ∀ F W ∈ F W GNN , then (iii) holds without taking permutations, i.e., Φ solu ( G, H ) =
Φ solu ( ˆ G, ˆ H ) .
This theorem demonstrates that the function spaces F GNN and F W GNN are rich enough to distinguish the characteristics of LP.
Given two LP instances ( G, H ) , ( ˆ G, ˆ H ) , as long as their feasibility or boundedness are different, there must exist F ∈ F GNN that can distinguish them: F ( G, H ) ̸ =
F ( ˆ G, ˆ H ) .
Moreover, as long as their optimal solutions with the smallest ℓ 2 -norm are different, there must exist F W ∈ F W GNN that can distinguish them: F W ( G, H )
̸
= F W ( ˆ G, ˆ H ) .
With Theorem 3.
served as a foundation, we can prove that GNN can approximate the three mappings Φ feas , Φ obj and Φ solu to arbitrary precision.
Before presenting those results, we first define some concepts of the space G m,n × H Vm × H Wn .
Topology and measure Throughout this paper, we consider G m,n × H Vm
× H Wn , where H V = R × {≤ , = , ≥} and H W = R ×
( R ∪ {−∞} ) × ( R ∪ { + ∞} ) , as a topology space with product topology and a measurable space with product measure.
It’s enough to define the topology and measure of each part separately.
Since each graph in this paper (without vertex features) can be represented with matrix A ∈ R m × n , the graph space G m,n is isomorphic to the Euclidean space R m × n : G m,n ∼ = R m × n
and we equip G m,n with the standard Euclidean topology and the standard
Lebesgue measure.
The real spaces R in H W and H V are also equipped with the standard Euclidean topology and Lebesgue measure.
All the discrete spaces {≤ , = , ≥} , {−∞} , and { + ∞} have the discrete topology, and all unions are disjoint unions.
We equip those spaces with a discrete measure µ ( S ) =
| S | , where | S | is the number of elements in a finite set S .
This finishes the whole definition and we denote Meas ( · ) as the measure on G m,n × H Vm × H Wn .
Theorem 3.2.
Given any measurable X ⊂ G m,n × H Vm × H Wn with finite measure, for any ϵ > 0 , there exists some F ∈ F GNN , such that
[Equation]
where I · is the indicator function, i.e., I F ( G,H ) > 1 / 2 = 1 if F ( G, H ) > 1 / 2 and
I F ( G,H ) >
1 / 2 =
otherwise.
This theorem shows that GNN is a good classifier for LP instances in X as long as X has finite measure.
If we use F ( G, H ) > 1 / 2 as the criteria to predict the feasibility, the classification error rate is controlled by ϵ/ Meas ( X ) , where ϵ can be arbitrarily small.
Furthermore, we show that GNN can perfectly fit any dataset with finite samples, which is presented in the following corollary.
Corollary 3.3.
For any D ⊂ G m,n × H Vm
× H Wn with finite instances, there exists F ∈ F GNN that
[Equation]
Besides the feasibility, GNN can also approximate ϕ obj and ϕ solu .
Theorem 3.4.
Given any measurable X ⊂ G m,n × H Vm × H Wn with finite measure, for any ϵ > 0 , there exists F 1 ∈ F GNN such that
[Equation]
and for any ϵ, δ > 0 , there exists F 2 ∈ F GNN such that
[Equation]
Recall the definition of Φ obj in (2.8) that it can take {±∞} as its value.
Thus, Φ obj ( G, H ) ∈ R means the LP corresponding to ( G, H ) is feasible and bounded with a finite optimal objective value, and inequality (3.1) illustrates that GNN can identify those feasible and bounded LPs among the whole set X , up to a given precision ϵ .
Inequality (3.2) shows that GNN can also approximate the optimal value.
The measure of the set of LP instances of which the optimal value cannot be approximated with δ -precision is controlled by ϵ .
The following corollary gives the results on dataset with finite instances.
Corollary 3.5.
For any D ⊂ G m,n × H Vm
× H Wn with finite instances, there exists F 1 ∈ F GNN such that
[Equation]
and for any δ > 0 , there exists F 2 ∈ F GNN , such that
[Equation]
Finally, we show that GNN is able to represent the optimal solution mapping Φ solu .
Theorem 3.6.
Given any measurable X ⊂ Φ − 1 obj ( R ) ⊂ G m,n
× H Vm
× H Wn with finite measure, for any ϵ, δ > 0 , there exists some F W ∈ F W GNN , such that
[Equation]
Corollary 3.7.
Given any D ⊂ Φ − 1 obj ( R ) ⊂ G m,n
× H Vm
× H Wn with finite instances, for any δ > 0 , there exists F W ∈ F W GNN , such that
[Equation]
S KETCH OF P ROOF
In this section, we will present a sketch of our proof lines and provide examples to show the intuitions.
The full proof lines are presented in the appendix.
Separation power
The separation power measures a neural network with whether it generates different outcomes given different inputs, which serves as a foundation of the representation power.
The separation power of GNNs is closely related to the Weisfeiler-Lehman (WL) test (Weisfeiler & Leman, 1968), a classical algorithm to identify whether two given graphs are isomorphic.
To apply the WL test on LP-graphs, we describe a modified WL test in Algorithm 1, which is slightly different from the standard WL test.
Require
A graph
( G, H ) ∈ G m,n
× H Vm
×
H
Wn
L
=
0 ,V ( h Vi ) , C
,Wj = HASH 0 ,W ( h Wj ) .
=
(cid:80) nj =
HASH ′
(cid:16) C l − 1 ,W j (cid:17)(cid:17) .
C l,Wj = HASH l,W (cid:16) C l − 1 ,W j , (cid:80) mi
=1 E i,j HASH ′
l,V (cid:16) C l − 1 ,V i (cid:17)(cid:17) .
5: end for 6: return
{{ C L,Vi }} mi =0 , {{ C L,Wj }} nj =0 .
We denote Algorithm 1 by WL LP ( · ) , and we say that two LP-graphs ( G, H ) , ( ˆ G, ˆ H ) can be distinguished by Algorithm 1 if and only if there exist a positive integer L and injective hash functions { HASH l,V , HASH l,W } Ll =0 ∪ { HASH ′ l,V , HASH
l,W } Ll =1 such that WL LP (cid:0) ( G, H ) , L (cid:1) ̸
=
WL LP (cid:0) ( ˆ G, ˆ H ) , L (cid:1) .
Unfortunately, there exist infinitely many pairs of non-isomorphic LP-graphs that cannot be distinguished by Algorithm 1.
Figure 2 provide such an example.
Since the separation power of GNNs is actually equal to the WL test (Xu et al., 2019), one would expect that the limitation of the WL test might restrict GNNs from universally representing LP.
However, any two LP-graphs that cannot be distinguished by the WL test must share some common characteristics even if they are not isomorphic.
For example, let us consider the six LP instances in
Figure 2.
In each of the three columns, the two non-isomorphic LP instances cannot be distinguished by the WL test.
It can be checked that the two instances in the same column share some common characteristics.
More specifically, both instances in the first column are infeasible; both instances in the second column are feasible but unbounded; both instances in the third column are feasible and bounded with (1 / 2 , 1 / 2 , 1 / 2 , 1 / 2) being the optimal solution with the smallest ℓ 2 -norm.
Actually, this phenomenon does not only happen on the instances in Figure 2, but also serves as an universal principle for all LP instances.
We summarize the results in the following theorem:
Theorem 4.1.
If ( G, H ) , ( ˆ G, ˆ H ) ∈ G m,n ×H Vm ×H Wn are not distinguishable by Algorithm 1, then Φ feas ( G, H ) =
Φ feas ( ˆ G, ˆ H ) and Φ obj ( G, H ) =
Φ obj ( ˆ G, ˆ H ) .
vvvvwwwwmin x 1 + x 2 + x 3 + x 4 , s.t. x 1 + x 2 = 1 , x 2 + x 3 = 1 , x 3 + x 4 = 1 , x 4 + x 1 = 1 , x j ≥ 1 , 1 ≤ j ≤ 4 .
min x 1 + x 2 + x 3 + x 4 , s.t. x
1 + x 2 ≤ 1 , x 2 + x 3 ≤ 1 , x 3 + x 4 ≤ 1 , x 4 + x 1 ≤ 1 , x j ≤ 1 , 1 ≤ j ≤ 4 .
min x 1 + x 2 + x 3 + x 4 , s.t. x 1 + x 2 = 1 , x 2 + x 3 = 1 , x 3 + x 4 = 1 , x 4 + x 1 = 1 , x j ≤ 1 , 1 ≤ j ≤ 4 .
vvvvwwwwmin x 1 + x 2 + x 3 + x 4 , s.t. x 1 + x 2 = 1 , x 2 + x 1 = 1 , x 3 + x 4 = 1 , x 4 + x 3 = 1 , x j ≥ 1 , 1 ≤ j ≤ 4 .
min x 1 + x 2 + x 3 + x 4 , s.t. x
1 + x 2 ≤ 1 , x 2 + x 1 ≤ 1 , x 3 + x 4 ≤ 1 , x 4 + x 3 ≤ 1 , x j ≤ 1 , 1 ≤ j ≤ 4 .
min x 1 + x 2 + x 3 + x 4 , s.t. x 1 + x 2 = 1 , x 2 + x 1 = 1 , x 3 + x 4 = 1 , x 4 + x 3 = 1 , x j ≤ 1 , 1 ≤ j ≤ 4 .
· · C l,Vand C l,W= · · · C l,Wfor all l ≥ 0 , whatever the hash functions are chosen.
Based on this graph pair, we construct three pairs of LPs that are both infeasible, both unbounded, both feasible bounded with the same optimal solution, respectively.
Furthermore, if ( G, H ) , ( ˆ G, ˆ H ) ∈
Φ − 1 obj
( R ) , then it holds that Φ solu ( G, H ) =
σ W (Φ solu ( ˆ G, ˆ H )) for some σ W ∈ S n .
In other words, the above theorem guarantees the sufficient power of the WL test for separating LP problems with different characteristics, including feasibility, optimal objective value, and optimal solution with smallest ℓ 2 -norm (up to permutation).
This combined with the following theorem, which states the equivalence of the separation powers of the WL test and GNNs, yields that GNNs also have sufficient separation power for LP-graphs in the above sense.
Theorem 4.2.
For any ( G, H ) , ( ˆ G, ˆ H ) ∈ G m,n
× H Vm
× H Wn , the followings are equivalent:
(i) ( G, H ) and ( ˆ G, ˆ H ) are not distinguishable by Algorithm 1.
(ii) F ( G, H )
=
F ( ˆ G, ˆ H ) , ∀ F ∈ F GNN .
(iii)
For any F W ∈ F W GNN , there exists σ W ∈ S n such that F W ( G, H ) =
σ W ( F W ( ˆ G, ˆ H )) .
Theorem 4.2 extends the results in Xu et al. (2019); Azizian & Lelarge (2021); Geerts & Reutter (2022) to the case with the modified WL test (Algorithm 1) and LP-graphs.
Representation power Based on the separation power of GNNs, one is able to investigate the representation/approximation power of GNNs.
To prove Theorems 3.2, 3.4, and 3.6, we first determine the closure of F GNN or F W GNN in the space of invariant/equivariant continuous functions with respect to the sup -norm, which is also named as the universal approximation .
The result of F GNN is stated as follows, where C ( X, R ) is the collection/algebra of all real-valued continuous function on X .
The result of F W GNN can be found in the appendix.
Theorem 4.3.
Let X ⊂ G m,n × H Vm
×
H
Wn be a compact set.
For any Φ ∈ C ( X, R ) that satisfies Φ( G, H )
=
Φ( ˆ G, ˆ H ) for all ( G, H ) , ( ˆ G, ˆ H ) ∈
X that are not distinguishable by Algorithm 1, and any ϵ > 0 , there exists F ∈ F GNN such that
[Equation]
Theorem 4.3 can be viewed as an LP-graph version of results in Azizian & Lelarge (2021);
Geerts &
Reutter (2022).
Roughly speaking, graph neural networks can approximate any invariant continuous function whose separation power is upper bounded by that of WL test on compact domain with arbitrarily small error.
Although our target mappings Φ feas , Φ obj , Φ solu are not continuous, we prove (in appendix) that they are measurable.
Applying Lusin’s theorem (Evans & Garzepy, 2018, Theorem 1.14), we show that GNN can be arbitrarily close to the target mappings except for a small domain.
N UMERICAL E XPERIMENTS
We present the numerical results that validate our theoretical results in this section.
We generate
LP instances with m = 10 and n = 50 that are possibly infeasible or feasible and bounded.
To check whether GNN can predict feasibility, we generate three data sets with 100 , 500 , 2500 independent LP instances respectively, and call the solver wrapped in scipy.optimize.linprog to get the feasibility, optimal objective value and an optimal solution for each generated LP.
To generate enough feasible and bounded LPs to check whether GNN can approximate the optimal objective value and optimal solution, we follow the same approach as before to generate LP randomly and discard those infeasible LPs until the number of LPs reach our requirement.
We train GNNs to fit the three LP characteristics by minimizing the distance between GNN-output and those solver-generated labels.
The building and the training of the GNNs are implemented using TensorFlow .
The codes are modified from Gasse et al. and can be found in https://github.com/liujl11git/GNN-LP.git .
We set L = 2 for all GNNs and those learnable functions f V in , f W in , f out , f W out , { f Vl , f Wl , g Vl , g Wl } Ll =0 are all parameterized with MLPs.
Details can be found in the appendix.
Our results are reported in Figure 3.
[Figure]
All the errors reported in Figure 3 are training errors since generalization is out of the scope of this paper.
In Figure 3a, the “rate of errors” means the proportion of instances with I F ( G,H ) > 1 / 2 ̸ =
Φ feas ( G, H ) .
This metric exactly equals to zeros as long as the number of parameters in GNN is large enough, which directly validates Corollary 3.3: the existence of GNNs that can accruately predict the feasibility of LP instances.
With the three curves in Figure 3a combined together, we conclude that such principle does not violate as the number of samples increases.
This consists with Theorem 3.2.
Mean squared errors in Figures 3b and 3c are respectively defined as E ( G,H )
| F ( G, H ) − Φ feas ( G, H ) | 2 and E ( G,H )
∥ F ( G, H ) −
Φ feas ( G, H ) ∥ 2 .
Therefore, Figures 3b and 3c validates Theorems 3.4 and 3.6 respectively.
Note that all the instances used in Figure 3b are feasible and bounded.
Thus, Figure 3b actually only validates (3.2) in Theorem 3.4.
However, due to the fact that feasibility of an LP is equal to the boundedness of its dual problem, one may dualize each LP and use the conclusion of Figure 3a to validate (3.1) in Theorem 3.4.
Some extra experimental results on generalization, i.e., the performance of the trained models on the test set, are presented in Appendix G.
C ONCLUSIONS
In this work, we show that graph neural networks, as well as the WL test, have sufficient separation power to distinguish linear programming problems with different characteristics.
In addition, GNNs can approximate LP feasibility, optimal objective value, and optimal solution with arbitrarily small errors on compact domains or finite datasets.
These results guarantee that GNN is a proper class of machine learning models to represent linear programs, and hence contribute to the theoretical foundation in the learning-to-optimize community.
Future directions include the size/complexity of
GNNs and the generalization, that are not covered in our current theory but are of great importance.
Another future topic is investing the representation power of graph neural networks for mixedinteger linear programming (MILP), which has been observed with promising experimental results in the literature.
A W EISFEILER -L EHMAN (WL) T EST AND C OLOR R EFINEMENT
The WL test can be viewed as a coloring refinement procedure if there are no collisions of hash functions and their weighted averages.
More specifically, each vertex is colored initially according to the group it belongs to and its feature – two vertices have the same color if and only if they are in the same vertex group and have the same feature.
The initial colors are denoted as
C 0 ,V 1 , C 0 ,V 2 , . .
, C 0 ,Vm , C 0 ,W, C 0 ,W, . .
, C 0 ,Wn .
Then at iteration l , the set of vertices with the same color at iteration l − 1 are further partitioned into several subsets according to the colors of their neighbours – two vertices v i and v i ′ are in the same subset if and only if C l − 1 ,V i
[Equation]
and for any C ∈ { C l − 1 ,W j : 1 ≤ j ≤ n } , (cid:88)
[Equation]
and it is similar for vertices w j and w j ′ .
After such partition/refinement, vertices are associated with the same color if and only if they are in the same subset, which is the coloring at iteration l .
This procedure is terminated if the refinement is trivial, meaning that no sets with the same color are partitioned into at least two subsets, i.e., the coloring is stable.
For more information about color refinement, we refer to Berkholz et al. (2017); Arvind et al.
We then discuss the stable coloring that Algorithm 1 will converge to, for which we made the following definition, where
S =
{ S 1 , S 2 , . . .
, S s } is called a partition of a set S if S 1 ∪ S 2 ∪· · ·∪ S s =
S and S i ∩ S i ′
= ∅ , ∀ 1 ≤ i < i ′ ≤ s .
Definition A.1 (Stable Partition Pair of Vertices) .
Let G =
( V ∪ W, E ) be a weighted bipartite graph with V
= { v 1 , v 2 , . .
. , v m } , W = { w 1 , w 2 , . . . , w n } , and vertex features H
= ( h V 1 , h V 2 , . .
, h Vm , h W 1 , h W 2 , . . .
, h Wn ) , and let I =
{ I 1 , I 2 , . . .
, I s } and J =
{ J 1 , J 2 , . . .
, J t } be partitions of { 1 , 2 , . . .
, m } and { 1 , 2 , . . . , n } , respectively.
We say that ( I , J ) is a stable partition pair of vertices for the graph G if the followings are satisfied:
(i) h
Vi =
h
Vi ′ holds if i, i ′ ∈
I p for some p ∈ { 1 , 2 , . . .
, s } .
(ii) h
Wj = h Wj ′ holds if j, j ′ ∈
J q for some q ∈ { 1 , 2 , . . .
, t } .
(iii)
For any p ∈ { 1 , 2 , . . .
, s } , q ∈ { 1 , 2 , . .
. , t } , and i, i ′
∈
I p , (cid:80) j ∈ J q E i,j = (cid:80) j ∈ J q E i ′ ,j .
(iv) For any p ∈ { 1 , 2 , . . .
, s } , q ∈ { 1 , 2 , . .
. , t } , and j, j ′ ∈
J q , (cid:80) i ∈ I
p E i,j = (cid:80) i ∈ I
p E i,j ′ .
We denote ( I l , J l ) as the partition pair corresponding the coloring at iteration l of Algorithm 1.
Suppose that there are no collisions.
Then it is clear that ( I l +1 , J l +1 ) is finer than ( I l , J l ) , denoted as ( I l +1 , J l +1 ) ⪯
( I l , J l ) , which means that for any I ∈ I
l
+1 and any J ∈ J l +1 , there exist
I ′ ∈
I l and J ′ ∈
J l such that I ⊂ I ′ and J ⊂ J ′ .
In addition, ( I l +1 , J l +1 ) =
( I l , J l ) if and only if ( I l , J l ) is a stable partition pair of vertices.
Note that there is at most O ( | V | + | W | ) iterations leading to strictly finer partition pair, i.e, ( I l +1 , J l +1 ) ⪯ ( I l , J l ) but ( I l +1 , J l +1 )
̸
= ( I l , J l ) .
We can immediately obtain the following result:
Theorem A.2.
If there are no collision of hash functions and their weighted averages, then Algorithm 1 terminates at a stable partition pair of vertices in O ( | V | + | W | ) iterations.
Furthermore, for every ( G, H ) ∈ G m,n × H Vm × H Wn , the coarsest stable partition pair of vertices exists and is unique, which can be proved using techniques similar to the proof of Berkholz et al. (2017, Proposition 3).
Algorithm 1 terminates at the unique coarsest stable partition pair.
This is because that the coloring gin each iteration of Algorithm 1 is always coarser than the unique coarsest stable partition pair (see Berkholz et al. (2017, Proposition 2)).
B S EPARATION P OWER OF THE WL T EST
This section gives the proof and some corollaries of Theorem 4.1.
First we present some definitions and lemmas, from which the proof of Theorem 4.1 can be immediately derived.
Definition B.1.
Given ( G, H ) , ( ˆ G, ˆ H ) ∈ G m,n × H Vm
×
H
Wn , we say that ( G, H ) and ( ˆ G, ˆ H ) can be distinguished by the WL test if there exists some L ∈ N and some choices of hash functions,
HASH 0 ,V , HASH 0 ,W , HASH l,V , HASH l,W , HASH ′
l,V , and HASH ′
l,W , for l = 1 , 2 , . . .
, L , such that the multisets of colors at the L -th iteration of the WL test are different for ( G, H ) and ( ˆ G, ˆ H ) .
Let ∼ be an equivalence relationship on G m,n × H Vm
× H Wn defined via: ( G, H ) ∼ ( ˆ G, ˆ H ) if and only if they can not be distinguished by the WL test.
It is clear that ( G, H ) ∼ ( ˆ G, ˆ H ) if they are isometric, i.e., there exist two permutations, σ V : { 1 , 2 , . . .
, m } → { 1 , 2 , . . .
, m } and σ W : { 1 , 2 , . . . , n } → { 1 , 2 , . .
. , n } , such that E σ V ( i ) ,σ W ( j )
=
ˆ E i,j , h Vσ V
( i ) =
ˆ h
Vi , and h Wσ W ( j ) =
ˆ h Wj , for any i ∈ { 1 , 2 , . . .
, m } and j ∈ { 1 , 2 , . . . , n } .
However, not every pair of WL-indistinguishable graphs consists of isometric ones; see Figure 2 for an example.
However, for LP problems that cannot be distinguished by the WL test will share some common properties, even if their associated graphs are not isomorphic.
Lemma B.2.
If two weighted bipartite graphs with vertex features corresponding to two LP problems are indistinguishable by the WL test, then either both problems are feasible or both are infeasible.
In other words, the WL test can distinguish two LP problems if one of them is feasible while the other one is infeasible.
Proof of Lemma B.2.
Let us consider two LP problems:
[Equation]
and
[Equation]
Let ( G, H ) and ( ˆ G, ˆ H ) , where G =
( V ∪ W, E ) and ˆ G =
( V ∪ W, ˆ E ) , be the weighted bipartite graphs with vertex features corresponding to (B.1) and (B.2), respectively.
Since ( G, H ) ∼ ( ˆ G, ˆ H ) , for some choice of hash functions with no collision during Algorithmfor ( G, H ) and ( ˆ G, ˆ H ) , Theorem A.2 guarantees that Algorithm 1 outputs the same stable coloring for ( G, H ) and ( ˆ G, ˆ H ) up to permutation.
More specifically, after doing some permutation, there exist I
= { I 1 , I 2 , . . .
, I s } and J =
{ J 1 , J 2 , . . .
, J t } that are partitions of { 1 , 2 , . . .
, m } and { 1 , 2 , . . . , n } , respectively, such that the followings hold:
• ( b i , ◦ i ) =
(ˆ b i , ˆ ◦ i ) and is independent of i ∈ I p , for any p ∈ { 1 , 2 , . . .
, s } .
• ( l j , u j ) =
(ˆ l j , ˆ u j ) and is independent of j ∈ J q for any q ∈ { 1 , 2 , . . .
, t } .
• For any p ∈ { 1 , 2 , . . .
, s } and q ∈ { 1 , 2 , . .
. , t } , (cid:80) j ∈ J q A
i,j = (cid:80) j ∈ J q ˆ A i,j and is independent of i ∈ I p .
• For any
p ∈ { 1 , 2 , . . .
, s } and q ∈ { 1 , 2 , . .
. , t } , (cid:80) i ∈ I
p
A i,j = (cid:80) i ∈ I
p
ˆ A
i,j and is independent of j ∈ J q .
Suppose that the problem (B.1) is feasible with x ∈ R n be some point in the feasible region.
Define y ∈ R t via y q = 1 | J q | (cid:80)
j ∈ J q x j and ˆ x ∈ R n via ˆ x j = y q , j ∈ J q .
Fix any p ∈ { 1 , 2 , . . .
, s } and some i
∈ I
p .
It holds for any i ∈ I p that
[Equation]
which implies that
[Equation]
Notice that (cid:80) i ∈ I
p A i,j is constant for j ∈ J q , ∀ q ∈ { 1 , 2 , . . .
, t } .
Let us denote α q = (cid:80) i ∈ I
p A i,j = (cid:80) i ∈ I
p
ˆ A
i,j for any j ∈ J q .
Then it holds that
[Equation]
Note that
[Equation]
and that (cid:80) j ∈ J q ˆ A i,j is constant for i ∈ I p .
So one can conclude that n
[Equation]
which leads to ˆ A ˆ x ˆ ◦ ˆ b .
It can also be seen that ˆ l = l ≤ ˆ x ≤ u ≤ ˆ u .
Therefore, ˆ x is feasible for (B.2).
We have shown above that the feasibility of (B.1) implies the feasibility of (B.2).
The inverse is also true by the same reasoning.
Hence, we complete the proof.
Lemma B.3.
If two weighted bipartite graphs with vertex features corresponding to two LP problems are indistinguishable by the WL test, then these two problems share the same optimal objective value (could be ∞ or −∞ ).
Proof of Lemma B.3.
If both problems are infeasible, then their optimal objective values are both ∞ .
We then consider the case that both problems are feasible.
We use the same setting and notations as in Lemma B.2, and in addition we have that c
j
= ˆ c j , which is part of h Wj =
ˆ h Wj , is independent of j ∈ J q for any q ∈ { 1 , 2 , . . .
, t } .
Suppose that x is an feasible solution to the problem (B.1) and let ˆ x ∈ R n be defined via ˆ x j = 1 | J q | (cid:80) j ′ ∈
J q x j ′ , j ∈ J q .
It is guaranteed by the proof of Lemma B. that ˆ x is a feasible solution to (B.2).
One can also see that
c ⊤ x = ˆ c ⊤ ˆ x .
Since this holds for any feasible solution x to (B.1), the optimal value of the objective function for (B.2) is smaller than or equal to that for (B.1).
The inverse is also true and the proof is completed.
Lemma B.4.
Suppose that two weighted bipartite graphs with vertex features corresponding to two
LP problems are indistinguishable by the WL test and the their optimal objective values are both finite.
Then these two problems have the same optimal solution with the smallest ℓ 2 -norm, up to permutation.
Proof of Lemma B.4.
We work with the same setting as in Lemma B.3, where permutations have already been applied.
Let x and x ′ be the optimal solution to (B.1) and (B.2) with the smallest ℓ 2 -norm, respectively.
(Recall that the optimal solution to a LP problem with the smallest ℓ 2 -norm is unique, see Remark 2.2.)
Let ˆ x ∈ R n be defined via ˆ x j =
[Equation]
q | (cid:80) j ′ ∈
J q x j ′ for j ∈ J q , q = 1 , 2 , . . .
, t .
According to the arguments in the proof of Lemma B.2 and Lemma B.3, ˆ x is an optimal solution to (B.2).
The minimality of x ′ yields that
[Equation]
which implies ∥ x ′ ∥ ≤ ∥ x ∥ .
The converse ∥ x ∥ ≤ ∥ x ′ ∥ is also true.
Therefore, we must have ∥ x ∥ =
∥ x ′ ∥ and hence, the inequalities in (B.3) must hold as equalities.
Then one can conclude that x j = x j ′ for any j, j ′ ∈
J q and q = 1 , 2 , . .
, t , which leads to x = ˆ x .
Furthermore, it follows from ∥ x ′ ∥ =
∥ ˆ x ∥ and the uniqueness of x ′ (see Remark 2.2) that x ′ =
ˆ x = x , which completes the proof.
One corollary one can see from the proof of Lemma B.4 is that the components of the optimal solution with the smallest ℓ 2 -norm must be the same if the two corresponding vertices have the same color in the WL test.
Corollary B.5.
Let ( G, H ) be a weighted bipartite graph with vertex features and let x be the optimal solution to the corresponding LP problem with the smallest ℓ 2 -norm.
Suppose that for some j, j ′ ∈ { 1 , 2 , . . . , n } , one has C l,Wj = C l,Wj ′ for any l ∈ N and any choices of hash functions, then x j = x j ′ .
Let us also define another equivalence relationship on G m,n × H Vm × H Wn where colors of w 1 , w 2 , . . .
, w j with ordering (not just multisets) are considered:
Definition B.6.
Given ( G, H ) , ( ˆ G, ˆ H ) ∈ G m,n × H Vm
×
H
Wn , ( G, H ) and ( ˆ G, ˆ H ) are not in the same equivalence class of W ∼ if and only if there exist some L ∈ N and some hash functions HASH 0 ,V , HASH 0 ,W , HASH l,V , HASH l,W , HASH ′
l,V , and HASH ′
l,W , l = 1 , 2 , . . .
, L , such that {{ C L,V, C L,V, . . .
, C L,Vm }} ̸
= {{ ˆ C L,V, ˆ C L,V, . . .
, ˆ C L,Vm }} or C l,Wj ̸
= ˆ C l,Wj for some j ∈ { 1 , 2 , . . . , n } .
It is clear that ( G, H ) W ∼ ( ˆ G, ˆ H ) implies ( G, H ) ∼ ( ˆ G, ˆ H ) .
One can actually obtain a stronger version of Lemma B.4 given ( G, H ) W ∼ ( ˆ G, ˆ H ) .
Corollary B.7.
Suppose that two weighted bipartite graphs with vertex features corresponding to two LP problems, ( G, H ) and ( ˆ G, ˆ H ) , satisfies that ( G, H ) W ∼ ( ˆ G, ˆ H ) .
Then these two problems have the same optimal solution with the smallest ℓ 2 -norm.
Proof of Corollary B.7.
The proof of Lemma B.4 still applies with the difference that there is no permutation on { w 1 , w 2 , . . .
, w n } .
Proof of Theorem 4.1.
The proof of Theorem 4.1 follows immediately from Lemmas B.2, B.3 and
B.4.
C S EPARATION P OWER OF G RAPH N EURAL N ETWORKS
This section aims to prove Theorem 4.2, i.e., the separation power of GNNs is equivalent to that of the WL test.
Similar results can be found in previous literature, see e.g. Xu et al. (2019); Azizian & Lelarge (2021); Geerts & Reutter (2022).
We first introduce some lemmas that can directly imply
Theorem 4.2.
The lemma below, similar to Xu et al. (2019, Lemma 2), states that the separation power of GNNs is at most that of the WL test.
Lemma C.1.
Let ( G, H ) , ( ˆ G, ˆ H ) ∈ G m,n × H Vm
×
H Wn .
If ( G, H ) ∼ ( ˆ G, ˆ H ) , then for any F W ∈
F W GNN , there exists a permutation σ W ∈ S n such that F W ( G, H ) =
σ W ( F W ( ˆ G, ˆ H )) .
Proof of Lemma C.1.
First we describe the sketch of our proof.
The assumption ( G, H ) ∼ ( ˆ G, ˆ H ) implies that, if we apply the WL test on ( G, H ) and ( ˆ G, ˆ H ) , the test results should be exactly the same whatever the hash functions in the WL test we choose.
In the first step, we define a set of hash functions that are injective on all possible inputs.
Second, we show that, if we apply an arbitrarily chosen GNN: F W ∈ F W GNN on ( G, H ) and ( ˆ G, ˆ H ) , the vertex features of the two graphs are exactly the same up to permutation, given the fact that the WL test results are the same.
Finally, it concludes that F W ( G, H ) should be the same with F W ( ˆ G, ˆ H ) up to permutation.
Let us first define hash functions.
We choose HASH 0 ,V and HASH 0 ,W that are injective on the following sets (not multisets) respectively:
[Equation]
Let { C l − 1 ,V i } mi =1 , { C l − 1 ,W j } nj =
and { ˆ C l − 1 ,V i } mi =1 , { ˆ C l − 1 ,W j } nj =1 be the vertex colors in the ( l − 1) - th iteration (1 ≤ l ≤ L ) in the WL test for ( G, H ) and ( ˆ G, ˆ H ) respectively.
Define two sets (not multisets) that collect different colors:
[Equation]
and
[Equation]
The hash function HASH ′
l,V and HASH ′
l,W are chosen such that the outputs are located in some linear spaces and that { HASH ′ l,V ( C ) : C ∈ C
Vl − 1 } and { HASH ′ l,W ( C ) :
C ∈ C
Wl − 1 } are both linearly independent.
Finally, we choose hash functions HASH l,V and HASH l,W such that HASH l,V is injective on the set (not multiset)
[Equation]
and that HASH l,W is injective on the set (not multiset)
[Equation]
Those hash functions give the vertex colors at the next iteration ( l -th layer): { C l,Vi } mi =1 , { C l,Wj } nj = and { ˆ C l,Vi } mi =1 , { ˆ C l,Wj } nj =1 .
Consider any F W ∈ F W GNN and let { h l − 1 ,V i } mi =1 , { h l − 1 ,W j } nj =
and { ˆ h l − 1 ,V i } mi =1 , { ˆ h l − 1 ,W j } nj =1 be the vertex features in the l -th layer ( 0 ≤ l ≤ L ) of the graph neural network F W .
(Update rule refers to equations (2.1),(2.2),(2.3),(2.5))
We aim to prove by induction that for any l ∈ { 0 , 1 , . . .
, L } , the followings hold:
(i) C l,Vi = C l,Vi ′ implies h l,Vi = h l,Vi ′ , for 1 ≤ i, i ′ ≤ m ; (ii) ˆ C l,Vi = ˆ C l,Vi ′ implies ˆ h l,
Vi =
ˆ h l,Vi ′ , for 1 ≤ i, i ′ ≤ m ; (iii) C l,Vi = ˆ C l,Vi ′ implies h l,Vi
=
ˆ h l,Vi ′ , for 1 ≤ i, i ′ ≤ m ; (iv) C l,Wj = C l,Wj ′ implies h l,Wj = h l,Wj ′ , for 1 ≤ j, j ′ ≤ n ; (v) ˆ C l,W j = ˆ C l,Wj ′ implies ˆ h l,W j = ˆ h l,Wj ′ , for 1 ≤ j, j ′ ≤ n ; (vi) C l,Wj
= ˆ C l,Wj ′ implies h l,Wj
= ˆ h l,Wj ′ , for 1 ≤ j, j ′ ≤ n .
The above claims (i)-(vi) are clearly true for l = 0 due to the injectivity of HASH 0 ,V and HASH 0 ,W .
Now we assume that (i)-(vi) are true for some l − 1 ∈ { 0 , 1 , . . .
, L − 1 } .
Suppose that C l,Vi = C l,Vi ′ , i.e.,
[Equation]
for some 1 ≤ i, i ′ ≤ m .
It follows from the injectivity of HASH l,V that
[Equation]
and
[Equation]
According to the linearly independent property of HASH ′ l,W , the above equation implies that
[Equation]
Note that the induction assumption guarantees that h l − 1 ,W j
[Equation]
as long as C
[Equation]
W
[Equation]
So one can assign for each C ∈ C
Wl − 1 some h ( C ) ∈ R
d l − 1 such that h l − 1 ,W j = h ( C ) as long as
C l − 1 ,W j = C for any 1 ≤ j ≤ n .
Therefore, it follows from (C.2) that
[Equation]
Note also that (C.1) and the induction assumption lead to h l − 1 ,V i =
h l − 1 ,V i ′ .
Then one can conclude that
[Equation]
i , n (cid:88) j
=
E i,j f Wl ( h l − 1 ,W j ) 
 =
g Vl   h l − 1 ,V i ′ , n (cid:88)
j =
E i ′ ,j f Wl ( h l − 1 ,W j ) 
 =
h l,Vi ′ .
This proves the claim (i) for l .
The other five claims can be proved using similar arguments.
Therefore, we obtain from ( G, H ) ∼ ( ˆ G, ˆ H ) that (cid:110)(cid:110) h L,V, h L,V, . . .
, h L,Vm (cid:111)(cid:111) =
(cid:110)(cid:110) ˆ h L,V, ˆ h L,V, . . .
ˆ h L,Vm (cid:111)(cid:111) , and that (cid:110)(cid:110) h L,W, h L,W, . .
, h L,Wn (cid:111)(cid:111) =
(cid:110)(cid:110) ˆ h L,W, ˆ h L,W, . . .
, ˆ h L,Wn (cid:111)(cid:111) .
By the definition of the output layer, the above conclusion guarantees that F W ( G, H ) =
σ W ( F W ( ˆ G, ˆ H )) for some σ W ∈ S n .
Lemma C.2.
Let ( G, H ) , ( ˆ G, ˆ H ) ∈ G m,n × H Vm × H Wn .
Suppose that for any F W ∈ F W GNN , there exists a permutation σ W ∈ S n such that F W ( G, H ) =
σ W ( F W ( ˆ G, ˆ H )) .
Then F ( G, H ) =
F ( ˆ G, ˆ H ) holds for any F ∈ F GNN .
Proof of Lemma C.2.
Pick an arbitrary F ∈ F GNN .
We choose F W ∈ F W GNN such that
[Equation]
Note that every entry in the output of F W is equal to the output of F .
Thus, it follows from
F W ( G, H ) =
σ W ( F W ( ˆ G, ˆ H ))
that F ( G, H ) =
F ( ˆ G, ˆ H ) .
The next lemma is similar to Xu et al. (2019, Theorem 3) and states that the separation power of
GNNs is at least that of the WL test.
Lemma C.3.
Let ( G, H ) , ( ˆ G, ˆ H ) ∈ G m,n × H Vm × H Wn .
If F ( G, H )
=
F ( ˆ G, ˆ H ) holds for any
F ∈ F GNN , then ( G, H ) ∼ ( ˆ G, ˆ H ) .
Proof of Lemma C.3.
It suffices to prove that, if ( G, H ) can be distinguished from ( ˆ G, ˆ H ) by the
WL test, then there exists F ∈ F GNN , such that F ( G, H )
̸ = F ( ˆ G, ˆ H ) .
The distinguish-ability of the WL test implies that there exists L ∈ N and hash functions, HASH 0 ,V , HASH 0 ,W , HASH l,V ,
HASH l,W , HASH ′ l,V , and HASH ′ l,W , for l = 1 , 2 , . . .
, L , such that
[Equation]
or (cid:110)(cid:110) C L
[Equation]
[Equation] [Equation]
We aim to construct some GNNs such that the followings hold for any l = 0 , 1 , . . .
, L :
(i) h l,Vi = h l,Vi ′ implies C l,Vi = C l,Vi ′ , for 1 ≤ i, i ′ ≤ m ; (ii) ˆ h l,
Vi =
ˆ h l,Vi ′ implies ˆ C l,Vi = ˆ C l,Vi ′ , for 1 ≤ i, i ′ ≤ m ; (iii) h l,Vi
= ˆ h l,Vi ′ implies C l,Vi = ˆ C l,Vi ′ , for 1 ≤ i, i ′ ≤ m ; (iv) h l,W j = h l,W j ′ implies C l,Wj = C l,Wj ′ , for 1 ≤ j, j ′ ≤ n ; (v) ˆ h l,Wj
= ˆ h l,Wj ′ implies ˆ C l,Wj
= ˆ C l,Wj ′ , for 1 ≤ j, j ′ ≤ n ; (vi) h l,Wj
= ˆ h l,Wj ′ implies C l,Wj = ˆ C l,Wj ′ , for 1 ≤ j, j ′ ≤ n .
It is clear that the above conditions (i)-(vi) hold for l = 0 as long as we choose f V in and f W in that are injective on the following two sets (not multisets) respectively:
[Equation]
We then assume that (i)-(vi) hold for some 0 ≤ l − 1 < L , and show that these conditions are also satisfied for l if we choose f Vl , f Wl , g Vi , g Wl properly.
Let us consider the set (not multiset): { α 1 , α 2 , . .
, α s } ⊂ R d l −that collects all different values in h l − 1 ,W, h l − 1 ,W, . .
, h l −
[Equation]
W n , ˆ h l −
[Equation]
W, ˆ h l −
[Equation]
W
[Equation]
, ˆ h l − 1 ,W n .
Let d l ≥ s and let e d l p = (0 , . .
. , 0) be the vector in R d l with the p -th entry being 1 and all other entries being 0 , for 1 ≤ p ≤ s .
Choose f
Wl : R d l − 1 → R d l as a continuous function satisfying f Wl ( α p ) =
e d l p , p = 1 , 2 , . . .
, s , and choose g Vl :
R d l − 1 × R d l → R d l that is continuous and is injective when restricted on the set (not multiset)     h l − 1 ,V i , n (cid:88)
j =
E i,j f Wl ( h l − 1 ,W j ) 
 : 1 ≤ i ≤ m   ∪     ˆ h l − 1 ,V i , n (cid:88)
j = ˆ E
i,j f Wl (ˆ h l − 1 ,W j ) 
 : 1 ≤ i ≤ m   .
Noticing that n (cid:88) j =
E i,j f Wl ( h l − 1 ,W j ) =
s (cid:88) p =   (cid:88)
h l − 1 ,W j = α p
E i,j  
e d l p , and that { e d l 1 , e d l 2 , . .
, e d l s } is linearly independent, one can conclude that h l,V i =
h l,V i ′ if and only if h l − 1 ,V i =
h l − 1 ,V i ′ and (cid:80) nj =1 E
i,j f Wl ( h l − 1 ,W j ) =
(cid:80) nj =1 E i ′ ,j f Wl ( h l − 1 ,W j ) , where the second condition is equivalent to (cid:88)
h l − 1 ,W j = α p
E i,j = (cid:88)
h l − 1 ,W j = α p
E i ′ ,j , ∀ p ∈ { 1 , 2 , . . .
, s } .
This, as well as the condition (iv) for l − 1 , implies that
[Equation]
and hence that C l,Vi = C l,Vi ′ by using h l − 1 ,V i =
h l − 1 ,V i and condition (i) for l − 1 .
Therefore, we know that (i) is satisfied for l , and one can show (ii) and (iii) for l using similar arguments by taking d l large enough.
In addition, f Vl and g Wl can also be chosen in a similar way such that (iv)-(vi) are satisfied for l .
Combining (C.3), (C.4), and condition (i)-(iv) for L , we obtain that
[Equation]
or
[Equation]
Without loss of generality, we can assume that (C.5) holds.
Consider the set (not multiset)
[Equation]
that collects all different values in h L,V,
[Equation] [Equation]
ˆ h L
[Equation]
, ˆ h L,Vm .
Let k > 1 be a positive integer that is greater than the maximal multiplicity of an element in the multisets {{ h L,V, h L,V, . .
, h L,Vm }} and {{ ˆ h L,V, ˆ h L,V, . . .
, ˆ h L,Vm }} .
There exists a continuous function φ : R d L → R such that φ ( β q ) =
k q for q = 1 , 2 , . . .
, t , and due to (C.5) and the fact that the way of writing an integer as k -ary expression is unique, it hence holds that
[Equation]
Set the dimension of ( L + 1) -th layer as 1 : d L +1 = 1 , and set f L +1 ,V = 0 , f L +1 ,W = 0 , g L +1 ,V ( h, 0)
=
φ ( h ) , and g L +1 ,W = 0 .
Then we have h L +1 ,V i =
φ ( h L,Vi ) ,
ˆ h L +1 ,V i =
φ (ˆ h L,
[Equation]
i ) , and h L +1 ,W j = ˆ h L
+1 ,W j for i = 1 , 2 , . . .
, m and j = 1 , 2 , · · · , n .
Define f out : R × R
→
R via f out ( h, h ′ )
=
h .
Then it follows that
[Equation]
i =
φ (ˆ h L,V i ) =
f out   m (cid:88) i
=
ˆ h L +1 ,V i , n (cid:88)
j = ˆ h L +1 ,W j   , which guarantees the existence of F ∈ F GNN that has L + 1 layers and satisfies F ( G, H ) ̸
=
F ( ˆ G, ˆ H ) .
Proof of Theorem 4.2.
The equivalence of the three conditions follow immediately from
Lemma C.1, C.2, and C.3.
Corollary C.4.
For any two weighted bipartite graphs with vertex features ( G, H ) , ( ˆ G, ˆ H ) ∈
G m,n × H Vm
× H Wn , the followings are equivalent:
(i) ( G, H ) W ∼ ( ˆ G, ˆ H ) .
(ii) For any F W ∈ F W GNN , it holds that F W ( G, H ) =
F W ( ˆ G, ˆ H ) .
Proof of Corollary C.4.
The proof follows similar lines as in the proof of Theorem 4.2 with the difference that there is no permutation on { w 1 , w 2 , . . . , w n } .
In addition to the separation power of GNNs for two weighted bipartite graphs with vertex features, one can also obtain results on separating different vertices in one weighted bipartite graph with vertex features.
Corollary C.5.
For any weighted bipartite graph with vertex features ( G, H ) and any j, j ′ ∈ { 1 , 2 , . . . , n } , the followings are equivalent:
(i) C l,Wj = C l,Wj ′ holds for any l ∈ N and any choice of hash functions.
(ii) F W ( G, H ) j = F W ( G, H ) j ′ , ∀ F W ∈ F W GNN .
Proof of Corollary C.5.
“(i) =
⇒ (ii)” and “(ii) =
⇒ (i)” can be proved using similar arguments in the proof of Lemma C.1 and Lemma C.3, respectively.
D U NIVERSAL A PPROXIMATION OF F GNN
This section provides the proof of Theorem 4.3.
The main mathematical tool used in the proof is the
Stone–Weierstrass theorem:
Theorem D.1 (Stone–Weierstrass theorem (Rudin, 1991, Section 5.7)) .
Let X be a compact Hausdorff space and let F ⊂ C ( X, R ) be a subalgebra.
If F separates points on X , i.e., for any x, x ′ ∈ X with x ̸ = x ′ , there exists F ∈ F such that F ( x ) ̸ = F ( x ′ ) , and 1 ∈ F , then F is dense in C ( X, R ) with the topology of uniform convergence.
Proof of Theorem 4.3.
Let π : X → X/ ∼ be the quotient map, where π ( X ) =
X/ ∼ is equipped with the quotient topology.
For any F ∈ F GNN , since F : X → R is continuous and by Theorem 4.2,
F ( G, H )
=
F ( ˆ G, ˆ H ) , ∀ ( G, H ) ∼ ( ˆ G, ˆ H ) , there exists a unique continuous ˜ F : π ( X ) → R such that F = ˜
F ◦ π .
Set
[Equation]
In addition, the assumption on Φ , i.e,
[Equation]
, leads to the existence of a unique ˜Φ ∈ C ( π ( X ) , R ) with Φ = ˜Φ ◦ π .
Since X is compact, then π ( X ) is also compact due to the continuity of π .
According to Lemma D. below, ˜ F GNN is a subalgebra of C ( π ( X ) , R ) .
By Theorem 4.2, ˜ F GNN separates points on π ( X ) .
This can further imply that π ( X ) is Hausdorff.
In fact, for any x, x ′ ∈ π ( X ) , there exists ˜ F ∈ ˜ F GNN with ˜ F ( x ) ̸
= ˜ F ( x ′ ) .
Without loss of generality, we assume that ˜ F ( x ) < ˜ F ( x ′ ) and choose some c ∈ R with ˜ F ( x ) < c < ˜ F ( x ′ ) .
By continuity of ˜ F , we know that ˜ F − 1 (( −∞ , c ))
∩ π ( X ) and ˜ F − 1 (( c, + ∞ ))
∩ π ( X ) are disjoint open subsets of π ( X ) with x ∈ ˜ F − 1 (( −∞ , c ))
∩ π ( X ) and x ′ ∈ ˜ F − 1 (( c, + ∞ ))
∩ π ( X ) , which leads to the Hausdorff property of π ( X ) .
Note also that 1 ∈ ˜ F GNN .
Using Theorem D.1, we can conclude the denseness of ˜ F GNN in C ( π ( X ) , R ) .
Therefore, for any ϵ > 0 , there exists F ∈ F GNN , such that
[Equation]
which completes the proof.
Lemma D.2.
F GNN is a subalgebra of C ( G m,n × H Vm
× H Wn , R ) , and as a corollary, ˜ F GNN is a subalgebra of C ( G m,n × H Vm
× H Wn / ∼ , R ) .
Proof of Lemma D.2.
It suffices to show that F GNN is closed under addition and multiplication.
Consider any F, ˆ F ∈ F GNN .
Thanks to Lemma D.3, we can assume that both F and ˆ F have L layers.
Suppose that F is constructed by
[Equation]
and that ˆ F is constructed by
[Equation]
One can then construct two new GNNs computing F + ˆ F and F · ˆ F as follows:
The input layer The update rule of the input layer is defined by
[Equation]
Then the vertex features after the computation of the input layer h 0 ,Vi and h 0 ,Wj are given by: h 0 ,
[Equation]
for i = 1 , 2 . . .
, m , and j = 1 , 2 , . . . , n .
The l -th layer ( 1 ≤ l ≤ L ) .
We set
[Equation]
and
[Equation]
Then the vertex features after the computation of the l -th layer h l,Vi and h l,Wj are given by:
[Equation]
l
[Equation]
and h l,W
[Equation]
= (cid:32)
[Equation]
(cid:32) h l − 1 ,W j , m (cid:88) i =
E i,j f Vl ( h l − 1 ,V i ) (cid:33) , ˆ g Wl (cid:32)
ˆ h l − 1 ,W j , m (cid:88) i =
E i,j ˆ f Vl (ˆ h l − 1 ,V i ) (cid:33)(cid:33)
=
( h l,
[Equation]
j
[Equation]
l
[Equation]
for i = 1 , 2 . . .
, m , and j = 1 , 2 , . . . , n .
The output layer To obtain F + ˆ F , we set
[Equation]
Then it holds that f addout  
[Equation]
i =
[Equation]
To obtain F · ˆ F , we set f
[Equation]
[Equation] [Equation]
· ˆ f out (ˆ h, ˆ h ′ ) .
Then it holds that f multiplyout   m (cid:88) i =
h L,Vi , n (cid:88)
j = h L,W
 = f out   m (cid:88)
i =
h L,Vi , n (cid:88)
j = h L,Wj   · ˆ f out   m (cid:88) i
= ˆ h L,Vi , n (cid:88)
j = ˆ h L,Wj 
 =
F ( G, H ) · ˆ F ( G, H ) .
The constructed GNNs satisfy F ( G, H ) + ˆ F ( G, H ) ∈ F GNN and F ( G, H ) · ˆ F ( G, H ) ∈ F GNN , which finishes the proof.
Lemma D.3.
If F ∈ F GNN has L layers, then there exists ˆ F ∈ F GNN with L + 1 layers such that
F = ˆ F .
Proof of Lemma D.3.
Suppose that F is constructed by f V in , f W in , f out , { f Vl , f Wl , g Vl , g Wl } Ll =0 .
We choose f VL +1 ≡ 0 , f WL +1 ≡ 0 , g VL +1 ( h, h ′ )
=
h , g WL +
( h, h ′ )
=
h .
Let ˆ F be constructed by f V in , f W in , f out , { f V l , f W l , g V l , g W l } L +1 l =0 .
Then ˆ F has L + 1 layers with ˆ F = F .
E U NIVERSAL A PPROXIMATION OF F W GNN
This section provides an universal approximation result of F W GNN .
Theorem E.1.
Let X ⊂ G m,n × H Vm
×
H
Wn be a compact subset that is closed under the action of
S m × S n .
Suppose that Φ ∈ C ( X, R n ) satisfies the followings
: (i) For any σ V ∈ S m , σ W ∈ S n , and ( G, H )
∈ X , it holds that
[Equation]
(ii) Φ( G, H ) =
Φ( ˆ G, ˆ H ) holds for all ( G, H ) ,
( ˆ G, ˆ H ) ∈
X with ( G, H ) W ∼ ( ˆ G, ˆ H ) .
(iii)
Given any ( G, H ) ∈ X and any j, j ′ ∈ { 1 , 2 , . . . , n } , if C l,Wj
[Equation]
(the vertex colors obtained in the l -th iteration in WL test) holds for any l ∈ N and any choices of hash functions, then Φ( G, H )
j = Φ( G, H ) j ′ .
Then for any ϵ > 0 , there exists F ∈ F W GNN such that
[Equation]
Theorem E.1 is a LP-graph version of results on the closure of equivariant GNN class in Azizian &
Lelarge (2021); Geerts & Reutter (2022).
The main tool in the proof of Theorem E.1 is the following generalized Stone-Weierstrass theorem for equivariant functions established in Azizian & Lelarge (2021).
Theorem E.2 (Generalized Stone-Weierstrass theorem (Azizian & Lelarge, 2021, Theorem 22)) .
Let X be a compact topology space and let G be a finite group that acts continuously on X and R n .
Define the collection of all equivariant continuous functions from X to R n as follows:
[Equation]
Consider any F ⊂ C E ( X, R n ) and any Φ ∈ C E ( X, R n ) .
Suppose the following conditions hold:
(i) F is a subalgebra of C ( X, R n ) and 1 ∈ F .
(ii)
For any x, x ′ ∈ X , if f ( x ) =
f
( x ′ ) holds for any f ∈ C ( X, R ) with f 1 ∈ F , then for any F ∈ F , there exists g ∈ G such that F ( x )
=
g ∗ F ( x ′ ) .
(iii)
For any x, x ′ ∈ X , if F ( x ) = F ( x ′ ) holds for any F ∈ F , then Φ( x ) =
Φ( x ′ ) .
(iv) For any x ∈ X , it holds that Φ( x ) j = Φ( x ) j ′ , ∀ ( j, j ′ )
∈ J ( x ) , where J ( x ) = {{ 1 , 2 , . . . , n } n : F ( x ) j = F ( x ) j ′ , ∀ F ∈ F
Then for any ϵ > 0 , there exists F ∈ F such that
[Equation]
We refer to Timofte (2005) for different versions of Stone-Weierstrass theorem, that is also used in Azizian & Lelarge (2021).
In the proof of Theorem E.1, we also need the following lemma whose proof is almost the same as the proof of Lemma D.2 and is hence omitted.
Lemma E.3.
F W GNN is a subalgebra of C ( G m,n × H Vm
× H Wn , R n ) .
Proof of Theorem E.1.
Let S m × S n act on R n via
[Equation]
Then it follows from (E.1) that Φ is equivariant.
In addition, the definition of graph neural networks directly guarantees the equivariance of functions in F W GNN .
Therefore, one only needs to verify the conditions with F as F W GNN and G as S n in Theorem E.2:
• Condition (i) in Theorem E.2 follows from Lemma E.3 and the definition of F W GNN .
• Condition (ii) in Theorem E.2 follows from Theorem 4.2 and F GNN 1 ⊂ F W GNN .
• Condition (iii) in Theorem E.2 follows from Corollary C.4 and Condition (ii) in Theorem E.1.
• Condition (iv) in Theorem E.2 follows from Corollary C.5 and Condition (iii) in Theorem E.1.
It finishes the proof.
F P ROOF OF M AIN T HEOREMS
We collect the proofs of main theorems stated in Section 3 in this section.
Proof of Theorem 3.1.
If F ( G, H )
=
F ( ˆ G, ˆ H ) holds for any F ∈ F GNN , then Theorem 4.2 guarantees that ( G, H ) ∼ ( ˆ G,
ˆ H ) .
Thus, Condition (i), (ii), and (iii) follow directly from Lemmas B.2,
B.3, and B.4, respectively.
Furthermore, if F W ( G, H ) = F W ( ˆ G, ˆ H ) , ∀ F W ∈ F W GNN , then it follows from Corollary C.4 and
Corollary B.7 that the two LP problems associated to ( G, H ) and ( ˆ G, ˆ H ) share the same optimal solution with the smallest ℓ 2 -norm.
Then we head into the proof of three main approximation theorems, say Theorem 3.2, 3.4, and 3.6, that state that GNNs can approximate the feasibility mapping Φ feas , the optimal objective value mapping Φ obj , and the optimal solution mapping Φ solu with arbitrarily small error, respectively.
We have established in Sections D and E several theorems for graph neural networks to approximate continuous mappings.
Therefore, the proof of Theorem 3.2, 3.4, and 3.6 basically consists of two steps:
(i) Show that the mappings Φ feas , Φ obj , and Φ solu are measurable.
(ii) Use continuous mappings to approximate the target measurable mappings, and then apply the universal approximation results established in Sections D and E.
Let us first prove the feasibility of Φ feas , Φ obj , and Φ solu in the following three lemmas.
Lemma F.1.
The feasibility mapping Φ feas defined in (2.7) is measurable, i.e., the preimages Φ − 1 feas (0) and Φ − 1 feas (1) are both measurable subsets of G m,n
× H Vm
× H Wn .
Proof of Lemma F.1.
It suffices to prove that for any ◦ ∈ {≤ , = ≥} m and any N l , N u ⊂ { 1 , 2 , . . . , n } , the set
X feas :
[Equation]
A, b, l, u )
[Equation]
: ∃ x ∈ R n , s.t.
Ax ◦ b, x
[Equation]
≥ l
[Equation]
, ∀ j ∈ N l , x j ≤ u j , ∀ j ∈ N u } , is a measurable subset in R m × n × R m × R | N l | × R | N u | .
Without loss of generality, we assume that ◦ =
( ≤ , . .
. , ≤ , = , . .
, = , ≥ , . .
. , ≥ ) where “ ≤ ”, “ = ”, and “ ≥ ” appear for k 1 , k 2 − k 1 , and m − k 1 − k 2 times, respectively, 0 ≤ k 1 ≤ k 2 ≤ m .
Let us define a function V feas : R m × n × R m × R | N l | × R | N u | × R n → R ≥ 0 that measures to what extend a point in R n violates the constraints:
[Equation]
where y + = max { y, 0 } .
It can be seen that V feas is continuous and V feas ( A, b, l, u, x )
=
if and only if Ax ◦ b , x j ≥ l j , ∀ j ∈ N l , and x j ≤ u j , ∀ j ∈ N u .
Therefore, for any ( A, b, l, u ) ∈
R m × n × R m × R | N l | × R | N u | , the followings are equivalent:
• ( A, b, l, u ) ∈
X feas .
• There exists R ∈ N + and x ∈ B R :=
{ x ′ ∈ R n : ∥ x ′ ∥ ≤ R } , such that V feas ( A, b, l, u, x ) = 0 .
• There exists R ∈ N + such that for any r ∈ N + , V feas ( A, b, l, u, x ) ≤ 1 /r holds for some x ∈ B R ∩ Q n .
This implies that X feas can be described via
[Equation]
Since Q n is countable and V is continuous, we immediately obtain from the above expression that
X feas is measurable.
Lemma F.2.
The optimal objective value mapping Φ obj defined in (2.8) is measurable.
Proof of Lemma F.2.
It suffices to prove that for any ◦ ∈ {≤ , = ≥} m , any N l , N u ⊂ { 1 , 2 , . . . , n } , and any ϕ ∈ R , the set
[Equation]
is a measurable subset in R m × n × R m × R n × R | N l | × R | N u | .
Then the proof follows the same lines as in the proof of Lemma F.1, with a different violation function
[Equation]
Lemma F.3.
The optimal solution mapping Φ solu defined in (2.9) is measurable.
Proof of Lemma F.3.
It suffices to show that for every j 0 ∈ { 1 , 2 , . . . , n } , the mapping
[Equation]
, is measurable, where π j 0 : R n → R maps a vector x ∈ R n to its j 0 -th component.
Similar as before, one can consider any ◦ ∈ {≤ , = ≥} m , any N l , N u ⊂ { 1 , 2 , . .
. , n } , and any ϕ ∈ R , and prove that the set
[Equation]
R | N u | :
The LP problem, min x ∈ R d c ⊤ x, s.t.
Ax ◦ b, x j ≥ l j ,
[Equation]
∈ N l , x j ≤ u j , ∀ j ∈ N u , has a finite optimal objective value, and its optimal solution with the smallest ℓ 2 − norm , x opt , satisfies ( x opt ) j 0 < ϕ } , is measurable.
Note that we have fixed ◦ ∈ {≤ , = ≥} m and N l , N u ⊂ { 1 , 2 , . . . , n } .
Let ι
[Equation]
m,n
[Equation]
be the embedding map.
Define another violation function
[Equation]
via
[Equation]
which is measurable with respect to ( A, b, c, l, u ) for any fixed x ∈ R n , due to the measurability of Φ obj and the continuity of V feas .
Moreover, V solu is continuous with respect to x .
Therefore, the followings are equivalent for ( A, b, c, l, u ) ∈ (Φ obj ◦ ι ) − 1 ( R ) :
• ( A, b, c, l, u ) ∈
X solu .
• There exists x ∈ R n with x j 0 < ϕ , such that V solu ( A, b, c, l, u, x ) =
and V solu ( A, b, c, l, u, x ′ ) > 0 , ∀ x ′ ∈ B ∥ x ∥ , x ′ j 0 ≥ ϕ .
• There exists R ∈ Q + , r ∈ N + , and x ∈ B R with x j 0 ≤ ϕ − 1 /r ,
such that V solu
A, b, c, l, u, x ) =and V solu ( A, b, c, l, u, x ′ ) > 0 , ∀ x ′
∈ B R , x ′ j 0 ≥ ϕ .
• There exists R ∈ Q + and r ∈ N + , such that for all r ′ ∈ N
+ , ∃ x ∈ B R ∩ Q n , x j 0 ≤ ϕ − 1 /r , s.t.
V solu ( A, b, c, l, u, x ) < 1 /r ′
and that ∃ r ′′ ∈ N + , s.t., V solu ( A, b, c, l, u, x ′ ) ≥ 1 /r ′′ , ∀ x ′ ∈ B R ∩ Q n , x ′ j 0 ≥ ϕ .
Therefore, one has that
[Equation]
which is measurable.
With the measurability of Φ feas
[Equation]
Φ obj
[Equation]
and Φ solu established
[Equation]
the next step is to approximate Φ feas , Φ obj , and Φ solu using continuous mappings
[Equation]
and hence graph neural networks.
Before proceeding, let us mention that G m,n × H Vm
× H Wn is essentially the disjoint union of finitely many product spaces of Euclidean spaces and discrete spaces that have finitely many points and are equipped with discrete measures.
More specifically,
[Equation]
Therefore, many results in real analysis for Euclidean spaces still apply for G m,n × H Vm × H Wn and
Meas ( · ) , including the following Lusin’s theorem.
Theorem F.4 (Lusin’s theorem (Evans & Garzepy, 2018, Theorem 1.14)) .
Let µ be a Borel regular measure on R n and let f : R n → R m be µ -measurable.
Then for any µ -measurable X ⊂ R n with µ ( X ) < ∞ and any ϵ > 0 , there exists a compact set E ⊂ X with µ ( X \ E ) <
ϵ , such that f | E is continuous.
Proof of Theorem 3.2.
Since X ⊂ G m,n ×H Vm ×H Wn is measurable with finite measure, according to
Lusin’s theorem, there is a compact set E ⊂ X such that Φ feas | E is continuous with Meas ( X \ E ) <
ϵ .
By Lemma B.2 and Theorem 4.3, there exists F ∈ F GNN such that
[Equation]
which implies that
[Equation]
Therefore, one obtains
[Equation]
and the proof is completed.
Proof of Corollary 3.3.
As a finite set, D is compact and Φ feas | D is continuous.
The rest of the proof is similar to that of Theorem 3.2, using Lemma B.2 and Theorem 4.3.
Proof of Theorem 3.4.
(i)
The proof follows the same lines as in the proof of Theorem 3.2, with the difference that we approximate
[Equation]
, instead of Φ feas
and we use Lemma B.3 instead of Lemma B.2.
(ii)
The proof is still similar to that of Theorem 3.2.
By Lusin’s theorem, there is a compact set
E ⊂ X ∩ Φ − 1 obj ( R ) such that Φ obj | E is continuous with Meas (cid:16) ( X ∩ Φ − 1 obj ( R ))
\ E (cid:17) <
ϵ .
Using
Lemma B.3 and Theorem 4.3, there exists F 2 ∈ F GNN with sup
[Equation]
G,
[Equation]
E
[Equation]
which implies that
[Equation]
Proof of Corollary 3.5.
The results can be proved using similar techniques as in Theorem 3.2 and
Theorem 3.4 by noticing that any finite dataset is compact on which any real-valued function is continuous.
Proof of Theorem 3.6.
Without loss of generality, we can assume that X ⊂ Φ − 1 obj ( R ) ⊂
G m,n × H Vm
× H Wn is closed under the action of S m × S n ; otherwise, we can replace X by (cid:83)
[Equation]
σ W ) ∈ S m × S n ( σ V , σ W ) ∗ X .
By Lusin’s theorem, there exists a compact subset E ′ ⊂
X such that Meas ( A \ X ) < ϵ/ | S m × S n | and that Φ solu |
E ′ is continuous.
Define another compact set:
E =
[Equation]
which is closed under the action of S m × S n and satisfies
[Equation]
Note that the three conditions in Theorem E.1 are satisfied by the definition of Φ solu , Corollary B.7, and Corollary B.6, respectively.
Using Theorem E.1, there exists F W ∈ F W GNN such that
[Equation]
Therefore, it holds that
[Equation]
which completes the proof.
Proof of Corollary 3.7.
One can assume that D is closed under the action of S m × S n ; otherwise, a larger but still finite dataset, (cid:83) ( σ V ,σ W ) ∈
S m × S n ( σ V , σ W ) ∗ D , can be considered instead of D .
The rest of the proof is similar to that of Theorem 3.6 since D is compact and Φ solu | D is continuous.
G D ETAILS OF THE N UMERICAL E XPERIMENTS AND E XTRA E XPERIMENTS
LP instance generation We generate each LP with the following way.
We set m = 10 and n = 50 .
Each matrix A is sparse with 100 nonzero elements whose positions are sampled uniformly and values are sampled normally.
Each element in b, c are sampled i.i.d and uniformly from [ − 1 , 1] .
Additionally, each element in c is scaled by 0 .
The variable bounds l, u are sampled with
N (0 , 10) .
If l j > u j , then we swap l j and u j for all 1 ≤ j ≤ n .
Furthermore, we sample ◦ i i.i.d with P ( ◦ i = “ ≤ ”) = 0 .
7 and P ( ◦ i = “ = ”)
= 0 .
With the generation approach above, the probability that each LP to be feasible is around 0 .
MLP architectures As we mentions in the main text, all the learnable functions in GNN are taken as MLPs.
The input functions f V in , f W in have one hidden layer and other functions f out , f W out , { f Vl , f Wl , g Vl , g Wl } Ll =0 have two hidden layers.
The embedding size d 0 , · · · , d L are uniformly taken as d that is chosen from { 2 , 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 } .
All the activation functions are ReLU.
Training settings We use Adam (Kingma & Ba, 2014) as our training optimizer with learning rate of 0 . 0003 .
The loss function is taken as mean squared error.
All the experiments are conducted on a
Linux server with an Intel Xeon Platinum 8163 GPU and eight NVIDIA Tesla V100 GPUs.
Extra experiments on generalization We generate the testing set consisting of 1000 LP problems using the same distribution as that of the training set.
The performance of the trained GNNs on training set and testing set is presented in Table 1, 2, and 3 for feasibility, optimal objective value, and optimal solution, respectively.
The metric in Table 1 is the rate of classification errors; the metric in Table 2 is a relative error defined as | F − Φ obj | / ( | Φ obj | + 1) ; the metric in Table 3 is defined with ∥ F W − Φ solu ∥ / ( ∥ Φ solu ∥ + 1) .
[Table]
[Table] [Table]
One can observe that, for a GNN with fixed size, its generalization performance,
i
[Table]
e., the performance on the testing set is increasing if it is trained with more training samples.
Given these numerical results, we believe that understanding the generalization quantitatively and theoretically deserves future research
[Table]