C OMBINING P HYSICS AND M ACHINE L EARNING FOR N ETWORK F LOW E STIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The ﬂow estimation problem consists of predicting missing edge ﬂows in a network (e.g., trafﬁc, power and water) based on partial observations.
These missing ﬂows depend both on the underlying physics (edge features and a ﬂow conservation law) as well as the observed edge ﬂows.
This paper introduces an optimization framework for computing missing edge ﬂows and solves the problem using bilevel optimization and deep learning.
More speciﬁcally, we learn regularizers that depend on edge features (e g., number of lanes in a road, resistance of a power line) using neural networks.
Empirical results show that our method accurately predicts missing ﬂows, outperforming the best baseline by up to 20% and is able to capture relevant physical properties in trafﬁc and power networks
I NTRODUCTION
In many applications, ranging from road trafﬁc to supply chains to power networks, the dynamics of ﬂows on edges of a graph is governed by physical laws/models (Bressan et al., 2014; Garavello & Piccoli, 2006).
For instance, the LWR model describes equilibrium equations for road trafﬁc
Lighthill & Whitham (1955); Richards (1956).
However, it is often difﬁcult to fully observe ﬂows in these applications and, as a result, they rely on off-the-shelf machine learning models to make predictions about missing ﬂows (Li et al., 2017; Yu et al., 2018).
A key limitation of these machine learning models is that they disregard the physics governing the ﬂows.
So, the question arises: can we combine physics and machine learning to make better ﬂow predictions?
This paper investigates the problem of predicting missing edge ﬂows based on partial observations and the underlying domain-speciﬁc physics deﬁned by ﬂow conservation and edge features (Jia et al., 2019).
Edge ﬂows depend on the graph topology due to a ﬂow conservation law—i.e. the total inﬂow at every vertex is approximately its total out-ﬂow.
Moreover, the ﬂow at an edge also depends on its features, which might regularize the space of possible ﬂow distributions in the graph.
Here, we propose a model that learns how to predict missing ﬂows from data using bilevel optimization (Franceschi et al., 2017) and neural networks.
More speciﬁcally, features are given as inputs to a neural network that produces edge ﬂow regularizers.
Weights of the network are then optimized via reverse-mode differentiation based on a ﬂow estimation loss from multiple train-validation pairs.
Our work falls under a broader effort towards incorporating physics knowledge to machine learning, which is relevant for natural sciences and engineering applications where data availability is limited (Rackauckas et al., 2020).
Conservation laws (of energy, mass, momentum, charge etc.) are essential to our understanding of the physical world.
The classical Noether’s theorem shows that such laws arise from symmetries in nature (Hanc et al., 2004).
However, ﬂow estimation, which is an inverse problem (Tarantola, 2005; Arridge et al., 2019), is ill-posed under conservation alone.
Regularization enables us to apply domain-knowledge in the solution of inverse problems.
We motivate the ﬂow estimation problem and evaluate our solutions using two application scenarios.
The ﬁrst is road trafﬁc networks (Coclite et al., 2005), where vertices represent locations, edges are road segments, ﬂows are counts of vehicles that traverse a segment and features include numbers of lanes and speed limits.
The second scenario is electric power networks (D¨orﬂer et al., 2018), where vertices represent power buses, edges are power lines, ﬂows are amounts of power transmitted and edge features are resistances and lengths of lines.
Other ﬂow graphs include irrigation channels, gas pipelines, blood circulation, supply chains, air trafﬁc, and telecommunication networks (Bressan et al., 2014).
In each of these scenarios, estimating missing ﬂows is a key problem.
O b s e r v e d
M i ss i n g
V a l i d a t i o n
P r e d i c t e d ﬂ o w F l o w g r a p h
Q 1,
Q 2,
Q 3,
Q 4,
Q 5,
Q 6,
Q 7,
Q 8,
E
d g e
R
e
g u l a r
i z e r Q e , e fold
K - f o l d c r o s s v a l i d a t i o n F l
o w e
s t
i m a t i o n ( c o n s e r v a t i o n + r e g u l a r
i z a t i o n ) f 2 ~ f 3 ~ f 5 ~ f 8 ~ x 1* x 4* x 2* x 3*
F l o w e s t
i m a t i o n l o s s E
d g e
r
e
g
u l a r
i z e r s
E d g e f e a t u r e s [ e ]
N e u r a l n e t w o r k w i t
h p a r a m e t e r s
Sec. 3.
Sec. 3.1, 3.
Sec. 3.
Our contributions can be summarized as follows: (1) We introduce a missing ﬂow estimation problem with applications in a broad class of ﬂow graphs; (2) we propose a model for ﬂow estimation that is able to learn the physics of ﬂows by combining reverse-mode differentiation and neural networks; (3) we show that our model outperforms the best baseline by up to 20%; and (4) we provide evidence that our model learns interpretable physical properties, such as the role played by resistance in a power transmission network and by the number of lanes in a road trafﬁc network.
F LOW ESTIMATION P ROBLEM
We introduce the ﬂow estimation problem, which consists of inferring missing ﬂows in a network based on a ﬂow conservation law and edge features.
We provide a list of symbols in the Appendix.
Flow Graph.
Let G ( V , E , X ) be a ﬂow graph with vertices V ( n = |V| ), edges E ( m = |E| ), and edge feature matrix X ∈ R m × d , where X [ e ] are the features of edge e .
A ﬂow vector f ∈ R m contains the (possibly noisy) ﬂow f e for each edge e ∈ E .
In case G is directed, f ∈ R m + , otherwise, a ﬂow is negative if it goes against the arbitrary orientation of its edge.
We assume that ﬂows are induced by the graph, and thus, the total ﬂow—in plus out—at each vertex is approximately conserved:
[Equation]
In the case of a road network, ﬂow conservation implies that vehicles mostly remain on the road.
Flow Estimation Problem.
Given a graph G ( V , E , X ) with partial ﬂow observations ˆ f ∈ R m (cid:48) for a subset E (cid:48) ⊆
E of edges ( ˆ f e is the ﬂow for e ∈ E (cid:48) , m (cid:48) =
|E (cid:48) | < m ), predict ﬂows for edges in E \ E (cid:48) .
In our road network example, partial vehicle counts ˆ f might be measured by sensors placed at a few segments, and the goal is to estimate counts at the remaining segments.
One would expect ﬂows not to be fully conserved in most applications due to the existence of inputs and outputs, such as parking lots and a power generators/consumers.
In case these input and output values are known exactly, they can be easily incorporated to our problem as ﬂow observations.
Moreover, if they are known approximately, we can apply them as priors (as will be detailed in the next section).
For the remaining of this paper, we assume that inputs and outputs are unknown and employ ﬂow conservation as an approximation of the system.
Thus, different from classical ﬂow optimization problems, such as min-cost ﬂow (Ahuja et al., 1988), we assume that ﬂows are conserved approximately.
Notice that our problem is similar to the one studied in Jia et al.
However, while their deﬁnition also assumes ﬂow conservation, it does not take into account edge features.
We claim that these features play important role in capturing the physics of ﬂows.
Our main contribution is a new model that is able to learn how to regularize ﬂows based on edge features using neural networks.
O UR A PPROACH : P HYSICS +L EARNING
In this section, we introduce our approach for the ﬂow estimation problem, which is summarized in Figure 1.
We formulate ﬂow estimation as an optimization problem (Section 3.1), where the interplay between the ﬂow network topology and edge features is deﬁned by the physics of ﬂow graphs.
Flow estimation is shown to be equivalent to a regularized least-squares problem (Section 3.2).
Moreover, we describe how the effect of edge features and the graph topology can be learned from data using bilevel optimization and neural networks in Section 3.3.
Finally, we propose a reverse-mode differentiation algorithm for ﬂow estimation in Section 3.4.
3. F LOW E STIMATION VIA O PTIMIZATION
The extent to which ﬂow conservation (Equation 2) holds for ﬂows in a graph is known as divergence and can be measured using the oriented incidence matrix B ∈ R n × m of G .
The matrix is deﬁned as follows, B ij = 1 if ∃ u such
that e j = ( v i , u ) ∈ E , B ij = − 1 if ∃ u such that e j = ( u, v i ) ∈ E , and
B ij = 0 , otherwise.
Given B and f , the divergence at a vertex u can be computed as:
[Equation]
And thus, we can compute the total (squared) divergence in the graph as || B f || 22 = f
[Equation]
( B f ) u ) 2 .
One could try to solve the ﬂow estimation problem by minimizing || B f || 22 while keeping the observed ﬂows ﬁxed, however, this problem is ill-posed—there might be multiple solutions to the optimization.
The standard approach in such a scenario is to resort to regularization.
In particular, we apply a generic regularization function Φ with parameters Θ as follows:
[Equation]
where Ω is the domain of f , f
(0) ∈
R m is a prior for ﬂows, f e ( ˆ f e ) are entries of f ( ˆ f ) for edge e and the constraint guarantees that observed ﬂows are not changed.
Priors f (0) , not be confused with observed ﬂows ˆ f , should be set according to the application (e.g., as zero, based on a black-box model or historical data).
Regarding the domain Ω , we consider Ω = R m and Ω
= R m + .
The second case is relevant for directed graphs—when ﬂows must follow edge orientations (e.g., trafﬁc).
In Jia et al.
(2019), the authors set Φ( f , X, f (0) ; Θ) as λ
2 ||
f || 22 for a regularization parameter λ , which implies a uniform zero prior with an L 2 penalty over edges.
We claim that the regularization function plays an important role in capturing the physics of ﬂow graphs.
As an example, for a power network, Φ should account for the resistance of the lines.
Thus, we propose learning the regularization from data.
Our approach is based on a least-squares formulation, which will be described next.
3. R EGULARIZED L EAST - SQUARES F ORMULATION
Flow estimation problem can be viewed as an inverse problem (Tarantola, 2005).
Let x ∈ R m − m (cid:48) be the vector of missing ﬂows and H ∈ R m × m − m (cid:48) be a matrix such that H ij = 1 if f i maps to x j (i.e., they are associated to the same edge), and H i,j = 0 , otherwise.
Moreover, let (cid:101) f ∈ R m be such that (cid:101)
f e = ˆ f e if e ∈ E (cid:48) and (cid:101) f
i = 0 , otherwise.
Using this notation, we deﬁne ﬂow estimation as
BH x = − B (cid:101) f
+ (cid:15) , where BH is a forward operator, projecting x to a vector of vertex divergences, and − B (cid:101) f
+ (cid:15) is the observed data, capturing (negative) vertex divergences for observed ﬂows.
The error (cid:15) can be interpreted as noise in observations or some level of model misspeciﬁcation.
We can also deﬁne a regularized least-squares problem with the goal of recovering missing ﬂows x :
[Equation]
where Ω (cid:48) is a projection of the domain of f to the space of x , || x
|| 2 M = x (cid:124)
M x is the matrixscaled norm of x and x (0) ∈
R m − m (cid:48) are priors for missing ﬂows.
The regularization function Φ( f , X ; f (0) , Θ) has the form || x − x (0)
||
2 Q , where the matrix Q is the output of a function Q ( X ; Θ) of parameters
Θ and edge features X .
We focus on the case where Q ( X ; Θ) is a non-negative diagonal matrix.
Equation 3 has a Bayesian interpretation, with x being a maximum likelihood estimate under a
Gaussian assumption—i.e., x ∼ N ( x (0) , Q ( X ; Θ) − 1 ) and B (cid:101) f ∼ N (0 , I ) (Tarantola, 2005).
Thus,
Q ( X ; Θ) captures the variance in ﬂow observations ˆ f in prior estimates f (0) compared to the one.
This allows the regularization function to adapt to different edges based on their features.
For instance, in our road network example, Q ( X ; Θ) might place a lower weight on ﬂow conservation for ﬂows at a road segment with a small number of lanes, which are possible trafﬁc bottlenecks.
Given the least-squares formulation described in this section, how do we model the regularization function Q and learn its parameters Θ ?
We would like Q to be expressive enough to be able to capture complex physical properties of ﬂows, while Θ to be computed accurately and efﬁciently.
We will address these challenges in the remaining of this paper.
3. B ILEVEL O PTIMIZATION FOR M ETA - LEARNING THE P HYSICS OF F LOWS
This section introduces a model for ﬂow estimation that is able to learn the regularization function
Q ( X ; Θ) in Equation 3 from data using bilevel optimization and neural networks.
Bilevel formulation.
We learn the parameters Θ that determine the regularization function Q ( X ; Θ) using the following bilevel optimization formulation:
[Equation]
where the inner (lower) problem is the same as Equation 3 and the outer (upper) problem is the expected loss with respect to ground truth ﬂows ˆ x —which we estimate using cross-validation.
Notice that optimal values for parameters Θ and missing ﬂows x are both unknown in the bilevel optimization problem.
The expectation in Equation 4 is a function of multiple instances of the inner problem (Equation 5).
Each inner problem instance has an optimal solution x ∗ that depends on parameters Θ .
In general, bilevel optimization is not only non-convex but also NP-hard (Colson et al., 2007).
However, recent gradient-based solutions for bilevel optimization have been successfully applied to large-scale problems, such as hyper-parameter optimization and meta-learning (Franceschi et al., 2018; Lorraine et al., 2020).
We will ﬁrst describe how we model the function Q ( X ; Θ) and then discuss how this problem can be solved efﬁciently using reverse-mode differentiation.
We propose to model Q ( X ; Θ) using a neural network, where X are inputs, Θ are learnable weights and the outputs are diagonal entries of the regularization matrix.
This is a natural choice due to the expressive power of neural nets (Cybenko, 1989; Xu et al., 2018).
Multi-Layer Perceptron (MLP).
An MLP-based Q ( X ; Θ) has the following form:
[Equation]
where MLP ( X ; Θ) ∈ R n − m (cid:48) .
For instance, Q ( X ; Θ) can be a 2-layer MLP:
[Equation]
where Θ = { W (1) , W (2) } , W (1) ∈ R d × h , W (2) ∈
R h × 1 , h is the number of nodes in the hidden layer, both a and b are activation functions, and the bias was omitted for convenience.
Graph Neural Network (GNN).
The MLP-based approach assumes that each entry [ Q ( X ; Θ)] e,e associated to an edge e is a function of its features X [ e ] only.
However, we are also interested in how entries [ Q ( X ; Θ)]
e,e might depend on the features of neighborhood of e in the ﬂow graph topology.
Thus, we consider the case where Q ( X ; Θ) is a GNN, which is described in the Appendix.
3. F LOW E STIMATION A LGORITHM
We now focus on how to solve our bilevel optimization problem (Equations 4 and 5).
Our solution applies gradient-based approaches (e.g., SGD (Bottou & Bousquet, 2008), Adam (Kingma & Ba, 2014)) and, for simplicity, our description will be based on the particular case of Gradient Descent and assume a zero prior ( x (0) = 0 ).
A key challenge in our problem is to efﬁciently approximate the gradient of the outer objective with respect to the parameters Θ , which, by the chain rule, depends on the gradient of the inner objective with respect to Θ .
We ﬁrst introduce some extra notation to describe the outer problem (Equation 4).
Let (ˆ f k , ˆ g k ) be one of K train-validation folds, both containing ground-truth ﬂow values, such that ˆ f k ∈ R p and ˆ g k ∈ R q .
For each fold k , we apply the inner problem (Equation 5) to estimate missing ﬂows x k .
Estimates for all folds are concatenated into a single vector x =
[ x 1 ; x 2 ; .
; x K ] and the same for validation sets ˆ g =
[ ˆ g 1 ; ˆ g 2 ; . . .
ˆ g K ] .
Moreover, we deﬁne a matrix R ∈ R q × ( m − m (cid:48) )
such that R ij = 1 if prediction x j corresponds to validation ﬂow ˆ g i and R ij = 0 , otherwise.
Using this representation, we can approximate the expectation in the outer objective as Ψ( x , Θ) =
(1 /K )
|| R x − ˆ g || 22 , where x depends implicitly on Θ .
We also introduce Υ Θ ( x ) as the inner problem objective.
Moreover, let Γ j ( x k,j − 1 , Θ i − 1 ) be one step of gradient descent for the value of x k at iterations j and Θ with learning rate β :
[Equation]
where H k , Q k and (cid:101)
f k are the matrix H , a sub-matrix of Q ( X ; Θ i − 1 ) and the observed ﬂows vector (cid:101) f
(see Section 3.2)) for the speciﬁc fold k .
We have assumed the domain ( Ω (cid:48) ) of ﬂows x k,j to be the set of real vectors.
For non-negative ﬂows, we add the appropriate proximal operator to Γ j .
Our algorithm applies Reverse-Mode Differentiation (RMD) (Domke, 2012; Franceschi et al., 2017) to estimate ∇ Θ Ψ and optimizes Θ also using an iterative algorithm.
The main idea of RMD is to ﬁrst unroll and store a ﬁnite number of iterations for the inner problem x 1 , x 2 , . . .
x J and then reverse over those iterations to estimate ∇ Θ Ψ , which is computed as follows:
[Equation]
In particular, our reverse iteration is based on the following equations:
[Equation]
where ∂ Q ( X ; Θ i ) /∂
Θ is the gradient of the regularization function Q ( X ; Θ) evaluated at Θ i .
In our case, this gradient is the same as the neural network gradients and is omitted here for convenience.
Algorithm 1 describes our RMD approach for ﬂow estimation.
It receives as inputs the ﬂow network G ( V , E , X ) , K train-validation folds { (ˆ f k , ˆ g k ) }
Kk =1 , and also hyperparameters T , J , α , and β , corresponding to the number of outer and inner iterations, and learning rates for the outer and inner problem, respectively.
Its output is a vector of optimal parameters Θ for the regularization function
Q ( X ; Θ) according to the bilevel objective in Equations 4 and 5.
We use ←− Θ to indicate our estimate of ∇ Θ Ψ(Θ i ) .
Iterations of the inner problem are unrolled and stored for each train-validation fold in lines 5-12.
Notice that ﬂow estimates x are initialized only once (line 2).
Reverse steps, which produce an estimate ←− Θ , are performed in lines 13-17.
We then use ←− Θ to update our estimate of Θ in line 17.
The time and space complexities of the algorithm are O ( TJKm ) and O ( Jm ) , respectively, due to the cost of computing and storing the inner problem iterations.
Require:
Flow network G ( V , E , X ) , train-validation folds { (ˆ f k , ˆ g k ) }
Kk =1 , number of outer iterations
T and inner iterations J , learning rates α and β
Ensure: Regularization parameters Θ 1: Initialize parameters Θ2:
Initialize missing ﬂows x k, 0 for all k 3: ˆ g ← [ ˆ g 1 ; . . .
ˆ g K ] 4: B ← incidence matrix of G 5: for outer iterations i = 1 , . . .
T do 6: for inner iterations j = 1 , . . .
J do 7: for folds k = 1 , . . .
K do 8: x k,j ← x k,j − 1 − 2 β
[ H (cid:124) k B (cid:124)
( BH k x k,j − 1 + B (cid:101) f
k )
+ 2 Q k x k,j − 1 ] 9: end for 10: x j ← [ x 1 ,j ; . . .
x K,j ] 11: end for 12: z J ← (2 /K )
R T ( R x J − ˆ g ) 13: for reverse inner iterations j = T − 1 , . . .
1 do 14: ←− Θ ← ←− Θ − 4 β
z j +1 ( ∂ Q ( X ; Θ i − 1 ) /∂
Θ)
x j + 15:
z j ← z j +
[ I − 2 β
( H (cid:124) B (cid:124) BH + Q ( X ; Θ i − 1 ))] 16: end for 17:
Update Θ i ← Θ i − 1 − α ←− Θ 18: end for 19: return parameters
Θ
I
As discussed in the previous section, bilevel optimization is non-convex and thus we cannot guarantee that Algorithm 1 will return a global optima.
In particular, the learning objective of our regularization function Q ( X ; Θ) is non-convex—it is a neural network.
However, the inner problem (Equation 5) in our formulation has a convex objective (least-squares).
In Franceschi et al.
(2018), the authors have shown that this property implies convergence.
We also ﬁnd that our algorithm often converges to a good estimate of the parameters in our experiments.
E XPERIMENTS
We evaluate our approaches for the ﬂow estimation problem using two real datasets and a representative set of baselines and metrics.
Due to space limitations, we provide an extended version of this section, with more details on datasets, experimental settings and additional results in the Appendix.
4. D ATASETS
This sections summarizes the datasets used in our evaluation.
We normalize ﬂows values to [0 , 1] and map discrete features to real vector dimensions using a one-hot enconding.
Trafﬁc:
Vertices represent locations and directed edges represent road segments between two locations in the Los Angeles County, CA.
Flows are daily average vehicle counts measured by sensors placed along highways in the year 2018.
We assign each sensor to an edge in the graph based on proximity and other sensor attributes.
Our road network covers the Los Angeles County area, with 5 , 749 vertices, 7 , 498 edges, of which 2 , 879 edges ( 38% ) have sensors.
The following features were mapped to a 18-dimensional vector: lat-lon coordinates, number of lanes, max-speed, and highway type (motorway, motorway link, trunk, etc.)
, in-degree, out-degree, and centrality (pagerank).
The in-degree and centrality of an edge are computed based on its source vertex.
Similarly, the out-degree of an edge is the out-degree of its target vertex.
Power: Vertices represent buses in Europe, undirected edges are power transmission lines and edge ﬂows measure the total active power (in MW) being transmitted through the lines.
The dataset is obtained from PyPSA-Eur (H¨orsch et al., 2018;
Brown et al., 2017)—an optimization model of the
European power transmission system—which generates realistic power ﬂows based on solutions of optimal linear power ﬂow problems with historical production and consumption data.
Default values [Table] metrics, with up to 20% lower RMSE than Div using Power. were applied for the PyPSA-Eur settings.
The resulting graph has 2,048 vertices, 2,729 edges and 14-dimensional feature vectors capturing resistance, reactance, length and number of parallel lines, nominal power, edge degree etc.
Please see the Appendix for more details.
4. E XPERIMENTAL S ETTINGS
Evaluation metrics: We apply Pearson’s correlation (CORR), Mean Absolute Percentage Error (MAPE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE) to compare groundtruth and predicted ﬂows.
These metrics are formally deﬁned in the Appendix.
Baselines: Divergence minimization ( Div ) (Jia et al., 2019) maximizes ﬂow conservation using a single regularization parameter λ —independent of edge features—and we optimize λ using line search in a validation set of ﬂows.
Multi-Layer Perceptron ( MLP ) is a 2-layer neural network with
ReLU activations for all layers, that learns to predict ﬂows based on edge features only.
Graph
Convolutional Network ( GCN ) is a 2-layer graph neural network, also with ReLU activations and
Chebyshev convolutions of degree 2, that learns to predict the ﬂows using both edge features and the topology but disregarding ﬂow conservation (Kipf & Welling, 2016; Defferrard et al., 2016). We also consider two hybrid baselines.
MLP-Div applies the predictions from MLP as priors to Div .
Similarly, predictions from GCN are used as priors for GCN-Div .
For both hybrid models we also optimize the regularization parameter λ .
Our approaches: We consider three variations of Algorithm 1.
However, one important modiﬁcation is that we perform the reverse iterations for each fold—i.e., folds are treated as batches in SGD.
Bil-MLP and Bil-GCN apply our reverse-mode differentiation approach using an MLP and a GCN as a regularizer.
Moreover, both approaches use zero as the prior x (0) .
Bil-GCN-Prior applies the
GCN predictions as ﬂow priors.
Architectures of the neural nets are the same as the baselines.
4. F LOW E STIMATION A CCURACY
Table 1 compares our methods and the baselines in terms of several metrics using the Trafﬁc and
Power datasets.
Values of CORR achieved by MLP and GCN for Trafﬁc are missing because they were undeﬁned—they have generated predictions with zero variance for at least one of the train-test folds.
All methods suffer from high MAPE errors for Power, which is due to an over-estimation of small ﬂows.
Bil-GCN achieves the best results in both datasets in terms of all metrics, with 6% and 20% lower RMSE than the best baseline for Trafﬁc and Power, respectively.
However, notice that
Bil-MLP and Bil-GCN achieve very similar performance for Power and Bil-GCN-Prior does not outperform our other methods.
We also show scatter plots with the true vs. predicted ﬂows for some of the best approaches in Figure 2.
Trafﬁc has shown to be the more challenging dataset, which can be explained, in part, by training data sparsity—only 38% of edges are labelled.
[Figure]
[Figure]
4. A NALYSIS OF R EGULARIZERS
Figure 3 illustrates the regularization function learned by Bil-MLP.
We focus on Bil-MLP because it can be analyzed independently of the topology.
Figures 3a-3c show scatter plots where the x and y axes represent the value of the regularizer and features, respectively.
For Power, Bil-MLP captures the effect of resistance over ﬂows (Fig. 3a).
However, only high values of resistance are mostly affected—that is the reason few points can be seen and also explains the good results for Div.
We did not ﬁnd signiﬁcant correlation for other features, with the exception of reactance, which is related to resistance.
For Trafﬁc, the model learns how the number of lanes constrains the ﬂow at a road segment (Fig. 3b).
Results for speed limit are more surprising, 45mph roads are less regularized (Fig. 3c).
This is an evidence that regularization is affecting mostly trafﬁc bottlenecks in highways—with few lanes but a 65mph speed limit.
To further investigate this result, we also show the regularizers over the Trafﬁc topology in Figure 3d.
High regularization overlaps with wellknown congested areas in Los Angeles, CA (e.g., Highway 5, Southeast).
These results are a strong evidence that our methods are able to learn the physics of ﬂows in road trafﬁc and power networks.
[Figure]
R ELATED W ORK
Flow graphs are quite ubiquitous in engineering, biomedical and social sciences.
Two important properties of ﬂow graphs are that their state space is deﬁned by a graph topology and their dynamics are governed by the physics (or logic) of the problem of interest.
We refer to Bressan et al.
(2014) for a uniﬁed characterization of the mathematical treatment of ﬂow graphs.
Notice that these studies do not address the ﬂow inference problem and their applications to real data is limited (Herrera et al., 2010; Work et al., 2010).
Moreover, we focus on long term ﬂows (e.g. daily vehicle trafﬁc ﬂows) and not on the dynamics.
This simpliﬁes the equations of our model to the conservation law.
Flow inference via divergence minimization was originally proposed in Jia et al.
However, their work has not considered edge features and instead applied a single regularization parameter to the norm of the ﬂow vector f in Equation 2.
Our work leverages relevant edge features to learn the interplay between ﬂow conservation and local predictions (priors).
Thus, we generalize the formulation from Jia et al.
(2019) to the case of a learnable regularization function Q (Θ , X ) .
Our experiments show that the proposed approach achieves superior results in two datasets.
Flow optimization problems, such as min-cost ﬂow, max-ﬂow and multi-commodity ﬂow, have a long history in computer science (Ahuja et al., 1988; Ford Jr & Fulkerson, 2015).
These problems impose ﬂow conservation as a hard constraint, requiring full knowledge of source and sink vertices and noiseless ﬂow observations.
Our approach relaxes these requirements by minimizing the ﬂow divergence (see Equation 2).
Moreover, our problem does not assume edge capacities and costs.
The relationship between ﬂow estimation and inverse problems is of particular interest due to the role played by regularization Engl et al.
in the solution of ill-posed problems.
Recent work on inverse problems has also focused on learning to regularize based on data and even learning the forward operator as well—see Arridge et al. for a review.
The use of the expression “learning the physics” is also popular in the context of the universal differential equation framework, which enables the incorporation of domain-knowledge from scientiﬁc models to machine learning (Raissi et al., 2019;
Long et al., 2018; Rackauckas et al., 2020).
Differential equations play a special role in expressing conservation laws in the sciences and engineering, including the conservation of ﬂows in physical models for ﬂow graphs Bressan et al. ; Garavello & Piccoli (2006).
Bilevel optimization in machine learning has been popularized due its applications in hyperparameter optimization (Bengio, 2000; Larsen et al., 1996).
In the last decade, deep learning has motivated novel approaches able to optimize millions of hyperparameters using gradient-based schemes (Maclaurin et al., 2015; Lorraine et al., 2020; Pedregosa, 2016).
Our ﬂow estimation algorithm is based on reverse-mode differentiation, which has been shown to be a scalable approach for bilevel optimization (Franceschi et al., 2017; Domke, 2012; Maclaurin et al., 2015).
Another application of bilevel optimization that is quite related to ours is meta-learning, where we are given multiple small datasets on different but related tasks and the goal is to learn meta-parameters that are shared among tasks (Franceschi et al., 2018; Grefenstette et al., 2019).
Flow estimation is also related to semi-supervised learning on graphs (Zhu et al., 2003; Belkin et al., 2006;
Zhou et al., 2004), which is often cast as inference over vertices of a graph given partial labels.
These approaches can be applied to the ﬂow estimation problem via the line graph transformation Jia et al.
A duality relation between between a recent approach for predicting vertex labels (or signals) (Hallac et al., 2015) and min-cost ﬂows was shown in Jung (2020).
However, the same relation does not hold for ﬂow estimation.
More recently, graph neural network models, which generalize deep learning to graph data, have been shown to outperform traditional semi-supervised learning methods in many tasks (Kipf & Welling, 2016; Hamilton et al., 2017;
Veli ˇ ckovi ´ c et al., 2018).
These models have also been applied for trafﬁc forecasting (Li et al., 2017; Yu et al., 2018; Yao et al., 2019).
One advantage of GNNs is that they are able to handle node/edge features.
Different from our approach, traditional GNNs do not conserve ﬂows.
We show that our models outperform GNNs at ﬂow prediction.
Moreover, we also apply GNNs as a regularization function in our model.
C ONCLUSIONS
We have introduced an approach for ﬂow estimation on graphs by combining a conservation law and edge features.
Our model learns the physics of ﬂows from data by combining bilevel optimization and deep learning.
Experiments using trafﬁc and power networks have shown that the proposed model outperforms a set of baselines and learns interpretable physical properties of ﬂow graphs.
As future work, we want to investigate whether the idea of learning to regularize can also be effective when the graph topology expresses more general relationships, such as correlation/similarity.
This could lead to an interesting alternative to current graph neural network architectures for semisupervised learning, such as GCNs.
Moreover, while we have focused on learning a diagonal regularization matrix, we want to apply our framework to the case of a full matrix—which likely will require regularizing the matrix itself.
Finally, we are also interested in combining different edge measurements in order to learn more complex physical laws, such as described by the fundamental diagram in the LWR model Lighthill & Whitham (1955); Daganzo (1994; 1995); Garavello & Piccoli (2006), and also the ﬂow dynamics.
This will allow the prediction of other ﬂow related measurements, such as density (or occupation) and speeds in trafﬁc networks.
A T ABLE OF S YMBOLS
the main symbols used in our paper.
B B ILEVEL O PTIMIZATION WITH G RAPH N EURAL N ETWORKS
This section is an extension of Section 3.3.
Here, we consider the case where Q ( X ; Θ) is a GNN:
[Equation]
For instance, we apply a 2-layer spectral Graph Convolutional Network (GCN) with Chebyshev convolutions Defferrard et al.
; Kipf & Welling (2016);
Hammond et al.
[Equation]
where (cid:101)
L = 2 /λ max L − I , L is the normalized Laplacian of the undirected version of the line graph
G (cid:48) of G , λ max is the largest eigenvalue of L , T z ( (cid:101) L ) is a Chebyshev polynomial of (cid:101) L with order z and W ( i )
z is the matrix of learnable weights for the z -th order polynomial at the layer i .
In a line graph, each vertex represents an edge of the undirected version of G and two vertices are connected if their corresponding edges in G are adjacent.
Morever L = I − D − 1 / 2 AD − 1 / 2 , where A and D are the adjacency and degree matrices of G (cid:48) .
Chebyshev polynomials are deﬁned recursively, with
T z ( y ) =
2 yT z − 1 ( y ) − T z − 2 ( y ) and T 1 ( y ) =
y .
In our experiments, we compare GCN against MLP regularization functions.
We have also applied the more popular non-spectral graph convolutional operator Kipf & Welling (2016) but preliminary results have shown that the Chebyshev operator achieves better performance in ﬂow estimation.
C E XTENDED E XPERIMENTAL S ECTION
This section in an extension of Section 4.
C. M ORE D ETAILS ON D ATASETS
Trafﬁc:
Flow data was collected from the Caltrans—the California Department of Transportation—
PeMS (Performance Measurement System).
1 Sensors are placed at major highways in the state.
We use sensor geo-locations and other attributes to approximately match them to a compressed version of road network extracted from Openstreetmap.
The compression merges any sequence of segments without a branch, as these extra edges would not affect the ﬂow estimation results.
We emphasize that this dataset is not of as high quality as Power, due to possible sensor malfunction and matchings of sensors to the wrong road segments.
This explains why ﬂow estimation is more challenging in
Trafﬁc.
Figure 4 is a visualization of our trafﬁc dataset with geographic (lat-lon) located vertices and colors indicating light versus heavy trafﬁc (compared to the average).
The road segments in the graph (approximately) cover the LA County area.
We show the map (from Openstreetmap) of the area covered by our road network in Figure 5.
Power: We will provide more details on how we build the power dataset.
PyPSA (Python for
Power System Analsys) is a toolbox for the simulation of power systems Brown et al.
We applied the European transmission system (PyPSA-Eur), which covers the ENTSO-E area H ¨ orsch et al.
(2018), to generate a single network snapshot.
Besides the PyPSA-Eur original set of edges, which we will refer to as line edges, we have added a set of bus edges.
These extra edges allow us to represent power generation and consumption as edge ﬂows.
For the line edges, we cover the following PyPSA attributes (with their respective PyPSA identiﬁers 3 ): reactance (x), resistance(r), capacity (s nom), whether the capacity s nom can be extended (s nom extendable), the capital cost of extending s nom (capital cost), the length of the line (length), the number of parallel lines (num parallel) and the optimized capacity (s nom opt).
For bus lines, the only attribute is the control strategy (PQ,
PV or Slack).
Notice that we create a single vector representation for both line and bus lines by adding an extra indicator position (line or bus).
Moreover, categorial attributes (e.g., the control strategy) were represented using a one-hot encoding.
Figure 6 is a visualization of our power dataset with geographic (lat-lon) located vertices and colors indicating high versus low power (compared to the average).
C. E VALUATION M ETRICS
We apply the following evaluation metrics for ﬂow estimation.
Let f true and f pred be m (cid:48) - dimensional vectors with true and predicted values for missing ﬂows associated to edges in E \ E (cid:48) .
Correlation (Corr): cov ( f pred , f true ) / ( σ ( f pred ) .σ
( f true ))
where cov is the covariance and σ is the standard deviation.
Better seen in color.
See Figure 5 for map of the area.
Mean Absolute Percentage Error (MAPE):
[Equation]
Mean Absolute Error (MAE):
[Equation]
Root Mean Squared Error (RMSE
[Equation] [Equation]
Divergence (Div):
[Equation]
C. M ORE ON E XPERIMENTAL S ETTINGS
Train/test splits: We report results of a 10-fold cross-validation based on the set of labelled ﬂows.
Moreover, we use 10% of training ﬂows for validation.
Implementation: We have implemented Algorithm 1 using PyTorch, CUDA, and Higher Grefenstette et al.
a meta-learning framework that greatly facilitates the implementation of bilevel optimization algorithms by implicitly performing the reverse iterations for a list of optimization algorithms, including SGD.
Moreover, our GCN implementation is based on the Deep Graph Library (DGL) Wang et al.
We have included our implementation as supplementary material.
Hardware:
We ran our experiments on a single machine with 4 NVIDIA GeForce RTX 2080 GPUs (each with 8Gb of RAM) and 32 Intel Xeon CPUs (2.10GHz and 128Gb of RAM).
Hyperparameter settings: We have selected the parameters based on RMSE for each method using grid search with learning rate over [10 0 , 10 − 1 , 10 − 2 , 10 − 3 ] and number of nodes in the hidden layer over [4 , 8 , 16] .
The total number of iterations was set to 3000 for Min-Div and 5000 for MLP and GCN, all with early stop on convergence after 10 iterations.
For our methods (both based on
Algorithm 1), we set T = 10 , J = 300 , α = 10 − 2 , β = 10 − 2 and K = 10 in all experiments.
C. D IVERGENCE R ESULTS
Although the main goal of ﬂow estimation is to minimize the ﬂow prediction loss, we also evaluate how our methods and the baselines perform in terms of divergence (or ﬂow conservation) in
Table 3.
As expected, MLP and GCN do not conserve the ﬂows.
However, interestingly, our methods (Bil-MLP and Bil-GCN) achiever higher ﬂow conservation than Min-Div.
This is due to the regularization parameter λ , which is tuned based on a set of validation ﬂows.
[Table]
D T RUE VS . P REDICTED F LOWS
Figure 7 shows scatter plots with the true vs. predicted ﬂows that are missing from Figure 2.
[Figure]
D. V ISUALIZATION OF R EGULARIZER FOR P OWER
Figure 8 shows the regularizers over the Power network topology.
As discussed in Section 4.4, the regularizer affects mostly a few top resistance edges.
For the remaining ones, regularizers have small value.
Notice that these high resistance edges are associated with lines transmitting small amounts of power, as shown in Figure Figure 6, and have a large impact on the overall ﬂow estimation accuracy.
[Table] [Table]
D. R UNNING T IME
Table 4 shows the average running times—over the 10-fold cross-validation—of our methods and the baselines for the Trafﬁc and Power datasets.
We show both training and test times.
The results show that our reverse-mode differentiation algorithm adds a signiﬁcant overhead on training time for
Trafﬁc, taking up to 4 times longer than Min-Div to ﬁnish.
As described in Section 3.4, this is due mainly to the cost of computing and storing the inner problem iterations.
On the other hand, all the methods are efﬁcient at testing.
GCN converged quickly (due to early stopping) for both datasets.
However, it achieved poor results for Power, as shown in Table 1, which is a sign of overﬁtting or underﬁtting.
Notice that the results reported are the best in terms of RMSE.