F INDING S TRUCTURE AND C AUSALITY IN L INEAR P ROGRAMS
A BSTRACT
Linear Programs (LP) are celebrated widely, particularly so in machine learning where they have allowed for effectively solving probabilistic inference tasks or imposing structure on end-to-end learning systems.
Their potential might seem depleted but we propose a foundational, causal perspective that reveals intriguing intra- and inter-structure relations for LP components.
We conduct a systematic, empirical investigation on general-, shortest path- and energy system LPs.
O PTIMIZATION P ROBLEMS , THEIR S TRUCTURE AND C AUSALITY
Linear Programs (LP) stand as the entry ticket to the literature around mathematical optimization.
Their simplicity is characterized by linear cost and constraint vectors that provide the problem of study with an inherent structure .
Both, the LPs simplicity and its structure have leveraged its success across many applications—also in machine learning.
While classical examples involve the diet problem (Dantzig, 1990), the assignment problem (Kuhn, 1955), or the shortest path (SP) problem (Bellman, 1958), more modern applications of LPs involve probabilistic reasoning as in MAP inference (Weiss et al., 2007).
The classical problems are special instances of LPs that assume a certain structure within their constraints A and decision variable x .
For instance in SP the decision x consists of indicators to path segments ( i, j ) to be chosen for the final path a → . . .
i →
j → . . .
b while A will dictate what constitutes a “legal” path.
As another example, in the highly non-trivial
MAP inference case we will act on a polytope of rather abstract objects while inducing no special structure on the problem itself.
While MAP inference is a concrete example of an important machine learning (ML) task that we might be interested in and lends itself to an LP description, the recent times have shown an increased interest by the ML community in general for combinatorial problems.
In research around adversarial examples, LPs and its integral variants were used to evaluate robustness of trained classifier in a defensive (Tjeng et al., 2019) and active setting (Wu et al., 2020).
In research around end-to-end learning systems, LPs have been made differentiable through the use of perturbations in the Gumbel-Max sense (Berthet et al., 2020).
More recently, both adversarials and the differentiable LPs were combined to define a new notion of adversarial attack which directly applies to LPs (and not arbitrary classifiers) by bridging the gap to Pearlian causality and its hidden confounders (Ze ˇ cevi ´ c et al., 2021).
Furthermore, a magnitude of efforts has been concerned with allowing learned agents to incorporate structured knowledge, in the constraint-sense that LPs but more general cone programs (like quadratic or semi-definite programs), to leverage performance on downstream tasks (Agrawal et al., 2019; Mandi & Guns, 2020; Paulus et al., 2021).
While it might seem that research around LPs has depleted all which LPs could possibly offer, we believe to show in this work that there is a side to LPs that has remained hidden and is yet to be discovered.
We have seen that LPs provide an inherent structure through their constraints but also through the way the semantics of the variables is decided (e.g. the meaning of the decision variable x ).
But what about the intra- and inter-structure of the LP, that is, the relations between say different ( x i , x j ) or even between ( x k , A l, : ) ?
Can we identify a notion of causality within
LPs using the formal, counterfactual theory provided by (Pearl, 2009)?
While pure curiosity and the success of LPs on a wide variety of seemingly different tasks (involving ML) might not allow for a proper justification of the aforementioned research questions, an intuition lies in the inherent structure provided by LPs in that their constraints commonly carry meaning just as the variables of interest do—they are governed by certain rules , which one might equate to structural equations in a Structural Causal Model (SCM).
An important case in support of this anticipation is omnipresent in research around sustainable energy modelling.
Energy systems researcher might model certain physical laws like the law of conservation of energy with constraints in an LP where the decisions x i will now involve the capacity of the photo-voltaic system or the market bought electricity for a single household, and then the question arises on how these x i are causally related to each other but also to the laws and constraints that govern them (Schaber et al., 2012).
While this example corroborates our previous justification, the example itself does not come to a surprise since causality (as a concept not just a specific formal notion) stands at the core of human cognition and transitively at the core of science, engineering, business, and law (Penn & Povinelli, 2007).
Developmental psychology has shown children to act in the “manner of a scientist” (Gopnik, 2012), while artificial intelligence research dreams of automating said manner (McCarthy & Hayes, 1981).
Recent strides in cognitive science have pursued a computational model of the broad term causality (Gerstenberg et al., 2021), whereas Pearlian causality specifically has pressured forward into ML research (Bareinboim et al., 2020).
Various works in ML take inspiration from the latter approach leveraging causal notions and invariances to drive performance (Lopez-Paz et al., 2017; Kipf et al., 2018).
B EYOND C LASSICAL C AUSAL D ISCOVERY
Since its core formalization, Pearlian causality has seen many iterations (Pearl, 2009; Peters et al., 2017; Bongers et al., 2021) and so have methods for identifying the causal structure given data (for an overview see Eberhardt (2017)).
Typically, for an SCM M := ⟨ U , V , F , P ( U ) ⟩ with exogenous “nature” terms U , its joint distribution P ( U ) , the endogenous nodes of interest V and structural equations of the form
v i ← f i (pa i , u i ) ∈ F , we are classically interested in relations on V and sometimes also with regards to relations arising from U that might include hidden confounder as in non-Markovian SCM.
As we motivated in the previous section, if we know that our data stems from an LP , then we can further partition V into sub-categories such as decision or input variable x , parameter variables like the constraints ϕϕϕ (includes A , b ) and the cost c , but also the optimal solution s .
The last might be expressed as a structural equation where the former act as causes and typically we know how to compute this equation since it corresponds to the solver .
Fig.1 (left) illustrates such a graph.
Now, we might go a step further and use this prior knowledge (or bias) on the role of certain elements in V to ask questions about the relations of parameter to parameter or parameter to the solution 1 , see Fig.1 (right).
To investigate these LP-specific relations we just described, we propose a data-driven approach.
For this, we do not need to assume any specific knowledge on the actual LP—we simply observe data stemming from the LP.
For interpretation we will assume to know what certain dimensions should refer to (e.g. feature k corresponding to constraint entry
A i,j ).
Consider the following example which is a simple instance of the diet problem where we try to find a cost-efficient plan for consuming pizza ( x 1 ) and salad ( x 2 ) subject to what defines a diet to be “healthy”, for instance
[Equation]
where A 1 , : denotes the amount of fiber in g (per 100 g ) and A 2 , : denotes the calories in kcal (per 100 g ), respectively per dish, and b 1 ,
b 2 denote the minimal requirements in fiber and calories (the − 1 assures the lower bound).
We can now define an indicator function ϕ ( x ) which is 1 if Ax ≤ b and 0 otherwise.
Now, we might encounter our data situation where we are given a data set
D := { ( x i , y i ) } ni of size n where y i = ϕ
( x ) and we can assume to know that x refers to pizza and salad consumption whereas y denotes whether the consumption is healthy or not.
Given only the data and the knowledge on what it represents, what can we infer from it?
Can we discover a structural relationship , perhaps causal ?
It turns out, applying an induction approach f to the data can reveal information implicit in this special type of data .
For our concrete example, one such resulting structure is G : x i w i
→
y where G = f ( D ) and ∀ i, w i < 0 and | w 1 | > | w 2 | .
In words, we can infer 3 statements: (a) we can choose whether we consume pizza or salad independently , (b) our pizza and salad consumption will dictate whether our diet is healthy or not, and (c) pizza is worse for a healthy diet.
Especially, statement (c) is interesting and highly non-trivial since it is the information about the constraints A , b that we don’t know about but that are captured implicitly by y .
Fig.2 captures the general idea alongside this concrete example schematically.
C ASE - BY -C ASE E MPIRICAL I NVESTIGATION
Upon establishing an intuition for strucural (causal) perspective on LPs (Sec.1) and illustrating our approach on a concrete example (Sec.2), we are now going to study the different discovery settings empirically (also including special types of LPs such as shortest path).
In the following, we choose as induction method the score-based approach from (Zheng et al.
(2018); abbreviated, NT) which finds the best linear fit of the data (under optional regularization) while satisfying an acyclicity constraint.
Therefore, NT itself is an optimization problem 2 , one that aims at finding the best weighted, directed acyclic graph (DAG) given the data.
NT makes no claims on causality but it finds correlation (the best linear fit).
This observation was recently pointed out by (Reisach et al., 2021) who showed that NT might only sort variances.
Nonetheless, we choose NT for two reasons (a) correlation being an arguably appropriate proxy for an LP with its linear cost/constraints and thus pre-requisite for causation 3 at least up to confounder, and (b) NT being a simple, controllable formulation for effective experimentation.
Therefore, we can expect to reveal structure in LPs as previously pointed out.
Still, for future iterations of this work direction it would be sensible to consider the vast zoo of methods (which also provide guarantees on the causal side) and whether its majority vote keeps consistent with the subsequent results.
Minor technical details can be read up in the appendix.
3. G ENERAL L INEAR P ROGRAMS
Case: D = { ( x i , y i ) } ni .
If y = 1 for Ax ≤ b , then G = f ( D ) suggests the linear relationship y
= (cid:80)
i w i x i where w i < 0 (red values).
Accordingly, flipping the encoding to y = 0 indicating feasibility we observe an expected flip in weights as well, w i > 0 .
The x i are reported mutually independent which reflects in the data-generation where we choose x at random.
We generally provide f with 1 , 000 data points for our experiments.
Furthermore, we observe some effects being more important to y than others, e.g. increasing x 3 would make y =
(infeasible) the “quickest”.
This observation corroborates once more the fact that G captures implicit information about constraints
A , b only from D .
The diagonals are marked out, as are any cyclic relations, since f converged thus guaranteeing a DAG.
The result also reflects the actual computation of y although f does not impose any guarantee regarding the direction of the fit since we could have fitted each x i via y .
Case: ( b , y ) .
In this setting we consider a family of LPs parameterized by b while A , c are kept constant and y is as before.
We observe the same overall message as in the first case, in that y will “control” the magnitude of the b which dictate the placement of the polytope faces.
Case: ( b , s ) .
Again, b is parametric but now we consider the actual optimal solution s = arg max x LP
( x ; A , b , c ) instead of the feasibility of an arbitrary solution.
The first interesting observation (red, dotted rectangles) is ( b 1 , s 1 ) > 0 which suggests that b 1 is more restrictive for obtaining a solution in that specific dimension of decision, looking at column A : ,supports this intuition (blue denotes entry a ij > 0 ).
The second interesting observation is the “competition” between the dimensions of s since exploiting one s i usually comes with a decrease in another s j , i ̸
= j .
Case: ( c , s ) .
We consider two different data sets sampled under different random seeds.
We observe matching tendencies for pairs on the main diagonal ( c i , s i ) >suggesting that a decrease in cost comes with an increase of selecting that dimension for the final decision.
Again, confirmation for the competition between s i .
More interestingly, on the right data set we observe a pattern which requires some detailled inspection.
Inspecting the row ( c 1 , s ) and ( c 2 , s ) they show a negative impact on the latter element of s shifted relative to each other.
Inspecting the columns of A reveals the same patterns suggesting that for e.g. ( c 1 , s 1 ) the constraints were already “exhausted” such that sis being negatively impacted—and the same holds for the relations between c 2 , s 2 , s 4 .
Case: ( c , A , b , s ) .
Finally, we inspect the general LP case with all defining elements being parametric.
The first key observation is that all the detected relations are exclusive to parameter and solution .
Generally, the extracted patterns confirm the results from the previous cases.
A new scenario is revealed by the rows in A and their relation to s which form repeating, negative diagonals that suggest a shrinking of the LP polytope, thus the smaller A i, : the smaller s i , ∀ i .
3. S HORTEST P ATH L INEAR P ROGRAMS
In the previous section we considered general LPs, but now we move onto the shortest path (SP) problem which is an integral variant of the LP—one of the classical guises we discussed at the beginning in Sec.1.
Specifically, we consider two cases of SP much in the same fashion as before.
Case: ( x , y ) .
In this setting the graph to the corresponding SP problem is fixed.
As before, we observe a linear relationship between the selected edges (or path segments)
x i and the validity of a path (or collection of such edges) on the route from the left most node to the right most node, y .
More interestingly, we observe a maximal activation in x 5 which suggests that this specific segment imposes the strongest influence on what defines a valid path.
Indeed, looking at the graph we observe that if a path p is valid, then x 5 ∈ p .
Case: ( A , s ) .
In addition to fixed edges, this setting also contains some dynamically generated edges.
Our first observation is the pattern within A which simply reflects the “laws” that are characteristic of the SP problem formulation, aspects like “a visited node has exactly one incoming and at most one outgoing edge.”
The second observation concerns s where we observe that certain path segments tend to occur together in the sense that they usually belong to a valid, if not optimal, path.
3. E NERGY S YSTEM LP OF A S INGLE F AMILY H OUSEHOLD
x c PV c Bat c Buy Demand Cap PV Cap Bat Own-GenTotexCapex c PV c Bat c Buy
Demand Cap PV Cap Bat
Own-Gen Totex
Capex =0.1, =0.
c P V c B a t c B u y D
e m a n d C a p P V C a p B a t O w n - G e n T o t e x C a p e x
C a p B a t O w n - G e n T o t e x C a p e x c P V c B a t c B u y D e m a n d C a p P V C a p B a t O w n - G e n T o t e x C a p e x c P V c B a t c B u y D
e m a n d C a p P V C a p B a t O w n - G e n T o t e x C a p
e
x = 0 .
1 , = 0 .
To conclude our empirical section, we consider one final LP setting in which we take inspiration from (Schaber et al., 2012) for a simple, yet large energy system modelling the yearly energy consumption of single family household per hour ( 365 ∗ 24 = 8 , 760 hours multiplied by the number of constraints that are timedependent resulting in an LP with more than 35 , 000 constraints).
The constraints of the LP model aspects such as energy balance, photo-voltaics production limit or battery equalities (the specific LP is listed in the appendix).
Input denotes aspects of the energy system ( c ), whereas output denotes the resulting optimal decisions ( s ).
Therefore, the investigated case corresponds to ( c , x ) from before.
We observe Demand to correlate with all output dimensions, which seems appropriate.
However, we also miss out on relations we’d expect to observe like for instance Cap PV onto Cap Bat .
C ONCLUSIONS AND F UTURE W ORK
Starting our discussion from the success of LPs in various applications ranging from classic ones (like diet-, assignment-, or shortest path problems) to machine learning tasks (like MAP inference or adversarial examples), we took a causal perspective and devised a new paradigm for structural induction from data originated in a (usually hidden) LP to reconstruct insights.
Opposed to classical discovery, we can now reason about parameter to parameter or parameter to solution relations.
We followed this perspective with a thorough empirical investigation on a case-by-case basis.
As pointed out in Sec.1, causality stands at the core of cognition, and transitively at the core of artificial intelligence.
A counterfactual theory like the Pearlian notion to causality suggests to be an appropriate formal candidate for this endeavour.
In this short work, we investigated a causal perspective but only correlations empirically, therefore, we propose that future research shall consider to investigate an SCM-type of perspective on LPs theoretically .
Furthermore, raising the potential for a more general integration and feed-back between concepts from causality and mathematical optimization through an extension of the proposed setting to greater, real-world based examples might suggest to be fruitful.
A CKNOWLEDGMENTS
The authors thank the anonymous reviewers for their valuable feedback.
This work was supported by the ICT-48 Network of AI Research Excellence Center “TAILOR” (EU Horizon 2020, GA No 952215), the Nexplore Collaboration Lab “AI in Construction” (AICO) and by the Federal Ministry of Education and Research (BMBF; project “PlexPlain”, FKZ 01IS19081).
It benefited from the
Hessian research priority programme LOEWE within the project WhiteBox and the HMWK cluster project “The Third Wave of AI” (3AI).