Fixing the Bethe Approximation: How Structural Modiﬁcations in a Graph Improve Belief Propagation
Abstract
Belief propagation is an iterative method for inference in probabilistic graphical models.
Its wellknown relationship to a classical concept from statistical physics, the Bethe free energy, puts it on a solid theoretical foundation.
If belief propagation fails to approximate the marginals, then this is often due to a failure of the Bethe approximation.
In this work, we show how modiﬁcations in a graphical model can be a great remedy for ﬁxing the Bethe approximation.
Speciﬁcally, we analyze how the removal of edges inﬂuences and improves belief propagation, and demonstrate that this positive effect is particularly distinct for dense graphs.
INTRODUCTION
Message passing algorithms are an effective method for approximate inference in probabilistic graphical models [Koller and Friedman, 2009].
Although they often perform well in practice, there are only few guarantees about their theoretical behavior.
A remarkable milestone was the discovery of a direct connection between message passing algorithms and concepts of statistical mechanics [Yedidia et al., 2001], perhaps the most famous being the relationship between belief propagation (BP)
[Pearl, 1988] and the so-called Bethe free energy
[Bethe, 1935, Peierls, 1936].
One favorable property of BP is its exactness on trees.
On loopy graphs, however, it frequently suffers from two major issues: ﬁrst, it may fail to converge to a ﬁxed point and thus to ﬁnd reasonable estimates of the marginals.
Second, the ﬁxed points themselves may induce bad estimates of the marginals, in which case even convergence would not help.
To solve the ﬁrst issue, various techniques have been developed that can improve the convergence behavior of BP; e.g., one can damp the message updates [Murphy et al., 1999] or utilize elaborate scheduling schemes [Elidan et al., 2006,
Sutton and McCallum, 2007, Knoll et al., 2015, Aksenov et al., 2020].
Moreover, it depends on the properties of the graphical model [Tatikonda and Jordan, 2002, Ihler et al., 2005, Mooij and Kappen, 2007] and on the initialization of the messages
[Koehler, 2019, Knoll et al., 2021, Leisenberger et al., 2021] if and to which ﬁxed point BP converges.
The second issue might be even harder to overcome, as it is inherently linked to a failure of the Bethe approximation
[Weller et al., 2014].
The detrimental inﬂuence of loops can make the Bethe free energy non-convex and cause its local minima – and thus BP ﬁxed points – to be far away from the exact marginals.
Enhanced variants of free energy approximations [Yedidia et al., 2005] or loop corrections [Mooij et al., 2007] are prudent alternatives, that improve the accuracy but also increase the complexity.
In this work, we follow a different path: we aim to improve the approximation quality of the Bethe free energy itself.
To address this problem, we modify the structure of the graphical model and show how this transforms the Bethe free energy in a way that moves its local minima closer to the exact marginals.
In particular, we analyze the effect of removing individual edges from the graph.
This loop-breaking approach enforces a ’reconvexiﬁcation’ of the Bethe free energy and therefore not only improves the accuracy of ﬁxed points, but also the convergence behavior of BP.
We make a series of interesting theoretical contributions that arise from analyzing the behavior of the Bethe free energy on a ’small scale’.
More precisely, we introduce a measure for the discrepancy between two different representations of the Bethe free energy, each induced by a different graphical model, and then utilize this tool to relate variations in the
Bethe free energy to the characteristics of the model and the behavior of BP.
Theoretically and experimentally, we address the following questions: (i) How does edge removal inﬂuence the estimated marginals?
(ii) How does edge re-
moval inﬂuence the estimated partition function?
(iii) Which and how many edges – if at all – should we remove?
The structure of this paper is as follows:
Sec. 2 summarizes all relevant background on graphical models, BP, and the
Bethe approximation.
Sec. 3 contains a detailed theoretical analysis and the main results.
We experimentally validate our ﬁndings in Sec. 4 and conclude the paper in Sec. 5.
BACKGROUND
This introductory section provides a compact overview of the topics that we deal with: probabilistic graphical models, belief propagation, and the Bethe approximation.
We further introduce the Ising model and discuss related work.
2.1 PROBABILISTIC GRAPHICAL MODELS
We consider an undirected graph G =
( X , E ) , where X = { 1 , . .
, N } is a set of nodes and E ⊆
( i, j ) : i, j ∈ X } is a set of edges.
An edge connects two nodes if ( i, j ) ∈ E .
Note that we assume all edges to be undirected and hence ( i, j ) =
( j, i ) for all pairs of connected nodes; speciﬁcally we do not count edges twice.
Furthermore, N ( i ) denotes the neighborhood of node i (i.e., the set of nodes that are connected to i ) and d i := | N ( i ) | denotes the degree of i .
Let X 1 , . . .
X N be random variables (RVs) with state spaces X 1 , . . .
, X N .
A probabilistic graphical model (PGM) represents a joint probability distribution P X ( x ) over the
RVs, where each node represents a RV 1 and edges indicate statistical dependencies between RVs.
Formally, a PGM is a pair ( G , Φ ) that associates a set of potential functions (or potentials) Φ = { Φ 1 ( x 1 ) , . . .
, Φ K ( x K ) } with the graph G that are deﬁned over joint realizations of subsets
X 1 , . . .
, X K ⊆ X .
We shall focus on the class of binary pairwise models 2 that satisfy the following two assumptions:
ﬁrst, each RV has two states, i.e., X i =
X = { +1 , − 1 } .
Second, the potentials are deﬁned over either one (singleton potentials Φ i ( x i ) ) or two (pairwise potentials Φ ij ( x i , x j ) )
RVs.
Then the joint distribution factorizes as
[Equation]
where Z is the normalization constant or partition function.
We consider the Ising model, whose potentials have the form Φ ij ( x i , x j ) =
exp( J ij x i x j ) and Φ i ( x i ) =
exp( θ i x i ) with J ij ∈ R being the coupling strength of edge ( i, j ) and θ i ∈ R being the local ﬁeld of node i .
We further call an edge ( i, j ) attractive if J ij > 0 , and repulsive if J ij < 0 .
Then (1) takes the exponential form
[Equation]
with E ( x
[Equation]
(cid:80)
[Equation]
x i x
[Equation]
Ni =
θ i x i being the state energy.
Finally, following the terminology in Knoll et al.
[2021], we specify three types of models: unidirectional models do only contain attractive edges and all variables are biased towards the same state (i.e., either θ i ≤ 0 or θ i ≥ 0 ); attractive models do only contain attractive edges, but there may be local ﬁelds that differ in sign; general models may contain both attractive and repulsive edges.
Note that, by deﬁnition, attractive models include unidirectional models, while general models include both unidirectional and attractive models.
2.2 BELIEF PROPAGATION
In this work, we consider two problems: ﬁrst, the computation of marginal distributions where our speciﬁc interest lies in singleton marginals P X
i ( x i ) ; and second, the computation of the partition function.
It is well known that an exact computation of these quantities is intractable [Valiant, 1979,
Cooper, 1990] and even the approximation of the marginals to a certain precision is NP-hard [Dagum and Luby, 1993].
Belief propagation (BP) approximates the marginals by iteratively exchanging local statistical information between nodes in form of ’messages’.
This process is governed by the recursive message update equations
[Equation]
where the superscript ( n ) refers to the current iteration and the subscript ij refers to the direction in which a message is sent (e.g., node i sends µ ij to node j ).
In principle, one can approximate the singleton marginals at any iteration, by multiplying all incoming messages with the local potential:
[Equation]
Generally, however, marginal estimates are considered to be more accurate when they are obtained from a BP ﬁxed point [Murphy et al., 1999, Knoll et al., 2017]; more precisely, BP has converged to a ﬁxed point µ ◦ ij , whenever an update of the form (3) does not alter the message values anymore (that is µ ( n +1) ij = µ ( n ) ij for all ( i, j ) ).
often facilitate the theoretical analysis as it associates a unique parameter vector – consisting of all J ij and θ i – with each distribution (a so-called minimal representation, Wainwright et al.
Actually, these two problems are closely related as marginals can be expressed as a ratio of sub-partition functions [Weller and
Jebara, 2014b].
2.3 BETHE APPROXIMATION
It is often useful to formulate marginal inference in terms of a variational problem.
For this purpose, we consider some trial distribution Q X ( x ) and deﬁne the Gibbs free energy as
[Equation]
with U = E Q [ E ( X )] being the average energy of the model and S = − (cid:80) x ∈ X N Q X ( x ) log Q X ( x ) being the entropy of
Q X ( x ) .
Let us further deﬁne the marginal polytope M as the set of all valid probability distributions over X (i.e., that satisfy all global and local marginalization and normalization constraints).
If one minimizes F over M , then one recovers the true distribution P X ( x ) with the negative log-partition function − log( Z ) as the functional value at the global minimum, i.e., − log( Z ) =
min M
[Equation]
Two aspects, however, render the minimiziation of the Gibbs free energy intractable: ﬁrst, the deﬁnition of the marginal polytope by exponentially many constraints; second, the evaluation of the entropy that requires summing over exponentially many terms.
The Bethe free energy approximation addresses these two issues as follows:
ﬁrst, it relaxes the marginal polytope M to the local polytope L that involves only local marginalization and normalization constraints of the pairwise and singleton ’pseudo-marginals’ ˜ P ij and ˜ P i :
[Equation]
Second, it replaces the entropy S by an accordingly weighted sum of entropy contributions from edges and nodes.
More concretly, the Bethe free energy is deﬁned as F B =
U B − S B where the Bethe average energy is
[Equation]
and the Bethe entropy is
[Equation]
While U B equals the true average energy U in the exact marginals, S B is only an approximation to the true entropy S – unless the graph is a tree [Yedidia et al., 2005].
To obtain locally consistent approximations to the exact marginals, one usually aims to minimize F B over L .
Also, the so-called
Bethe partition function Z B , that is implicitely deﬁned by − log( Z B ) =
min L F B , provides an estimation of Z .
Binary variables allow for a particularly simple description of the local polytope.
Following the notation of Welling and Teh [2001], Weller and Jebara [2013], we deﬁne the pseudo-marginal distribution of X i by ˜ P
i ( X i = +1)
=
q i (implying ˜ P
i ( X i = − 1) = 1 − q i ) and, for any pair of connected nodes, we denote the joint pseudo-marginal probability ˜ P ij ( X i
=
+1 , X j = +1) by ξ ij .
Then the local marginalization and normalization constraints induce a full speciﬁcation of the joint probability table between X i and
X j in terms of the three parameters q i , q j , ξ ij (Tab. 1).
[Equation]
If we assume that all probabilities are strictly positive, then ξ ij is bounded by
[Equation]
Inserting singleton and pairwise pseudo-marginals from
Table 1 together with the Ising potentials from Sec.
into (7) and (8), the Bethe free energy becomes
[Equation]
where the pairwise entropies are
[Equation]
[Equation]
and the local entropies are
[Equation]
[Equation]
Then, with ( q ; ξ ) being the vector that contains all q i and ξ ij , the local polytope takes the simpliﬁed form
[Equation]
To further facilitate the task of minimizing F B over L ,
Welling and Teh [2001] have derived necessary conditions for points ( q ; ξ ) of L to be located at local minima of F B .
By setting the partial derivative ∂∂ξ ij F B for an arbitrary edge to zero, they proved that the resulting quadratic equation
[Equation]
where α ij = e 4 J ij − 1 , (15) has a unique valid (i.e., inside the bounds (9)) solution
[Equation]
This means that for each edge ( i, j ) , the only ξ ∗ ij ( q i , q j ) , that can be located at a stationary point of F B , depends directly on q i and q j and may therefore be inserted in the deﬁnition of F B (10).
This is advantageous for two reasons: ﬁrst, it considerably reduces the number of independent variables that are involved in optimizing F B (i.e., from | X | + | E | to | X | ); second, it simpliﬁes the shape of the domain, as F B is now deﬁned on a box-constrained domain, the Bethe box
[Equation]
In this work, we do always refer to the Bethe free energy by F
B , be it deﬁned over the local polytope or the Bethe box.
2.4 RELATED WORK
Belief propagation and the Bethe free energy.
Since the seminal work of Yedidia et al.
[2001], it is well known that ﬁxed points of BP correspond one-to-one to stationary points of the Bethe free energy; moreover, stable ﬁxed points of BP must always be associated to local minima of the Bethe free energy [Heskes, 2003].
Consequently, one can try to overcome the convergence issue of BP by minimizing the Bethe free directly.
To solve the problem, Welling and Teh [2001], Shin [2012] have derived gradient-based algorithms; Yuille [2002], Heskes [2006] have proposed provably convergent double-loop algorithms.
Variational free energy approximations.
Yedidia et al.
[2005] have shown that BP is only a special case of a general class of message passing algorithms, the generalized belief propagation (GBP).
Likewise, ﬁxed points of these algorithms correspond to stationary points of the so-called
Kikuchi free energies that try to approximate the true entropy by a sum over entropy contributions from larger node clusters [Kikuchi, 1951, Pelizzola, 2005].
In practice, many of these methods can be prohibitively slow and may suffer in the same way as BP from non-convexity of the particular free energy approximation; i.e., they may – if at all – converge to suboptimal minima.
This inspired various researchers to design free energy approximations that are convex [Wainwright et al., 2005, Globerson and Jaakkola, 2007b], some of which are related to convergent message passing algorithms [Kolmogorov and Wainwright, 2006,
Globerson and Jaakkola, 2007a, Hazan and Shashua, 2008,
Meltzer et al., 2009, Jancsary and Matz, 2011].
Theoretical work on the Bethe approximation.
The
Bethe approximation proves often to be superior to other methods in terms of a tradeoff between efﬁciency and accuracy
[Meshi et al., 2009].
Its theoretical properties have therefore been intensely studied: Heskes [2004], Pakzad and
Anantharam [2005] derived conditions for the convexity of the Bethe free energy.
Chertkov and Chernyak [2006] formulated the so-called loop series expansion that directly relates the Bethe partition function to the true partition function.
Others have found interesting connections between the
Bethe approximation and classical graph theory [Watanabe and Fukumizu, 2009, Vontobel, 2013].
Moreover, Weller and Jebara [2014a] derived an FPTAS 7 to approximate the
Bethe partition function in attractive models.
Graphical model approximation.
Another line of research, that is in some sense complementary to variational inference, tries to approximate the graphical model itself.
The classical Chow-Liu algorithm
[Chow and Liu, 1968] ﬁnds a spanning tree such that the Kullback-Leibler (KL) divergence between the original distribution and the induced tree distribution is minimal.
Furthermore, two different techniques have been applied to reduce the complexity of exact inference in a graphical model: ﬁrst, the ’annihilation’ of small probabilities that are below a certain treshold [Jensen and Andersen, 1990]; second, the deletion of one or more edges from the model (not necessarily until a spanning tree is reached).
Due to its empirical success, the second method deserves special attention: Kjaerulff [1994] carefully selected edges whose removal decreases the treewidth of a graph.
van Engelen [1997] studied how the removal of edges in a directed graph inﬂuences the KL divergence.
Choi and
Darwiche [2006] showed that a particular class of GBP, the so-called join graph propagation [Dechter et al., 2002], can be equivalently cast in terms of a procedure that consecutively deletes and recovers edges.
In the past, these methods were primarily applied to perform exact inference in the approximated model.
For large graphs, this does often remain a hard computational challenge.
THEORETICAL ANALYSIS
We shall now devote our attention to the central topic of this work: how removing edges from a graphical model inﬂuences the behavior of BP.
While the accuracy of the exact marginals degrades if one approximates a model by a sparser one [van Engelen, 1997], one might expect a similar behavior for the marginals estimated by BP.
We show, that the opposite is the case: sparsifying the graph does often signiﬁcantly improve the marginal accuracy of BP.
The quality of the estimated partition function, however, tends to degrade by deviating from the original model.
In this section, we explain the second of these phenomena theoretically.
We further analyze the role of an ’optimal’ edge to be removed and relate this problem to the Bethe free energy.
In particular, we prove an inherent relationship between global error measures on the Bethe free energy and the coupling strength of the edges.
Our detailed analysis of the Bethe free energy on a ’small scale’ extends the work of Welling and Teh [2001], Weller and Jebara [2013], Weller et al.
[2014] and leads to better understanding of BP and the
Bethe approximation in general.
3.1 PROBLEM SPECIFICATION
We brieﬂy clarify the problem to be considered.
Let ( G , Φ ) be a PGM and let ( G (cid:48) , Φ (cid:48) ) be a second PGM that is obtained by removing a set of edges ˜ E (and the associated pairwise potentials) from the original model.
8 Let P :=
{ P i : i ∈ X } be the set of exact (singleton) marginals on ( G , Φ ) and Z be the partition function.
Assume that we run BP on both models and obtain pseudo-marginals ˜ P :=
{ ˜ P i : i ∈
X } on ( G , Φ ) resp.
˜ P (cid:48) :
= { ˜ P (cid:48)
i : i ∈ X } on ( G (cid:48) , Φ (cid:48) ) , together with partition function estimates ˜
Z resp.
˜ Z (cid:48) .
Then we are interested in comparing the following quantities: ﬁrst, the l 1 -errors |
| P X − ˜ P X
|| l 1 and || P X − ˜ P (cid:48)
X
|| l 1 ; and second, the absolute errors | log Z − log
˜ Z | and | log Z − log ˜ Z (cid:48) | .
Ideally, we would like to remove a (possibly empty) set of edges, such that the induced errors || P X − ˜ P (cid:48)
X
|| l 1 and | log Z − log ˜ Z (cid:48) | become minimal over all subsets ˜ E ⊆ E .
That is, we want to ﬁnd the model for which BP best approximates the marginals and the partition function of the original model.
If one premises that, for comparison, we require the access to these exact quantities, the ﬁnding of such an edge set is of course an intractable problem.
Still, it remains a crucial question whether and to what extent the removal of edges has a positive impact on the estimates.
To identify edges to be deleted, we need to deﬁne an objective that contains information about the discrepancy between different graphical models.
Note that a global comparison via the KL divergence and its generalizations [Minka, 2005] is prohibitive as this would involve a summation over exponentially many terms.
Likewise, it is intractable to compare between different representations of the Gibbs free energy (5).
To relax the problem, we focus on the analysis of local discrepancies between two models.
The Bethe free energy (10) provides an ideal tool to explicitly measure these local differences.
3.2 THE BETHE ENERGY DIFFERENCE
Our main idea to make model comparison tractable lies in comparing between two different representations of the
Bethe free energy.
We formalize this concept as follows: assume for now that we remove a single edge ( i, j ) from a model ( G , Φ ) and let ( G \ ( i,j ) , Φ \ ( i,j ) denote the resulting model.
Let further F B resp.
F \ ( i,j ) B be the representations of the Bethe free energy that are associated with ( G , Φ ) resp.
( G \ ( i,j ) , Φ \ ( i,j ) .
Speciﬁcally, F \ ( i,j ) B does not contain the pairwise energy and entropy contributions from edge ( i, j ) , while the local entropy contributions from nodes i and j are counted once less than in the deﬁnition of F B (10).
Then we deﬁne the Bethe free energy difference ∆ F ( i,j ) B as the difference between F B and F \ ( i,j ) B , i.e.,
[Equation]
where ∆ U ( i,j ) B is the difference in the Bethe average energy and I ( i,j ) B is the mutual information between X i and X j .
Depending on whether we consider F B on the local polytope
L (13) or the Bethe box B (17), ∆ F ( i,j ) B is deﬁned on slices of these objects, that is either on the sliced local polytope
[Equation]
19) or the sliced Bethe box
B
[Equation]
It only depends on three resp.
two variables and may therefore be considered as a function that contains variational information about local changes in a model when removing an edge.
Moreover, it entails an effective way of measuring the local discrepancy between two graphical models, e.g., by computing an arbitrary norm of ∆ F ( i,j ) B on L ( i,j ) or B ( i,j ) .
In this work, we consider L p -norms as the most natural choice and analyze the special cases of p = ∞ and p = 2 in Sec.
3.3 (Theorem 1, Corollary 1, and Theorem 2).
In principle, one can generalize the above idea to compare between models that result from removing multiple edges ˜
E in one step, as the associated Bethe free energy difference ∆ F ˜ E B is then simply the sum over energy differences ∆ F ( i,j ) B for all ( i, j ) in ˜ E .
However, this increases both the number of variables to be integrated over and the number of edge sets to be taken into account for removal.
To facilitate the theoretical and experimental analysis of edge removal, we shall therefore focus on removing edges one by one.
To make statements about the global effects of removing individual edges on the Bethe free energy and BP, we must carefully analyze the functional behavior of the
Bethe free energy difference and its components on a small scale.
In the following, we derive a series of auxiliary theorems where we consider the mathematical properties of ξ ∗ ij from (16) (Lemma 1 and 2), the Bethe mutual information
I ( i,j ) B (Lemma 3), and the Bethe energy difference ∆ F ( i,j ) B (Lemma 4, 5, and 6) on their joint domain, the sliced Bethe box B ( i,j ) .
These results – besides being interesting in themselves – are rather of technical nature and will help us in proving our main results in Sec. 3.3.
All proofs for Sec. 3.
and 3.3 are contained in the Appendix A.
First, we compute values of ξ ∗ ij in the center point of the sliced Bethe box:
Lemma 1.
Let ( i, j ) be an edge.
In the center point (0 . 5 , 0 . 5) of the sliced Bethe box B ( i,j ) , the unique ξ ∗ ij that can be located at a stationary point of F B has the form
[Equation]
We will also have to analyze the behavior of ξ ∗ ij if q i and q j approach the boundary ∂ B ( i,j ) 10 of the sliced Bethe box:
Lemma 2.
Let ( i, j ) be edge and let k ∈ [0 , 1] .
The limits of ξ ∗ ij at the boundary ∂ B ( i,j ) of the sliced Bethe box are
[Equation]
Moreover, we shall prepare bounds and compute the boundary limits at ∂ B ( i,j ) of the mutual information
I ( i,j ) B (
[Equation]
Lemma 3.
Let ( i, j ) be an edge.
(a) In the interior of the sliced Bethe box B ( i,j ) , the mutual information I ( i,j ) B is bounded by
[Equation]
(b) The limit of I ( i,j ) B at the boundary ∂ B ( i,j ) is
[Equation]
Next, we compute ﬁrst-order and second-order derivatives of ∆ F ( i,j ) B on B ( i,j ) .
The proof utilizes results from Welling and Teh [2001], Weller and Jebara [2013] (Appendix B).
Lemma 4.
Let ( i, j ) be an edge.
(a) The ﬁrst-order derivatives of ∆ F ( i,j ) B on B ( i,j ) are
[Equation]
(b) The second-order derivatives of ∆ F ( i,j ) B on B ( i,j ) are
[Equation]
where T ij :=
q i q j (1 − q
i
[Equation]
(1 − q j ) − ( ξ ∗ ij − q i q j
[Equation] [Equation]
The following result formulates a useful property of the
Bethe free energy difference on the sliced Bethe box:
Lemma 5.
Let ( i, j ) be an edge.
∆ F ( i,j ) B has precisely one stationary point on B ( i,j ) , which is (¯ q i , ¯ q j ) =
(0 . 5 , 0 . 5) and is neither a maximum nor a minimum (i.e., a saddle point).
Lemma 5 implies that ∆ F ( i,j ) B cannot possess a maximum nor a minimum in the interior of B ( i,j ) .
This implies that the supremum and inﬁmum of ∆ F ( i,j ) B must lie at the boundary.
Finally, we characterize regions of B ( i,j ) on which ∆ F ( i,j ) B contributes always negatively to the Bethe free energy F B :
Lemma 6.
Let ( i, j ) be an edge.
(a) For an attractive edge, ∆ F ( i,j ) B ( q i , q j ) is negative if either both q i , q i < 0 .
5 or both q i , q i > 0 .
(b) For a repulsive edge, ∆ F ( i,j ) B ( q i , q j ) is negative if either q i < 0 .
5 and q j > 0 .
5 or q i < 0 .
5 and q j > 0 .
3.3 MAIN RESULTS
After having prepared the technical framework in Sec. 3.2, we now proceed by presenting our main results.
First, we directly relate the Bethe free energy difference to the local properties of the graphical model (Theorem 1).
Then, we address the problem of an ’Bethe-optimal’ edge to be deleted (Corollary 1 and Theorem 2).
Finally, we conclude about the approximation quality of BP regarding the estimated partition function if edges are deleted (Theorems 3 and 4).
Theorem 1.
Let ( i, j ) be an arbitrary edge.
Then the L ∞ - norm of the Bethe free energy difference is
[Equation]
with
[Equation]
The inﬁmum and supremum are not taken by ∆ F ( i,j ) B but exist only as limits at the boundary of B ( i,j ) .
In particular, we have for an attractive edge
[Equation]
and, conversely, for a repulsive edge
[Equation]
Theorem 1 reveals a monotonic dependence between the strength of the couplings and absolute changes in the Bethe free energy that are caused by local modiﬁcations in the graphical structure.
In terms of edge deletion, this implies that the ’Bethe-optimal’ choice of an edge to be removed from the graph is the one with the weakest coupling strength:
Corollary 1.
Suppose we aim to remove an edge from the graphical model such that the induced maximum error in the
Bethe free energy is minimal.
Then this ’ L ∞ -Bethe-optimal’ edge is the one with the lowest absolute coupling strength:
[Equation]
An analogous property holds for the L 2 -error of F B on L :
Theorem 2.
Suppose we aim to remove an edge from the graphical model such that the induced mean squared error in the Bethe free energy on the local polytope L is minimal.
Then this ’ L 2 -Bethe-optimal’ edge is the one with the lowest absolute coupling strength:
[Equation]
Next, we conclude about the quantitative change in the
Bethe partition function Z B if an edge is removed:
Theorem 3.
Let Z B be the Bethe partition function associated with some graphical model, i.e., the quanitity that satisﬁes − log( Z B ) =
min B F B .
Suppose we remove an (attractive or repulsive) edge from the graph.
Let F \ ( i,j ) B be the representation of the Bethe free energy associated with the new model, together with the new Bethe partition function Z B \ ( i,j ) that is implicitly deﬁned by − log( Z B \ ( i,j ) )
=
min B F \ ( i,
[Equation]
B .
Then the following error estimate holds:
[Equation]
Finally, we conclude about the quality of the estimated partition function if edges are removed.
We consider unidirectional models (Sec. 2.1) that allow for a precise statement:
Theorem 4.
Consider a unidirectional model, i.e., where all edges are attractive and all variables are biased towards the same state.
Let Z resp.
Z B be the associated partition resp.
Bethe partition function.
Suppose we remove an arbitrary edge from the graph and let Z B \ ( i,j ) be the Bethe partition function associated with the new model.
Then the quality of the estimated partition function degrades, i.e.,
[Equation]
Theorem 4 does not formally extend to models that contain both positive and negative local ﬁelds.
Generally, however, the error between the true and the BP-estimated partition function tends to increase, the more edges we remove.
This negative result is contrasted by the positive effect of edge removal on the estimated marginals.
While existing theoretical bounds on the marginal errors are often loose and typically hard to compute [Wainwright et al., 2003, Taga and Mase, 2006, Ihler, 2007, Mooij and Kappen, 2008], we validate and explain our statement in Sec. 4.
EXPERIMENTS
We now demonstrate empirically how removing edges can have an astonishingly positive impact on the approximation accuracy of BP.
We perform a range of experiments on a fully connected graph on 10 vertices.
11 Further experiments including a 5 × 5 - grid graph are contained in Appendix C.
We consider both attractive and general models (Sec. 2.1).
In Sec. 4.1, we focus on attractive models and sample J ij uniformly from [0 , ˆ J ] for ˆ J ∈ { 0 . 1 , 0 . 2 , . . .
In Sec. 4.2, we focus on general models and sample J ij uniformly from [ − ˆ J, ˆ J ] for ˆ J ∈ { 0 . 1 , 0 . 2 , . . .
For both settings, we create two scenarios: ﬁrst, models with weak local ﬁelds (each θ i is sampled uniformly from [ − 0 .
2 , 0 . 2] ); second, models with strong local ﬁelds (each θ i is sampled uniformly from [ − 0 .
For each conﬁguration, we create 200 models.
For each individual model, we remove edges one by one until we reach a spanning tree.
We do not remove edges, whose deletion makes the graph disconnected.
12 We compare two criteria for selecting the next edge to be removed:
ﬁrst, the
Bethe-optimal criterion (Corollary 1, Theorem 2); second, we remove edges that induce the lowest mutual information between two connected variables in the original model.
More precisely: assume that we have already removed edge set ˜ E from a model; then the next edge ( i, j ) to be removed is the one that minimizes either of the following criteria:
BETHE-OPT :
[Equation]
Note that by applying the second criterion, we end up in a
Chow-Liu tree [Chow and Liu, 1968], i.e., the spanning tree with the lowest KL divergence from the original model.
For each intermediate model during the edge deletion process, we run BP 100 times with random message initialization to approximate the marginals.
For each run, we perform at most 1000 iterations.
If BP has not converged, we estimate the marginals from the ﬁnal iteration.
We utilize a randomized message scheduling to achieve better convergence [Elidan et al., 2006].
For the error evaluation, we compute the l 1 -distance between the exact and estimated marginals.
The results for each model are averaged over the 100 runs.
Finally, the results are averaged over all 200 models, each based on a different conﬁguration of the potentials.
4.1 ATTRACTIVE MODELS
For weak couplings, BP ﬁnds accurate marginal estimates in the original model.
If the strength of the couplings increases, this favorable property suddenly disappears at some critical treshold and BP fails to approximate the marginals for larger values of ˆ J (Fig. 1).
This behavior is not due to worse convergence properties of BP, but results from inaccurate
BP ﬁxed points and thus inaccurate minima of the Bethe free energy [Weller et al., 2014].
While the Bethe free energy is convex for weaker couplings and possesses a unique global minimum, this minimum becomes an (unstable) saddle point if ˆ J increases and cannot be reached by BP any longer [Heskes, 2003, Mooij and Kappen, 2005, Knoll and
Pernkopf, 2017].
For even larger couplings, the landscape of the Bethe free energy becomes increasingly complex and the (possibly many 14 ) Bethe minima approach the boundary of the domain, thus moving away from the exact marginals.
If we remove edges from the graph, the marginal accuracy of BP in the new model is often much better than in the original model.
This can be explained by a ’reconvexiﬁcation’ of the Bethe free energy that makes unstable saddle points or maxima stable minima again and allows BP to converge to accurate ﬁxed points.
The question on how many edges we should actually remove, is a difﬁcult one.
In Fig.
1, we observe that there appears to be a ’channel’ that deﬁnes an optimal number of edges to be removed.
The stronger the couplings become, the more preferable is it to rely on tree approximations, while BP outperforms the edge removal techniques for regimes with lower coupling strength (Fig. 2).
For stronger local potentials, the channel becomes narrower and edge removal loses some of its beneﬁt (although the results are mostly superior in comparison to the original model).
Also, in Fig.
1 we observe that BETHE-OPT performs slightly better than CHOW-LIU criterion, with an increasing advantage for stronger local potentials.
4.2 GENERAL MODELS
The situation for general models is similar as in the attractice case.
We can observe certain differences though (Fig. 3): ﬁrst, the critical treshold of the couplings, beyond which the
Bethe free energy becomes non-convex, is higher than for attractive models.
Second, for models with strong local potentials, edge removal based on the BETHE-OPT criterion improves the marginal accuracy only slightly.
Interestingly, the Chow-Liu tree induces strikingly accurate Bethe minima for all models (Fig. 4).
As in the attractive case, we observe that the problem becomes more difﬁcult if both the pairwise and the local potentials become stronger at the same time.
CONCLUSION
We have proposed to approximate a graphical model as a ’preprocessing step’ for approximate inference.
We focused
ˆ J r e m o v e d e d g e s θ ∈ [ − 0 .
BETHE-OPT 0 .0 .0 .0 .
8 l(a) 0 .
ˆ J r e m o v e d e d g e s θ ∈ [ − 0 .
CHOW-LIU 0 .0 .0 .0 .
8 l(b) 0 .
ˆ J r e m o v e d e d g e s θ ∈ [ − 0 .
BETHE-OPT 0 .0 .0 .0 .
8 l(c) 0 .
ˆ J r e m o v e d e d g e s θ ∈ [ − 0 .
CHOW-LIU 0 .0 .0 .0 .
8 l(d)
[Figure] [Figure]
[Figure]
J l 1 e rr o r
s
i n g .
m a r g .
Tree
[Figure] [Figure]
ˆ J l 1 e rr o r
s
i n g .
m
[Figure]
on the removal of single edges and showed that this can have a beneﬁcial impact on the behavior of belief propagation.
We have exploited the relationship between belief propagation and the Bethe free energy to explain the success of such an approach.
Subsequently, we have validated our ﬁndings in an experimental study.
Most importantly, our analysis contributes to an improved understanding of belief propagation and the Bethe approximation in general.
We are convinced that our observations inspire the development of further sophisticated methods that try to approximate a graphical model and improve the behavior of message passing algorithms.
We believe that one logical extension lies in the modiﬁcation of the local potentials to compensate for the lost information caused by edge removal
[Figure] [Figure]
Acknowledgements
This work was supported by the Graz University of Technology LEAD project “Dependable Internet of Things in
Adverse Environments”.
We further acknowledge support from the Wireless Lab, Huawei Technologies Sweden
[Figure]