NeuroBE: Escalating NN Approximations to Bucket Elimination
Abstract
A major limiting factor in graphical model inference is the complexity of computing the partition function.
Exact message-passing algorithms such as Bucket Elimination (BE) require exponentially high levels of memory to compute the partition function, therefore approximations are necessary.
In this paper, we build upon a recently introduced methodology called Deep Bucket Elimination (DBE) that uses classical Neural Networks (NNs) to approximate messages gener- ated by BE when buckets have large memory requirements.
The main feature of our new scheme called NeuroBE is that it customizes the architecture and learning of the NNs to the message size and its distribution.
We also explore a new loss function for training taking into account the estimated message cost distribution.
Our experiments demonstrate that these enhancements provide significant improvements over
DBE in both time and accuracy.
We also study the impact of the messages local errors on the global accuracy of the estimate of the partition function.
Introduction
Two of the critical goals of probabilistic modeling are the compact representation of probability distributions and the efficient computation of their marginals and modes.
Probabilistic graphical models, such as Markov networks (Pearl 1988; Darwiche 2009; Dechter 2013) provide a framework to represent distributions compactly as normalized products or factors : P ( X ) = 1 Z (cid:81)
α f α ( X α ) , where X is a set of variables, each potential f
α is a function over a subset X α of the variables (its scope) and Z = (cid:80) X (cid:81) α f α
( X α ) is the partition function .
Computing the partition function, or performing inference, is still exponential in the induced width of the model’s graph even for distributions that admit a compact representation.
The partition function Z is defined by two types of operations: sums and products.
It can be evaluated efficiently if (cid:80)
[Equation]
f
[Equation]
( X α ) can be reorganized using the distributive law along a variable ordering (Dechter 2003).
This organization can be described using buckets as data structures, one for each variable in the ordering.
When a bucket is processed, its associated variable is removed by creating a bucket output function, also called a message , that is passed to a subsequent bucket.
The complexity of computing this function is exponential in its number of arguments, called scope or the bucket’s width.
Overall, Bucket Elimination (BE) (Dechter 1999a), is time and memory exponential in the induced-width of the model’s graph along the ordering.
A common approach for approximating BE is to approximate each bucket message with a surrogate fnction whenever it cannot be computed exactly, that is, when its width is too high.
Such schemes that bound the time and space complexity of BE include the (weighted) mini-bucket scheme (Dechter and Rish 2003; Liu and Ihler 2012) and generalized belief propagation schemes (Yedidia, Freeman, and Weiss 2000;
Mateescu et al. 2010).
Our recent approach, Deep Bucket
Elimination (DBE) (Razeghi et al. 2021), approximates each bucket function with a neural network (NN).
While this approach is inherently time consuming requiring the independent training of many NNs to compute the partition function of a single problem, it has yielded more accurate approximations on several benchmarks.
Unlike its main competition of weighted bucket-elimination it can improve with time even with bounded memory, providing a potential anytime framework for reasoning.
Yet, DBE ’s original design can be improved significantly as we show in this paper.
Contributions.
We present NeuroBE , a re-design of
DBE , that addresses it’s
one size fits all policy by customizing the NN construction and training sample size to each bucket separately, in proportion to it’s message size.
We also consider a new loss function for training that takes into account the distribution of messages.
We compare NeuroBE with DBE and with other approximation schemes of BE .
Lastly, we provide an analysis of the global error associated with the estimated partition function, relating it to the local errors associated with individual messages.
The paper is organized as follows.
We first provide a background to BE and DBE ; then we present NeuroBE followed by error analysis; lastly, we demonstrate the efficiency of NeuroBE empirically.
Related work.
As noted, approximating and bounding
Bucket Elimination has been carried out extensively over the years for all probabilistic queries.
Well known is the
Mini-Bucket Elimination scheme (Dechter and Rish 2003) and its variants, such as Weighted Mini-Bucket Elimination
(WMBE) , augmented with message-passing cost-shifting (Liu and Ihler 2011).
Neural network approximation to BE was introduced in (Razeghi et al. 2021).
The idea is closest in spirit to the
Neuro-Dynamic Programming scheme as outlined in (Bertsekas and Tsitsiklis 1996) where the cost-to-go functions (similar to messages) generated by dynamic programming can be approximated by NNs.
This is also highly related to Deep Reinforcement Learning (DRL) (Mnih et al. 2015) where, in the absence of a model, the value function is approximated by neural networks learned from temporal trajectories.
Recently, Graph Neural Networks (GNNs) (Scarselli et al.
2009) are used to learn messages following the messagepassing reasoning methods in graphical models (Abboud,
Ceylan, and Lukasiewicz 2020; Yoon et al.
Heess, Tarlow, and Winn 2013).
However, (Yoon et al. 2018;
Heess,
Tarlow, and Winn 2013) is restricted to small instances (i.e., ∼ 40 variables) and (Abboud, Ceylan, and Lukasiewicz 2020) tackles problems with a known polynomial-time approximation.
GNN based methods derive a supervised endto-end learning algorithm which generalize across different problem instances.
In contrast, we consider a different class of algorithms, where we confine learning to within a problem instance only .
Background
A graphical model can be defined by a 3-tuple M =
( X , D , F ) ,
where X =
X i : i ∈ V, V = { 1 , ..., n }} is a set of n variables indexed by V and D
=
{ D i : i ∈ V } is the set of finite domains for each X
i (i.e. each X i can only assume values in D i , and each D i is finite).
Each function f α ∈ F is defined over a subset of the variables called its scope, X α , where α ⊆ V are the indices of variables in its scope and D α denotes the Cartesian product of their domains, so that f α : D α
→ R ≥ 0 .
The primal graph of a graphical model associates each variable with a node.
An edge between node i and node j is created if and only if there is a function containing
X i and X j in its scope.
Figure 1a shows a primal graph of a graphical model with variables indexed from A to G with functions over pairs of variables that are connected by an edge.
Graphical models can be used to represent a global function, often a probability distribution, defined by
Pr ( X ) ∝ (cid:81) α f α
( X α ) .
An important task is to compute the normalizing constant, also known as the partition function
Z = (cid:80) X (cid:81)
α f α ( X α ) .
Bucket Elimination
Bucket Elimination (BE) (Dechter 1999b) is a universal exact algorithm for probabilistic inference.
It is a variable elimination algorithm that can answer a wide-range of queries, including the partition function ranging from constraint satisfaction, to pure combinatorial optimization (e.g., Most
Probable Explanation (MPE/MAP)), and weighted counting (Partition Function, Probability of Evidence).
Given a variable ordering d , BE (presented in Algorithm 1, omitting steps 9-12) creates a bucket tree where each node is
[Figure]
a bucket representing a variable in the ordering d .
Figure 1b shows a bucket tree for the primal graph in Figure 1a along an ordering.
Each bucket in this tree contains a set of the model’s functions depending on the given order of processing.
For example, Bucket G in Figure 1b has functions { f ( A, G ) , f ( F, G ) } , an exhaustive set of model’s functions with variable G in its scope.
There is an arc from a bucket, say B c , to a parent bucket, B p , if X p is the latest variable in bucket B c ’s message scope along the ordering (constants are placed in B 1 ).
In the same example, there is an arc from
Bucket G to Bucket F.
BE then, performs inference along the bucket tree as a 1-iteration message-passing algorithm (bottom-up).
It processes each bucket from leaves to the root passing messages from child ( c ) to parent ( p ).
For a child variable X c , it considers all the functions in bucket B c .
This includes the original functions in the graphical model as well as the messages received by processing previous variables.
It then marginalizes X c out from the product of functions in B c generating a new, so called, bucket function or message, denoted λ c → p , or λ c for short:
[Equation]
The λ c function is placed in B p , the bucket of X p .
Once all the variables are processed, BE outputs all the messages and the exact value of Z by taking the product of all the constant present in the bucket of the first variable.
We illustrate
BE message flow on our example problem in Figure 1b.
Complexity.
Both the time and space complexity of BE are exponential in the induced width , which is the size of the largest number of variables in the scope of any message in a graph (Dechter 2013).
Clearly, BE becomes impractical if the induced width is large.
Deep Bucket Elimination
Given a variable ordering d , Deep Bucket Elimination (DBE) (Razeghi et al. 2021), approximates each message generated in the bucket tree by training a NN when the scope of any bucket message ( S c ) is high ( > i -bound).
For example, in figure 1b, if we use an i -bound = 2 , instead of sending an exact function from the bucket of D to the bucket of C , λ D → C ( A, B, C ) , DBE sends a NN approximation µ θ,D → C ( A, B, C ) with parameters θ , as we describe next.
Let B c be a bucket with width w c > i -bound and output message λ c ( S c ) with scope S c .
DBE then, constructs a fully-connected feed-forward NN having w c nodes in the input layer.
This is followed by L hidden layers with a constant h hidden nodes per layer with ReLU activation function.
Finally, the output layer contains one node with a realvalued output.
Subsequently, DBE generates a training set { ( s n , λ c ( s n )) } of size N, where s n denotes a configuration over S c sampled uniformly at random and λ c ( s n ) is the message value defined in Eq.
The NN parameters, θ , are then trained to minimize the mean square error loss:
[Equation]
where s n is the n th sample in the training set and µ θ,c ( s n ) is the NN output.
Once training is complete, DBE passes the trained NN µ θ,c ( s n ) to its parent bucket.
Typically, during its course, DBE may approximate many bucket messages in order to compute the partition function Z .
However, it often requires a large training sample size per message to achieve better performance than its competitors.
This leads to a substantial increase in the algorithm’s time and memory requirements.
Therefore, to make DBE efficient, we re-design the message approximation procedure, elaborated in the following section.
NeuroBE
We rename DBE to NeuroBE acknowledging the use of shallow neural networks (2 layers) to approximate each message.
Algorithm 1 presents NeuroBE .
Similar to DBE ,
NeuroBE first creates a bucket tree along a given ordering in line 2.
While processing each bucket along the ordering, if it’s width ≤ i − bound, then the message, µ ∗ c → p , is computed exactly in line 7.
Otherwise, the message is approximated using a NN in line 10.
Note that in either case, if a bucket contains a NN function, then computing µ ∗ in Line 7 or estimating it as described in NN-train function (Algorithm 2) requires evaluating the trained NN. Finally, Line 14 calculates the partition function using the functions in bucket B 0 . Note that we denote µ ∗ c → p as the exact message computed in a bucket while reserve the notation λ c → p to the messages computed by exact BE . We do this to distinguish the exact local computation of a message that may be based on inexact functions in the bucket from the globally exact messages
Input : Graphical model M =
( X , D , F ) , Ordering d =
X 1 , ..., X n
Parameters : i -bound i , #layers L , constants b, η
Output : the partition function constant and bucket messages 1: for c in n...1 do 2: (initialize buckets and form a bucket-tree) put all unplaced functions mentioning X c in B c .
3: end for 4: for c in n...1 do 5:
Let X p be the parent variable of X c in the bucket-tree 6: if width ( B c ) ≤ i then 7: µ ∗ c → p ← (cid:80) X c (cid:81) f α ∈ B c f α 8: else 9: (denote by µ ∗ c → p the function (cid:80)
X
c (cid:81) f
α ∈ B c f α ) 10: µ θ c ,c
→ p ← NN-train ( µ ∗ c → p , L, b, η ) 11: end if 12:
Put µ ∗ c → p or µ θ c ,c → p in B p 13: end for 14: Z = (cid:80) X 0 (cid:81)
f α ∈ B 0 f α 15: return Z and All messages generated λ computed by BE .
In the latter, each bucket function is computed exactly and also all the functions in a bucket are exact.
Hence, we refer to µ ∗ c as the local exact message .
Further, we denote µ θ c ,c → p the NN approximations of the local exact message, µ ∗ c → p .
The difference between NeuroBE and DBE is solely in the individual message approximation scheme, NNtrain .
As noted before, DBE often uses a constant, large sized training set for each message approximation.
A simple brute-force reduction of the sample size only to reduce training time, may lead to overfitting .
Hence NeuroBE customizes the NN architecture and its training set size to the message complexity (see (Vapnik 1999)).
NN Architecture selection
It is obvious that the NN size should be dependant on the dimensionality of the message function.
In our case, the function’s scope size is the induced-width, w .
We propose to adjust the NN size by making the number of hidden units, h , a function of w while keeping the number of layers, L , constant.
We select h = b · w , where b is a constant satisfying b ≥ 1 .
Figure 2 illustrates an example NN model with an input layer of size w and 2 hidden layers with dimension h, varying linearly with b .
When b = 1 all layers are of the same size w .
Through such a rule, NeuroBE fits NN to message size.
We now quantify the capacity of such NNs and apply it to determine it’s train sample sizes.
NN complexity.
The pseudo-dimension (Pollard 1984;
Anthony and Bartlett 2002) is used to estimate the expressive power or complexity of NNs in regression problems.
Bounds to the pseudo-dimension of NN architectures with
ReLU activation functions is provided by (Bartlett et al.
We use the lower bound as a proxy to estimate the pseudo-dimension ( ρ c ) of the NN approximating a message µ ∗ c having width w c , yielding (see appendix for derivation):
[Equation]
We use the above equation as corelated with the complexity of the candidate NN approximating a message µ ∗ c .
Sample Complexity As suggested in (Vapnik 1999).
we choose a sample size ( N ) that is proportional to the pseudodimension (Eq. 2) of each NN:
[Equation]
where η is a constant, and w is the number of arguments to the estimated function. We will use N ( w ) to emphasize that N varies with w .
The sample size N ( w ) often exceed memory limits for higher width buckets with even the simplest NN architecture ( L = 1 , b = 1 ).
Hence, we threshold sample size per bucket to 1000k.
In general, for high induced-width problems, we keep η small to favour training NNs with small sample sizes.
However, for problems with small induced-width, we let η take high values.
Learning NNs
For training a NN, DBE generated samples in a bucket uniformly at random and used the mean square error as the loss function.
We wondered whether generating the samples in a non-uniform distribution, in a way that is related to the message distribution, would be more effective.
Alternatively, we asked if a loss function that is dependant on the actual message values should be used.
To that end we define the distribution, F c ( S ) , of an output message µ ∗ c ( S ) by: [Equation] where the denominator is summed over each configuration s from the set of all possible configurations D ( S ) over scope
Algorithm Input : µ ∗ c message on a set of variables X Parameters : L : # layers in NN, # epochs , η , b : constants Output : µ θ c ,c : NN message approximation, ˆ ϵ : an estimated bucket error bound, ˆ ϵ avg : estimated average bucket error 1: w c ← scope( µ ∗ c ) 2: # h ← b ∗ w c 3:
Initialize NN µ θ c ,c with L layers, # h hidden-units 4: N ← sample-size( w c , η, L, b ) 5: trainSet, valSet, testSet ← generate-samples( µ ∗ c , N ) 6: p =1, error val = + ∞ , µ θ ← µ θ c ,c 7: while p ≤ # epochs and ¬ early stopping(error val) do 8: µ θ c ,c ← µ θ 9: for train i in batches(trainSet) do 10: µ θ train i ←
NN(train i , µ θ ) 11: loss ← w.m.s.e(train i , µ θ train i ) 12: µ θ ← optimize(Adam, loss, θ ) 13: end for 14: µ θ val ← NN(valSet, µ θ ) 15: error val ← w.m.s.e(valSet, µ θ val) 16: p ← p +17: end while 18: error ← log(testSet \ NN(testSet, µ θ c ,c )) 19:
ˆ ϵ c ← max(error), ˆ ϵ avgc ← avg(error) 20: return µ θ c ,c , ˆ ϵ c , ˆ ϵ avgc
S .
However, since sampling from F c ( S ) is hard we sampled from the uniform distribution, but changed our loss to a weighted estimate of the mean square error (w.m.s.e):
[Equation]
s ∈ D ( S ) ( µ
[Equation]
[Equation] c ( [Equation] where c is an index to the bucket, s is a uniformly sampled configuration in scope S c and µ θ c ( s ) is the approximated message value computed by the NN; ˆ F ( s ) is the probability of s following the message distribution (eq 4), and ˆ U ( s ) is the probability of s according to the uniform distribution.
Algorithm 2 describes this learning procedure.
For a bucket B c of variable X c having local exact output message µ ∗ c ; Algorithm NN-train, using its input parameters L, b, η constructs a NN with L layers and b · w c hidden units (line 3). It then determines it’s training sample size N (line 4) using Eq. 3.
A major step occurs next (line 5) where it creates the training, validation and testing sets by generating samples uniformly from the domain of the function scope S as follows.
For each sample configuration s , and a variable X c , we compute it’s target value µ ∗ ( s ) , as:
[Equation]
If the bucket B c contains a trained NN, then this step requires evaluating that NN.
Lines 9-12 then trains the NN parameters by dividing the train set into batches.
Lineshows the function NN(data, µ θ ) as the output computed by the NN µ θ c on input set data.
Line 11 computes the w.m.s.e between the target values in the train set and the NN output, µ θ train.
The NN parameters ( θ c ) are then updated using the Adam optimizer (Kingma and Ba 2014) (line 12).
After each epoch, the current trained model is evaluated on a holdout validation set (line 14-15).
We stop training when either the maximum limit # epochs is reached or the validation error meets our early stopping criteria, that is if the validation error increases for two consecutive epochs.
Once training is complete, we compute the maximum and average log relative errors between the target and NN approximated messages over a test set (lines 18-19).
In the next section, we use this to analyse error propagation in NeuroBE .
The NN - train procedure then returns the approximated message µ θ c ,c along with it’s estimated error.
Complexity.
Clearly, the time and space complexity for learning a single message in NeuroBE is linear in the sample size.
In contrast to DBE , here the sample size is customized to the message complexity, or bucket width.
Error Analysis
We next analyse the relationship between the local errors contributed by each approximated message and the global partition function error.
Definition 1 (local and global bucket errors) .
Let λ c be the (global) exact message generated in B c by the exact BE algorithm, µ ∗ c be the local exact message in B c computed by the functions in it and µ c = NN -train( µ ∗ c ) be its NN approximation.
Then, the Local Bucket Error is the function
[Equation]
The Global Bucket Error is
[Equation]
The above error corresponds to a log of the relative errors.
We use log relative error here because bounding the global error as a function of the local errors turned out to be easier.
Note that the true partition function, Z ∗ =
λ 1 .
So, the global bucket error G 1 is the error in the estimated partition function, thus measuring NeuroBE ’s performance.
Theorem 1.
Assume a bucket-chain along an ordering d and let B c be a bucket along the chain.
Let E c ( x ) =
lnµ ∗ c ( x ) − lnµ c ( x ) as defined above and let ϵ c = max s ∈ D ( S c )
| E c ( s ) | , where S c is the scope of outgoing message from B c and D ( S c ) is the set of all possible configurations on S c .
Then,
[Equation]
In particular, since λ 1 = Z , the partition function
[Equation]
For the proof see the Appendix.
Calculating e c is hard because it involves computing the local bucket error E c over all configurations in the scope of the bucket.
Therefore, we calculate the maximum error over a sampled test set in steps 18-19 of algorithm 2 as ˆ ϵ c .
Additionally, we calculate the average local bucket error, ˆ ϵ avgc over the same test set.
According to Eq. 6, summing over all bucket error bounds, ˆ ϵ c , bounds the global error of the estimated partition function.
Clearly, this bound is very lose.
We therefore also use the average local bucket error, ˆ ϵ avg to give us some additional information on the global error empirically.
In the next section we evaluate NeuroBE and provide some information on local vs global errors.
Experiments Experiment Setup
We ran experiments comparing NeuroBE against the
Weighted Mini Bucket Elimination scheme ( WMBE ) (Dechter and Rish 2003; Liu and Ihler 2012) and DBE (Razeghi et al. 2021).
Following the methodology in DBE , we evaluated NeuroBE on instances selected from three well-known benchmarks from the UAI repository used in (Kask et al. 2020), i.e. grids (vision domain), pedigree (genetic linkage analysis) and DBNs.
We targeted diverse benchmarks (in structure and level of determinism) and aimed for different levels of hardness.
Thus, in each benchmark, we distinguish between problems that can be solved exactly, which we call ”easy”, and those that cannot be solved, called ”hard”.
We also distinguish benchmarks that possess determinism , namely have a high proportion of zero probabilities, a feature which can impact training.
We randomly selected 14 instances from Grids, with easy ones (i.e., width 20-30) and hard ones (i.e., 1600 variables, width 55),from pedigrees, which posses high level of determinism and 6 from DBNs, totalling 25 instances.
To trigger bucket message approximations, we used i -bound= 10 for easy problems and at most i -bound= 20 for hard ones.
For problems with determinism, such as pedigree, the structure of the NN in NeuroBE is the same as that of a
MaskedNet in DBE ((Razeghi et al. 2021)), varying only the number of hidden units per layer.
NNs architecture Here, we show how the NN architecture and sample size is tuned across the different benchmarks.
We keep the # layers fixed (=2) across all benchmarks.
We then pick a random problem instance from each benchmark to fix the hyper-parameters regulating the NN architectures and sample sizes.
For each problem instance with it’s width w ∗ , we first keep h = w and the number of samples corresponding to w ∗ 2 roughly around 300 k for hard problems and 100 k for easy problems as a heuristic and derive a value for η from equation 3.
Keeping η fixed and then again, varying h ∈ [ w, 5 w ] , we pick the value of h which gives the smallest average error to estimate the partition function for each representative problem instance.
We then use this configuration for varying NN architecture and sample size for the rest of the problem instances in that benchmark.
In particular, we selected h = 3 w and N avg ∈ [149 k, 350 k ] for pedigrees; h
= { 3 w, 5 w } and N avg ∈ [80 k, 180 k ] for DBN; h = w and N avg ∈
[12 k, 121 k ] for grid-easy; and h = w and N avg ∈ [60 k, 209 k ] for grid-hard.
[Figure]
Training NNs.
Bucket output messages can either have very small values (eg. exp ( − 11) ) as in the pedigree benchmarks (and possess determinism) or can be very large
(eg. exp (51) )
as for grids and DBNs.
To handle large values in messages of non-deterministic benchmark domains, we use log transformations to handle overflow issues.
In addition, we normalize the input and output values for NNs across benchmarks to be in [ − 1 , 1] and [0 , 1] respectively to accelerate training (Cun, Kanter, and Solla 1991)).
As per algorithm 2, we create the training set of size N ( w c )
(Eq 3), validation set of size N ( w c ), and test set of size 50 k .
We then train the network using the Adam optimizer with a learning rate of 0 .and
a batch-size of 256 across all benchmarks.
Performance measures We evaluate the performance of
NeuroBE using: error = | log e Z − log e ˆ Z | where ˆ Z is the generated estimate of the partition function, Z .
When the exact Z is not available (for hard Grid benchmark), Z ∗ is a surrogate to Z , which is obtained using an advanced sampling scheme for a duration of 100 ∗ 1 hr (Kask et al. 2020).
Results
Figure 3 compares NeuroBE against DBE and WMB over the 3 benchmarks.
We report the results over 5 runs for each instance of both NeuroBE and DBE due to stochasticity of their behavior.
The first few columns show the problem statistics for instances in the respective benchmarks (pedigree, DBN and grids).
We then show wmb error.
For NeuroBE , we report the average and maximum #training samples, ( N max , N avg ) and maximum #hidden units, ( h max ) across all buckets of the instance.
We also report the average time (in hours), the average error, minimum error and standard deviation over the 5 runs.
Pedigrees We see a consistent decrease in both average error and standard deviation for the partition function estimates with NeuroBE when compared to DBE , being ≥ 5 times more accurate than DBE for 5 out of the 6 instances.
It achieves this better estimates with less time, since it uses far less training samples.
Also NeuroBE outperforms WMB on 5 instances ( ≥ 5 times more accurate for 4 instances).
Here DBE yields either a similar accuracy as
WMB or even a worse one (instances 3,5,6).
Grids.
Here too we observe that NeuroBE outperforms
DBE in accuracy as reflected by the average error and standard deviation, even though it uses far less time.
In most cases we see a reduction in time by a factor of 2 or more (IDs 1,3,4,5 from grid-easy and IDs 1,2,3,4,6,7 from grid-hard) still producing a far better estimate.
For 2 instances, however, (easy #2, hard #5)
we see slightly worse performance than DBE .
NeuroBE and DBE outperform
WMB across all problem instances.
DBN We report results for the DBN benhcmark for 2 ibounds.
For i -bound=20, NeuroBE achieves a higher accuracy than DBE with far less time (instances 3,5,6).
It is superior to wmb on instances 2,3.
However, WMB performs better on instance 1,4 & 6, as the induced-width is closer to the i -bound and is comparable to DBE in accuracy, yet it takes half of the time.
For i -bound=10, NeuroBE shows better accuracy than WMB for all three instances.
It outperforms DBE on instances 2,3.
Overall, compared with DBE , NeuroBE is faster ongrid + DBN instances by a factor of 2; 6 times more accurate on 4 easy grid instances; more accurate on 5 grid-hard instances by a factor of 2 and 5 times more accurate onpedigree instances.
The impact of loss functions.
Overall, on most of the instances of Grids and DBNs we did not observe a significant impact of weighted loss function on the quality of training (results are not explicitly shown).
Yet, for pedigree instances we did observe a significant impact.
We suspect this for pedigrees because the NN outputs are messages (i.e., they do not undergo log transformation) and the weights in the w.m.s.e reflect the message distribution.
Figure 4 compares the w.m.s.e and m.s.e loss for pedigrees.
We observe that with w.m.s.e loss, NeuroBE shows better performance on all instances (for instance 2, training with w.m.s.e is 20 times more accurate).
How local errors impacts the global error.
Figure 5 reports some statistics on local bucket errors (Def. 1) and test w.m.s.e across buckets for 4 Grid problem instances; We also show the global errors estimated by Eq. 6; and the empirical global errors averaged over 5 runs.
We report the results for 2 sets of average sample size N avg = { 60 k, 150 k } .
We observe a consistent decrease in the local bucket and global errors when more samples are given.
However, instance 4 shows an increase in the empirical global error, which we account to stochastic behaviour.
We also observe a direct correlation between the estimated local bucket errors and empirical average global errors across instances.
For example, the average local bucket error for instanceis 1 .
67 with an empirical global error of 24 . 01 ; instanceshows an average local bucket error of 0 .
784 with 21 .as
the empirical global error.
Note that both errors simultaneously decrease for instance 2.
This trend continues for instance 3, where the local bucket error is 0 .
336 while the global error is 5 .
We observe such a correlation of the empirical average global errors on the estimated local bucket errors across all instances, which is affirmative.
Time & accuracy.
Figure 6 shows the expected relationship between time and accuracy on a few problem instances.
We depict the average error of our partition function estimate [Figure] (what we call global error), and standard deviation (overruns) and the computation time for 2 different NN architecture and their associated sample sizes.
As expected we see that increasing the NN and sample sizes increases the time and accuracy for pedigrees.
For grid-hard instances, we just increased the sample sizes for the same architecture where h = w , and observe that the average error is reduced by about a factor of 3.
Instances from grid-easy and DBN show a similar improvement in performance with a larger NN and a corresponding larger training sample size.
This shows that the algorithm has an anytime characteristics, as it can improve its performance by controlling the size of the approximating NN matched by a suitable sample size for training
Conclusion & Future Work
In this work we advance our earlier theme of using the power of Neural networks to approximate the class of bucketelimination algorithms that is at the heart to probabilistic reasoning.
The central aim of our paper is to improve the efficiency of such schemes by enhancing its NN training aspect.
NeuroBE ’s main new design feature is that it customizes the
NN architectures and the training samples to the messages, thus achieving higher accuracy often with less time when compared to DBE .
We presented NeuroBE and illustrated, on challenging instances over three benchmarks that it can be far more accurate and requires less time compared with its earlier version of Deep Bucket Elimination DBE and is also superior to weighted mini-bucket WMB that cannot improve its accuracy once their memory is exhausted.
Future Work.
While NeuroBE adjusts the NN architectures according to a bucket’s scope, it keeps the # layers fixed and seeks for user inputs for such variations.
We will explore how to fine-tune NN capacity dynamically during training.
Since the number of buckets trained have a direct effect on NeuroBE ’s accuracy and time performance, we will explore reducing the number of trained functions by training a single function per union of buckets, which yield a cluster in a tree-decomposition (Dechter 2013).
This can significantly reduce the number of trained functions at the cost of more time for sample generation, a trade-off we plan to study.
We will also explore the task of parameter sharing and thus, training of multiple bucket functions simultaneously.
Appendix Estimating the pseudo-dimension of a NN:
In our work, we use NN architectures with ReLU activation functions.
To construct a NN with L layers and a variable #hidden-units per layer to model a specific bucket message λ c , we pick the rule h
= b ∗ w c where w c is the width and b is a constant.
By doing this, the # parameters in the NN is :
[Equation]
We make use of the lower bound of pseudo-dimension for NNs with ReLU activation functions from the work in (Bartlett et al. 2019) to get:
[Equation]
By substituting (7) in (8) and ignoring all linear in w c terms we get that ρ c ( w c ) can be dominated by:
[Equation]
Estimating error in partition
function:
Theorem.
1 Let B c be a bucket in a bucket chain along an ordering d ; let B c contain the original functions as ϕ c and µ c +1 as the message passed to it from the previous bucket; let λ c be the (global) exact message generated in B c , µ ∗ c be the local exact message in B c and µ c = APP ( µ ∗ c ) its approximation (e.g., by a trained neural network).
Let E c
=
lnµ ∗ c − lnµ c
and and ϵ c = max B c | E c | .
Then,
[Equation]
In particular, since λ 1 = Z , the partition function
[Equation]
Proof.
We will next derive the recursion, starting at the first processed bucket B n and going down in order.
Remember throughout that ln µ ∗ n − i =
(cid:80) X n − i ln
( e lnϕ n − i + lnµ n − i +1 )
For B n λ n = µ ∗ n , therefore
[Equation]
For B n − 1 , by definition
[Equation]
Substituting lnλ n from B n = ln (cid:88)
[Equation]
n
[Equation]
If max scope ( µ ∗ n ) | E n | = ϵ n , then, ≤ ln [ e ϵ n (cid:88)
[Equation]
≤
[Equation]
Since ln (cid:80) X n − 1 e lnϕ n − 1 + lnµ n = lnµ ∗ n − 1 we get lnλ n
[Equation]
− lnµ
[Equation]
≤ ϵ
[Equation]
+
[Equation]
∗
[Equation]
−
[Equation]
n
[Equation]
(10) or equivalently,
[Equation]
Moving to B n − 2 , by definition:
[Equation]
Substituting lnλ n − 1 from Eq.
(10) we get lnλ
[Equation]
(cid:88)
[Equation]
Taking max scope (
[Equation]
=
[Equation] [Equation]
yielding,
[Equation]
Moving to bucket B n − 3 , by definition
[Equation]
Substituting for λ n − 2 from Eq.
(13) we get with some algebra
[Equation]
yielding
[Equation]
and so on.
Clearly the emerging expression for bucket B c is
[Equation]
or,
[Equation]
The general transition from n − i to n − i − 1 can be easily followed to complete the inductive proof.
Assuming that we control the derivation of µ c for each B c to ensure that E c = lnµ ∗ c − lnµ c ≤ ϵ c and substituting in the expression we get from Eq.
(15) that
[Equation]