Original: NeuroBE: Escalating Neural Network Approximations of Bucket Elimination
Revised: NeuroBE: Escalating Neural Network Approximations of Bucket Elimination

Original: Abstract
Revised: Abstract

Original: A major limiting factor in graphical model inference is the complexity of computing the partition function.
Revised: A major limiting factor in graphical model inference is the complexity of computing the partition function.

Original: Exact message-passing algorithms such as Bucket Elimination (BE) require exponential memory to compute the partition function; therefore, approximations are necessary.
Revised: Exact message-passing algorithms such as Bucket Elimination (BE) require exponential memory to compute the partition function; therefore, approximations are necessary.

Original: In this paper, we build upon a recently introduced methodology called Deep Bucket Elimination (DBE) that uses classical Neural Networks to approximate messages generated by BE for large buckets.
Revised: In this paper, we build upon a recently introduced methodology called Deep Bucket Elimination (DBE) that uses classical Neural Networks to approximate messages generated by BE for large buckets.

Original: The main feature of our new scheme, renamed NeuroBE , is that it customizes the architecture of the neural networks, their learning process and in particular, adapts the loss function to the internal form or distribution of messages.
Revised: The main feature of our new scheme, renamed NeuroBE , is that it customizes the architecture of the neural networks, their learning process and in particular, adapts the loss function to the internal form or distribution of messages.

Original: Our experiments demonstrate significant improvements in accuracy and time compared with the earlier DBE scheme.
Revised: Our experiments demonstrate significant improvements in accuracy and time compared with the earlier DBE scheme.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Two of the critical goals of probabilistic modeling are the compact representation of probability distributions and the efficient computation of their marginals and modes.
Revised: Two of the critical goals of probabilistic modeling are the compact representation of probability distributions and the efficient computation of their marginals and modes.

Original: Probabilistic graphical models, such as Markov networks [Pearl, 1988, Darwiche, 2009, Dechter, 2013] provide a framework to represent distributions compactly as normalized products or factors : P ( X ) = 1 Z (cid:81)
Revised: Probabilistic graphical models, such as Markov networks [Pearl, 1988, Darwiche, 2009, Dechter, 2013] provide a framework to represent distributions compactly as normalized products or factors : P ( X ) = 1 Z (cid:81)

Original: α f α ( X α ) , where X is a set of variables, each potential f
Revised: α f α ( X α ) , where X is a set of variables, each potential f

Original: α is a function over a subset X α of the variables (its scope) and Z = (cid:80) X (cid:81) α f α
Revised: α is a function over a subset X α of the variables (its scope) and Z = (cid:80) X (cid:81) α f α

Original: ( X α ) is the partition function .
Revised: ( X α ) is the partition function .

Original: Computing the partition function is exponential in the induced width of the model’s graph even for distributions that admit a compact representation.
Revised: Computing the partition function is exponential in the induced width of the model’s graph even for distributions that admit a compact representation.

Original: The partition function Z is defined by two types of operations: sums and products.
Revised: The partition function Z is defined by two types of operations: sums and products.

Original: It can be evaluated efficiently if (cid:80) X (cid:81)
Revised: It can be evaluated efficiently if (cid:80) X (cid:81)

Original: α f α ( X α ) can be reorganized using the distributive law along a variable ordering [Dechter, 2003].
Revised: α f α ( X α ) can be reorganized using the distributive law along a variable ordering [Dechter, 2003].

Original: This organization can be described using buckets as data structures, one for each variable in the ordering.
Revised: This organization can be described using buckets as data structures, one for each variable in the ordering.

Original: When a bucket is processed, its associated variable is removed, creating a bucket output function, also called a message , that is passed to a subsequent bucket.
Revised: When a bucket is processed, its associated variable is removed, creating a bucket output function, also called a message , that is passed to a subsequent bucket.

Original: The time and space complexity of computing this function is exponential in its number of arguments, called scope or the bucket’s width.
Revised: The time and space complexity of computing this function is exponential in its number of arguments, called scope or the bucket’s width.

Original: Overall, Bucket
Revised: Overall, Bucket

Original: Elimination ( BE ) [Dechter, 1999b] is time and memory exponential in the induced-width of the model’s graph along the ordering.
Revised: Elimination ( BE ) [Dechter, 1999b] is time and memory exponential in the induced-width of the model’s graph along the ordering.

Original: Providing good approximations to BE is important not only because it generates an answer to a query, but primarily because it compiles a structure and a set of messages that can be used to answer multiple queries (e.g., the probability of evidence for various evidence variables Darwiche [2009]).
Revised: Providing good approximations to BE is important not only because it generates an answer to a query, but primarily because it compiles a structure and a set of messages that can be used to answer multiple queries (e.g., the probability of evidence for various evidence variables Darwiche [2009]).

Original: Also, the messages can be used as building blocks for generating heuristics for search to further improve performance.
Revised: Also, the messages can be used as building blocks for generating heuristics for search to further improve performance.

Original: We therefore consider and evaluate NeuroBE in the context of approximate BE , generating approximation to its messages.
Revised: We therefore consider and evaluate NeuroBE in the context of approximate BE , generating approximation to its messages.

Original: Schemes that approximate BE include (weighted) minibucket ( WMB ) [Dechter and Rish, 2003, Liu and Ihler, 2012] and generalized belief propagation schemes [Yedidia et al., 2000, Mateescu et al., 2010].
Revised: Schemes that approximate BE include (weighted) minibucket ( WMB ) [Dechter and Rish, 2003, Liu and Ihler, 2012] and generalized belief propagation schemes [Yedidia et al., 2000, Mateescu et al., 2010].

Original: A recently introduced scheme,
Revised: A recently introduced scheme,

Original: Deep Bucket Elimination (DBE) [Razeghi et al., 2021] approximates each bucket function with a neural network (NN).
Revised: Deep Bucket Elimination (DBE) [Razeghi et al., 2021] approximates each bucket function with a neural network (NN).

Original: While this approach is inherently time consuming, requiring the independent training of many NNs to solve the partition function of a single problem, it has yielded more accurate approximations on several benchmarks when compared against competing schemes.
Revised: While this approach is inherently time consuming, requiring the independent training of many NNs to solve the partition function of a single problem, it has yielded more accurate approximations on several benchmarks when compared against competing schemes.

Original: Both WMB and DBE are restricted by memory.
Revised: Both WMB and DBE are restricted by memory.

Original: Yet the memory demanded by WMB (notwithstanding recent work [Forouzan and Ihler, 2015]) increases exponentially with its i -bound not accommodating refined steps of memory increase.
Revised: Yet the memory demanded by WMB (notwithstanding recent work [Forouzan and Ihler, 2015]) increases exponentially with its i -bound not accommodating refined steps of memory increase.

Original: In contrast, NN architectures can grow more gradually and may facilitate a more flexible memory-accuracy balance.
Revised: In contrast, NN architectures can grow more gradually and may facilitate a more flexible memory-accuracy balance.

Original: Contributions.
Revised: Contributions.

Original: We present NeuroBE , a re-design of DBE , that addresses the shortcomings of its one size fits all policy by customizing the NN construction and training sample size to each bucket separately, in proportion to its message size.
Revised: We present NeuroBE , a re-design of DBE , that addresses the shortcomings of its one size fits all policy by customizing the NN construction and training sample size to each bucket separately, in proportion to its message size.

Original: We also introduce a new loss function that is sensitive to a bucket’s message distribution, also called local structure .
Revised: We also introduce a new loss function that is sensitive to a bucket’s message distribution, also called local structure .

Original: We also provide an analysis relating the local errors to an upper bound on the global error.
Revised: We also provide an analysis relating the local errors to an upper bound on the global error.

Original: In an extensive empirical evaluation we show that NeuroBE outperforms DBE across all benchmarks using far less resources, such as training samples and NN size, while yielding higher accuracy with less time.
Revised: In an extensive empirical evaluation we show that NeuroBE outperforms DBE across all benchmarks using far less resources, such as training samples and NN size, while yielding higher accuracy with less time.

Original: We provided the source code to reproduce the results of this paper at https://github.com/dechterlab/NeuroBE.
Revised: We provided the source code to reproduce the results of this paper at https://github.com/dechterlab/NeuroBE.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: We first provide a background to BE , WMB and DBE ; then we present NeuroBE ; followed by error analysis; lastly, we demonstrate the efficiency of NeuroBE empirically.
Revised: We first provide a background to BE , WMB and DBE ; then we present NeuroBE ; followed by error analysis; lastly, we demonstrate the efficiency of NeuroBE empirically.

Original: Related work.
Revised: Related work.

Original: As noted, approximating and bounding
Revised: As noted, approximating and bounding

Original: Bucket Elimination has been carried out extensively over the years for all probabilistic queries.
Revised: Bucket Elimination has been carried out extensively over the years for all probabilistic queries.

Original: Well known is the
Revised: Well known is the

Original: Mini-Bucket Elimination scheme [Dechter and Rish, 2003] and its variants, such as Weighted Mini-Bucket Elimination (WMB) , augmented with message-passing cost-shifting [Liu and Ihler, 2011b].
Revised: Mini-Bucket Elimination scheme [Dechter and Rish, 2003] and its variants, such as Weighted Mini-Bucket Elimination (WMB) , augmented with message-passing cost-shifting [Liu and Ihler, 2011b].

Original: Neural network approximation to BE was introduced in
Revised: Neural network approximation to BE was introduced in

Original: Razeghi et al.
Revised: Razeghi et al.

Original: The idea is closest in spirit to the
Revised: The idea is closest in spirit to the

Original: Neuro-Dynamic Programming scheme as outlined in Bertsekas and Tsitsiklis [1996] where the cost-to-go functions (similar to messages) generated by dynamic programming can be approximated by NNs.
Revised: Neuro-Dynamic Programming scheme as outlined in Bertsekas and Tsitsiklis [1996] where the cost-to-go functions (similar to messages) generated by dynamic programming can be approximated by NNs.

Original: This technique is also highly related to Deep Reinforcement Learning (DRL)
Revised: This technique is also highly related to Deep Reinforcement Learning (DRL)

Original: [Mnih et al., 2015] where, in the absence of a model, the value function is approximated by NNs learned from temporal trajectories.
Revised: [Mnih et al., 2015] where, in the absence of a model, the value function is approximated by NNs learned from temporal trajectories.

Original: Recently, Graph Neural Networks (GNNs)
Revised: Recently, Graph Neural Networks (GNNs)

Original: [Scarselli et al., 2009] are used to learn messages following the messagepassing reasoning methods in graphical models [Abboud et al., 2020, Yoon et al., 2018, Heess et al., 2013].
Revised: [Scarselli et al., 2009] are used to learn messages following the messagepassing reasoning methods in graphical models [Abboud et al., 2020, Yoon et al., 2018, Heess et al., 2013].

Original: However,
Revised: However,

Original: Yoon et al.
Revised: Yoon et al.

Original: [2018], Heess et al.
Revised: [2018], Heess et al.

Original: [2013] is restricted to small instances (i.e., ∼ 40 variables) and Abboud et al.
Revised: [2013] is restricted to small instances (i.e., ∼ 40 variables) and Abboud et al.

Original: [2020] tackles problems with a known polynomial-time approximation.
Revised: [2020] tackles problems with a known polynomial-time approximation.

Original: GNN based methods derive a supervised end-to-end learning algorithm, generalizing across different problem instances.
Revised: GNN based methods derive a supervised end-to-end learning algorithm, generalizing across different problem instances.

Original: In contrast, we consider a different class of algorithms, where we confine learning to within a problem instance only .
Revised: In contrast, we consider a different class of algorithms, where we confine learning to within a problem instance only .

Original:  BACKGROUND
Revised:  BACKGROUND

Original: A graphical model can be defined by a 3-tuple M =
Revised: A graphical model can be defined by a 3-tuple M =

Original: ( X , D , F ) ,
Revised: ( X , D , F ) ,

Original: where X =
Revised: where X =

Original: X i : i ∈ V, V = { 1 , ..., n }} is a set of n variables indexed by V , and D
Revised: X i : i ∈ V, V = { 1 , ..., n }} is a set of n variables indexed by V , and D

Original: =
Revised: =

Original: { D i : i ∈ V } is the set of finite domains for each X
Revised: { D i : i ∈ V } is the set of finite domains for each X

Original: i (i.e. each X i can only
Revised: i (i.e. each X i can only

Original: A"
Revised: A"

Original: B"
Revised: B"

Original: C" D"
Revised: C" D"

Original: E"
Revised: E"

Original: F"
Revised: F"

Original: G" (
Revised: G" (

Original: [Figure]
Revised: [Figure]

Original: primal Graph
Revised: primal Graph

Original: [Figure]
Revised: [Figure]

Original: assume values in D i , and each D i is finite).
Revised: assume values in D i , and each D i is finite).

Original: Each function f α ∈ F is defined over a subset of the variables called its scope, X α , where α ⊆ V are the indices of variables in its scope, and D α denotes the Cartesian product of their domains so that f α : D α
Revised: Each function f α ∈ F is defined over a subset of the variables called its scope, X α , where α ⊆ V are the indices of variables in its scope, and D α denotes the Cartesian product of their domains so that f α : D α

Original: → R ≥ 0 .
Revised: → R ≥ 0 .

Original: The primal graph of a graphical model associates each variable with a node.
Revised: The primal graph of a graphical model associates each variable with a node.

Original: An edge between node i and node j is created if and only if there is a function containing X i and
Revised: An edge between node i and node j is created if and only if there is a function containing X i and

Original: X j in its scope.
Revised: X j in its scope.

Original: Figure 1a shows a primal graph of a graphical model with variables indexed from A to G and functions over pairs of variables are connected by an edge.
Revised: Figure 1a shows a primal graph of a graphical model with variables indexed from A to G and functions over pairs of variables are connected by an edge.

Original: Graphical models can be used to represent a global function, often a probability distribution, defined by Pr ( X ) ∝ (cid:81)
Revised: Graphical models can be used to represent a global function, often a probability distribution, defined by Pr ( X ) ∝ (cid:81)

Original: α f α ( X α ) .
Revised: α f α ( X α ) .

Original: An important task is to compute the normalizing constant, also known as the partition function Z = (cid:80) X
Revised: An important task is to compute the normalizing constant, also known as the partition function Z = (cid:80) X

Original: (cid:81)
Revised: (cid:81)

Original: α f α ( X α ) .
Revised: α f α ( X α ) .

Original: 2.1 BUCKET ELIMINATION
Revised: 2.1 BUCKET ELIMINATION

Original: Bucket Elimination (BE) [Dechter, 1999a] is a universal exact algorithm for probabilistic inference.
Revised: Bucket Elimination (BE) [Dechter, 1999a] is a universal exact algorithm for probabilistic inference.

Original: It is a variable elimination algorithm that can answer a wide-range of queries, including the partition function ranging from constraint satisfaction, to pure combinatorial optimization (e.g., Most
Revised: It is a variable elimination algorithm that can answer a wide-range of queries, including the partition function ranging from constraint satisfaction, to pure combinatorial optimization (e.g., Most

Original: Probable Explanation (MPE/MAP)), and weighted counting (Partition Function, Probability of Evidence).
Revised: Probable Explanation (MPE/MAP)), and weighted counting (Partition Function, Probability of Evidence).

Original: Given a variable ordering d , BE (presented in Algorithm 1, omitting steps 9-12) creates a bucket tree where each node is a bucket representing a variable in the ordering d .
Revised: Given a variable ordering d , BE (presented in Algorithm 1, omitting steps 9-12) creates a bucket tree where each node is a bucket representing a variable in the ordering d .

Original: Figure 1b shows a bucket tree for the primal graph in Figure
Revised: Figure 1b shows a bucket tree for the primal graph in Figure

Original: 1a along an ordering.
Revised: 1a along an ordering.

Original: Each bucket in this tree contains a set of the model’s functions depending on the given order of processing.
Revised: Each bucket in this tree contains a set of the model’s functions depending on the given order of processing.

Original: For example, Bucket G in Figure 1b has functions { f ( A, G ) , f ( F, G ) } , an exhaustive set of model’s functions with variable G in its scope.
Revised: For example, Bucket G in Figure 1b has functions { f ( A, G ) , f ( F, G ) } , an exhaustive set of model’s functions with variable G in its scope.

Original: There is an arc from a bucket, say B c , to a parent bucket, B p , if X p is the latest variable in bucket B c ’s message scope along the ordering (constants are placed in B 1 ).
Revised: There is an arc from a bucket, say B c , to a parent bucket, B p , if X p is the latest variable in bucket B c ’s message scope along the ordering (constants are placed in B 1 ).

Original: In the same example, there is an arc from Bucket G to Bucket F.
Revised: In the same example, there is an arc from Bucket G to Bucket F.

Original: BE performs inference along the bucket tree as a 1-iteration message-passing algorithm (bottom-up).
Revised: BE performs inference along the bucket tree as a 1-iteration message-passing algorithm (bottom-up).

Original: It processes each bucket from leaves to the root passing messages from child ( c ) to parent ( p ).
Revised: It processes each bucket from leaves to the root passing messages from child ( c ) to parent ( p ).

Original: For a child variable X c , BE encompasses all the functions in bucket B c .
Revised: For a child variable X c , BE encompasses all the functions in bucket B c .

Original: This includes the original functions in the graphical model as well as the messages received by processing previous variables.
Revised: This includes the original functions in the graphical model as well as the messages received by processing previous variables.

Original: It then marginalizes X c out from the product of functions in B c generating a new, so called, bucket function or message, denoted λ c → p , or λ c for short:
Revised: It then marginalizes X c out from the product of functions in B c generating a new, so called, bucket function or message, denoted λ c → p , or λ c for short:

Original: [Equation]
Revised: [Equation]

Original: The λ c function is placed in B p , the bucket of X p .
Revised: The λ c function is placed in B p , the bucket of X p .

Original: Once all the variables are processed, BE outputs all the messages and the exact value of Z by taking the product of all the constants present in the bucket of the first variable.
Revised: Once all the variables are processed, BE outputs all the messages and the exact value of Z by taking the product of all the constants present in the bucket of the first variable.

Original: We illustrate BE message flow in our example problem in Figure 1b.
Revised: We illustrate BE message flow in our example problem in Figure 1b.

Original: Complexity.
Revised: Complexity.

Original: Both the time and space complexity of BE are exponential in the induced width , which is the size of the largest number of variables in the scope of any message over all buckets
Revised: Both the time and space complexity of BE are exponential in the induced width , which is the size of the largest number of variables in the scope of any message over all buckets

Original: [Dechter, 2013].
Revised: [Dechter, 2013].

Original: Clearly, BE becomes impractical if the induced width is large.
Revised: Clearly, BE becomes impractical if the induced width is large.

Original: 2.2 WEIGHTED MINI-BUCKET
Revised: 2.2 WEIGHTED MINI-BUCKET

Original: Given a variable ordering d , Weighted Mini-Bucket ( WMB ) [Dechter and Rish, 2003, Liu and Ihler, 2011b] approximates BE by partitioning each bucket B c with high width into several disjoint “mini-buckets” B jc to ensure that individual B jc has low ( ≤ i − bound) width.
Revised: Given a variable ordering d , Weighted Mini-Bucket ( WMB ) [Dechter and Rish, 2003, Liu and Ihler, 2011b] approximates BE by partitioning each bucket B c with high width into several disjoint “mini-buckets” B jc to ensure that individual B jc has low ( ≤ i − bound) width.

Original: The method also assigns a weight p cj to each mini-bucket B jc .
Revised: The method also assigns a weight p cj to each mini-bucket B jc .

Original: WMB then eliminates the bucket’s variable X in the j th mini-bucket
Revised: WMB then eliminates the bucket’s variable X in the j th mini-bucket

Original: B jc using the power sum following Holder’s inequality [Liu and Ihler, 2011a]:
Revised: B jc using the power sum following Holder’s inequality [Liu and Ihler, 2011a]:

Original: [Equation]
Revised: [Equation]

Original: and µ jc is passed to a parent bucket B p .
Revised: and µ jc is passed to a parent bucket B p .

Original: For example, using an i -bound = 2 in Figure 1b, WMB approximates the exact message λ D → C (A, B, C), passed from bucket D to bucket
Revised: For example, using an i -bound = 2 in Figure 1b, WMB approximates the exact message λ D → C (A, B, C), passed from bucket D to bucket

Original: C , by three messages corresponding to partitioning bucket
Revised: C , by three messages corresponding to partitioning bucket

Original: B D into three mini-buckets each with a single function f ( A, D ) , f ( B, D ) , f ( C, D ) .
Revised: B D into three mini-buckets each with a single function f ( A, D ) , f ( B, D ) , f ( C, D ) .

Original: Based on Holder’s inequality [Liu and Ihler, 2011a], the exact message is bounded by the product of the mini-bucket messages when the weights p cj ’s are non-negative and sum to one.
Revised: Based on Holder’s inequality [Liu and Ihler, 2011a], the exact message is bounded by the product of the mini-bucket messages when the weights p cj ’s are non-negative and sum to one.

Original: Thus, for any i -bound
Revised: Thus, for any i -bound

Original: WMB generates an upper bound of the partition function.
Revised: WMB generates an upper bound of the partition function.

Original: Generally, time and accuracy in WMB increases with the i -bound.
Revised: Generally, time and accuracy in WMB increases with the i -bound.

Original: Yet, due to memory constraints it can run with a maximum i -bound of about 20 and therefore, the generated bounds can be extremely loose when a problem’s inducedwidth is high.
Revised: Yet, due to memory constraints it can run with a maximum i -bound of about 20 and therefore, the generated bounds can be extremely loose when a problem’s inducedwidth is high.

Original: Interestingly, when it is run, WMB terminates quickly, taking a few seconds and up to a minute.
Revised: Interestingly, when it is run, WMB terminates quickly, taking a few seconds and up to a minute.

Original: 2.3 DEEP BUCKET ELIMINATION
Revised: 2.3 DEEP BUCKET ELIMINATION

Original: Given a variable ordering d , Deep Bucket Elimination (DBE) approximates each message generated in the bucket tree whenever the scope ( S ) of a message is high ( > i -bound) using a neural network (NN).
Revised: Given a variable ordering d , Deep Bucket Elimination (DBE) approximates each message generated in the bucket tree whenever the scope ( S ) of a message is high ( > i -bound) using a neural network (NN).

Original: Following the previous example of i -bound = 2 in Figure 1b, rather than sending the exact message from bucket D to bucket C , DBE sends a NN µ θ,D → C ( A, B, C ) parameterized by θ that approximates the exact message λ D → C ( A, B, C ) , as we elaborate next.
Revised: Following the previous example of i -bound = 2 in Figure 1b, rather than sending the exact message from bucket D to bucket C , DBE sends a NN µ θ,D → C ( A, B, C ) parameterized by θ that approximates the exact message λ D → C ( A, B, C ) , as we elaborate next.

Original: We use µ ∗ c → p to denote the local exact message computed using all functions in bucket c , regardless of the local functions being exact or approximate (as defined by the right side of Eq.
Revised: We use µ ∗ c → p to denote the local exact message computed using all functions in bucket c , regardless of the local functions being exact or approximate (as defined by the right side of Eq.

Original: However, if we execute exact BE , in which case the bucket contains exact messages only, we denote the output message as λ c → p and refer to it as the global exact message .
Revised: However, if we execute exact BE , in which case the bucket contains exact messages only, we denote the output message as λ c → p and refer to it as the global exact message .

Original: Let B be a bucket with width w > i -bound and µ ∗ ( S ) be its local exact message having scope S whose size is the bucket’s width, w .
Revised: Let B be a bucket with width w > i -bound and µ ∗ ( S ) be its local exact message having scope S whose size is the bucket’s width, w .

Original: DBE constructs a fully-connected feedforward NN having w nodes in the input layer, followed by
Revised: DBE constructs a fully-connected feedforward NN having w nodes in the input layer, followed by

Original: L hidden layers each having h hidden nodes with ReLU activation function.
Revised: L hidden layers each having h hidden nodes with ReLU activation function.

Original: The output layer contains one node with a real-valued output.
Revised: The output layer contains one node with a real-valued output.

Original: Subsequently, DBE generates a training set { ( s n , µ ∗ ( s n )) } of size N, where s n is the n th configuration of S , sampled uniformly at random and where µ ∗ ( s n ) is the local exact message value defined in Eq.
Revised: Subsequently, DBE generates a training set { ( s n , µ ∗ ( s n )) } of size N, where s n is the n th configuration of S , sampled uniformly at random and where µ ∗ ( s n ) is the local exact message value defined in Eq.

Original: The NN function µ θ ( S ) approximating µ ∗ ( S ) is trained to minimize the mean square error loss :
Revised: The NN function µ θ ( S ) approximating µ ∗ ( S ) is trained to minimize the mean square error loss :

Original: [Equation]
Revised: [Equation]

Original: Once training is complete, DBE passes the trained NN, µ θ to its parent bucket.
Revised: Once training is complete, DBE passes the trained NN, µ θ to its parent bucket.

Original: While DBE showed superior quality of solutions compared with WMB , its time performance was quite inferior.
Revised: While DBE showed superior quality of solutions compared with WMB , its time performance was quite inferior.

Original: In particular, training each bucket message used the same fixed architecture and the same sample size (quite large), needlessly resulting in a high total time.
Revised: In particular, training each bucket message used the same fixed architecture and the same sample size (quite large), needlessly resulting in a high total time.

Original: This paper is devoted to a redesign of the algorithm, aiming to improve both time and accuracy, as we elaborate in the following section.
Revised: This paper is devoted to a redesign of the algorithm, aiming to improve both time and accuracy, as we elaborate in the following section.

Original:  NEUROBE
Revised:  NEUROBE

Original: Algorithm NeuroBE advances DBE , and is described in this section, focused on the partition function task.
Revised: Algorithm NeuroBE advances DBE , and is described in this section, focused on the partition function task.

Original: However, extension to other queries is straightforward, and requires altering only the message definition in Eq.
Revised: However, extension to other queries is straightforward, and requires altering only the message definition in Eq.

Original: (1) to fit the corresponding task (e.g., replacing summation by maximization in Eq.
Revised: (1) to fit the corresponding task (e.g., replacing summation by maximization in Eq.

Original: (1) for the MAP query.)
Revised: (1) for the MAP query.)

Original: We rename DBE to NeuroBE since we use mostly shallow neural networks (up to 2 layers).
Revised: We rename DBE to NeuroBE since we use mostly shallow neural networks (up to 2 layers).

Original: Algorithm 1 describes
Revised: Algorithm 1 describes

Original: NeuroBE .
Revised: NeuroBE .

Original: The algorithm first creates a bucket tree along a given ordering (line 2).
Revised: The algorithm first creates a bucket tree along a given ordering (line 2).

Original: It then processes buckets one by one along the ordering from last to first.
Revised: It then processes buckets one by one along the ordering from last to first.

Original: If the current processed bucket has width w ≤ i -bound, then the message, µ ∗ c → p is computed exactly (line 7).
Revised: If the current processed bucket has width w ≤ i -bound, then the message, µ ∗ c → p is computed exactly (line 7).

Original: Otherwise, the bucket’s message is approximated by a neural net (line 9).
Revised: Otherwise, the bucket’s message is approximated by a neural net (line 9).

Original: The message is placed in the appropriate parent bucket in the bucket tree.
Revised: The message is placed in the appropriate parent bucket in the bucket tree.

Original: Finally, line 13 calculates the partition function using the functions in bucket B 1 .
Revised: Finally, line 13 calculates the partition function using the functions in bucket B 1 .

Original: Note that if a bucket contains a NN function, then computing µ ∗ (line 7 or 9) requires evaluating the trained NN (see Algorithm 3, line 4).
Revised: Note that if a bucket contains a NN function, then computing µ ∗ (line 7 or 9) requires evaluating the trained NN (see Algorithm 3, line 4).

Original: The difference between NeuroBE and DBE is solely in the individual message approximation scheme, NN-train .
Revised: The difference between NeuroBE and DBE is solely in the individual message approximation scheme, NN-train .

Original: In contrast to DBE , NeuroBE dynamically customizes the NN architecture and training set size to the bucket’s message complexity, and it modifies the loss function to depend on the message distribution.
Revised: In contrast to DBE , NeuroBE dynamically customizes the NN architecture and training set size to the bucket’s message complexity, and it modifies the loss function to depend on the message distribution.

Original: These modifications are described in the sequel.
Revised: These modifications are described in the sequel.

Original: do 5:
Revised: do 5:

Original: Let X p be the parent variable of X c in the bucket-tree 6: if width ( B c ) <
Revised: Let X p be the parent variable of X c in the bucket-tree 6: if width ( B c ) <

Original: i then 7: compute µ ∗ c → p ← (cid:80) X c (cid:81) f
Revised: i then 7: compute µ ∗ c → p ← (cid:80) X c (cid:81) f

Original: α ∈ B c f α , 8: else 9: µ θ,c
Revised: α ∈ B c f α , 8: else 9: µ θ,c

Original: → p ← NN-train (cid:0) { f α | f α ∈ B c } , L , b , η (cid:1) 10: end if 11:
Revised: → p ← NN-train (cid:0) { f α | f α ∈ B c } , L , b , η (cid:1) 10: end if 11:

Original: Put µ ∗ c → p or µ θ,c → p in B p 12: end for 13: ˆ Z = (cid:80) X 0 (cid:81)
Revised: Put µ ∗ c → p or µ θ,c → p in B p 12: end for 13: ˆ Z = (cid:80) X 0 (cid:81)

Original: f α ∈ B 1 f α 14: return ˆ Z and all messages generated
Revised: f α ∈ B 1 f α 14: return ˆ Z and all messages generated

Original: NN Architecture selection.
Revised: NN Architecture selection.

Original: Clearly, the NN size should depend on both the approximated function’s complexity and, especially, its dimensionality.
Revised: Clearly, the NN size should depend on both the approximated function’s complexity and, especially, its dimensionality.

Original: Since a bucket message’s scope size is the induced-width, w , we make the number of hidden units, h , a function of w while keeping the number of layers, L , constant.
Revised: Since a bucket message’s scope size is the induced-width, w , we make the number of hidden units, h , a function of w while keeping the number of layers, L , constant.

Original: Specifically, we use a simple function h
Revised: Specifically, we use a simple function h

Original: = b · w , where b ≥ 1 to fit the NN’s architecture to the message size.
Revised: = b · w , where b ≥ 1 to fit the NN’s architecture to the message size.

Original: Figure 2 is an example NN model architecture with an input layer of size w and 2 hidden layers with dimension h .
Revised: Figure 2 is an example NN model architecture with an input layer of size w and 2 hidden layers with dimension h .

Original: Next, we provide a rule to determine sample sizes to train a NN, depending on a notion of its complexity.
Revised: Next, we provide a rule to determine sample sizes to train a NN, depending on a notion of its complexity.

Original: NN complexity.
Revised: NN complexity.

Original: The notion of a Pseudo-dimension [Pollard, 1984, Anthony and Bartlett, 2002] is often used to measure the expressive power of a set of functions that can be learned by any statistical regression algorithm.
Revised: The notion of a Pseudo-dimension [Pollard, 1984, Anthony and Bartlett, 2002] is often used to measure the expressive power of a set of functions that can be learned by any statistical regression algorithm.

Original: The work in Bartlett et al.
Revised: The work in Bartlett et al.

Original: [2019] derived lower bounds to the pseudo-dimension for NNs with ReLU activation function (an architecture used in our work).
Revised: [2019] derived lower bounds to the pseudo-dimension for NNs with ReLU activation function (an architecture used in our work).

Original: We use the derived lower bound to estimate the pseudo-dimension ( ρ ) of a NN ( µ θ ), having an architecture of L layers and b · w hidden units, yielding (see Appendix for derivation):
Revised: We use the derived lower bound to estimate the pseudo-dimension ( ρ ) of a NN ( µ θ ), having an architecture of L layers and b · w hidden units, yielding (see Appendix for derivation):

Original: [Equation]
Revised: [Equation]

Original: Since in our experiments, the pair ( L, b ) are fixed for a given problem instance, our ρ estimate only varies with w and is used to determine the sample complexity.
Revised: Since in our experiments, the pair ( L, b ) are fixed for a given problem instance, our ρ estimate only varies with w and is used to determine the sample complexity.

Original: Sample Complexity.
Revised: Sample Complexity.

Original: As suggested in Vapnik [1999], we choose a sample size for training a NN, µ θ , proportional to its pseudo-dimension, Eq.
Revised: As suggested in Vapnik [1999], we choose a sample size for training a NN, µ θ , proportional to its pseudo-dimension, Eq.

Original: We therefore select a number of samples N satisfying the expression
Revised: We therefore select a number of samples N satisfying the expression

Original: [Equation]
Revised: [Equation]

Original: where η is a constant allowing us to tweak N linearly.
Revised: where η is a constant allowing us to tweak N linearly.

Original: Since the triplet ( L, b, η ) of a problem is fixed, the number of samples for training, N , is a function of w only.
Revised: Since the triplet ( L, b, η ) of a problem is fixed, the number of samples for training, N , is a function of w only.

Original: Sample Generation Let B be a generic bucket where variable X is eliminated; let F be the set of functions from
Revised: Sample Generation Let B be a generic bucket where variable X is eliminated; let F be the set of functions from

Original: Input : X , a variable to be eliminated, F , a set of functions over scope S ∪ { X } , N , an integer,
Revised: Input : X , a variable to be eliminated, F , a set of functions over scope S ∪ { X } , N , an integer,

Original: Output : D , a set of N samples 1: initialize D = {}, 2: for i = 1 ..
Revised: Output : D , a set of N samples 1: initialize D = {}, 2: for i = 1 ..

Original: N do 3: s ← sample uniformly from domain( S ) 4: µ ∗ ( s ) ← (cid:80) x
Revised: N do 3: s ← sample uniformly from domain( S ) 4: µ ∗ ( s ) ← (cid:80) x

Original: (cid:81) f ∈ F f ( s, x ) {Eq.
Revised: (cid:81) f ∈ F f ( s, x ) {Eq.

Original: Add ( s , µ ∗ ( s ) ) to D 6:
Revised: Add ( s , µ ∗ ( s ) ) to D 6:

Original: Update µ ∗ min , µ ∗ max 7: end for 8: Normalize D (Eq 4) 9: return D the graphical model (initialized in line 2, Algorithm 1) as well as messages from the previous buckets (line 11, Algorithm 1) residing in B and S be the scope of the output message function µ ∗ .
Revised: Update µ ∗ min , µ ∗ max 7: end for 8: Normalize D (Eq 4) 9: return D the graphical model (initialized in line 2, Algorithm 1) as well as messages from the previous buckets (line 11, Algorithm 1) residing in B and S be the scope of the output message function µ ∗ .

Original: Then, Algorithm 2 generates a dataset
Revised: Then, Algorithm 2 generates a dataset

Original: D containing a given number of samples N .
Revised: D containing a given number of samples N .

Original: The algorithm iteratively and uniformly at random, samples a configuration { S = s } from the domain of S and computes the exact local bucket function value for s using Eq 1 (lines 3,4).
Revised: The algorithm iteratively and uniformly at random, samples a configuration { S = s } from the domain of S and computes the exact local bucket function value for s using Eq 1 (lines 3,4).

Original: The pair < s, µ ∗ ( s ) > is added to the dataset D (line 5).
Revised: The pair < s, µ ∗ ( s ) > is added to the dataset D (line 5).

Original: A normalization step occurs in line 10, where each sample s is shifted and scaled to the range [ − 1 , 1] and µ ∗ ( s ) is shifted and scaled to [0 , 1] , to accelerate training of the NN [Le Cun et al., 1991], by:
Revised: A normalization step occurs in line 10, where each sample s is shifted and scaled to the range [ − 1 , 1] and µ ∗ ( s ) is shifted and scaled to [0 , 1] , to accelerate training of the NN [Le Cun et al., 1991], by:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where µ ∗ min , µ ∗ max (line 6) are defined relative to the dataset
Revised: ) where µ ∗ min , µ ∗ max (line 6) are defined relative to the dataset

Original: D by µ ∗ min = min s ∈ D µ ∗ ( s ) and µ ∗ max = max s ∈ D µ ∗ ( s ) .
Revised: D by µ ∗ min = min s ∈ D µ ∗ ( s ) and µ ∗ max = max s ∈ D µ ∗ ( s ) .

Original: Loss Function Algorithm DBE sampled each message input configuration uniformly and uses the mean square error loss function for training.
Revised: Loss Function Algorithm DBE sampled each message input configuration uniformly and uses the mean square error loss function for training.

Original: However, it seems intuitive that generating the samples by taking into account the message distribution could lead to more effective training of the function.
Revised: However, it seems intuitive that generating the samples by taking into account the message distribution could lead to more effective training of the function.

Original: Since sampling directly from the message distribution is hard, we instead weight each sample by an importance weight within the loss function, described next.
Revised: Since sampling directly from the message distribution is hard, we instead weight each sample by an importance weight within the loss function, described next.

Original: Definition 1 (I.m.s.e loss) .
Revised: Definition 1 (I.m.s.e loss) .

Original: Let µ θ be the NN for approximating the function µ ∗ norm .
Revised: Let µ θ be the NN for approximating the function µ ∗ norm .

Original: Let D
Revised: Let D

Original: =
Revised: =

Original: D
Revised: D

Original: Train be the training set.
Revised: Train be the training set.

Original: Then, the I.m.s.e loss function for a given mini-batch,
Revised: Then, the I.m.s.e loss function for a given mini-batch,

Original: D i ∈
Revised: D i ∈

Original: D of size # D
Revised: D of size # D

Original: i is defined by:
Revised: i is defined by:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Log transformations Usually in our experiments we apply a log transformation to the input functions, for computational reasons.
Revised: Log transformations Usually in our experiments we apply a log transformation to the input functions, for computational reasons.

Original: The algorithms presented here remain the same; however the values µ ∗ , µ ∗ min and µ ∗ max in this case refer to the log of the original function values.
Revised: The algorithms presented here remain the same; however the values µ ∗ , µ ∗ min and µ ∗ max in this case refer to the log of the original function values.

Original: In cases when we use the log-space computation, the weight function W ( s ) (Eq. 6) is not suitable.
Revised: In cases when we use the log-space computation, the weight function W ( s ) (Eq. 6) is not suitable.

Original: We instead use modified importance weights,
Revised: We instead use modified importance weights,

Original: [Equation]
Revised: [Equation]

Original: Note that the importance weight, W ( s ) or W log ( s ) , are computed in the original function space that is not normalized, which is either W ( s ) or W log ( s ) .
Revised: Note that the importance weight, W ( s ) or W log ( s ) , are computed in the original function space that is not normalized.

Original: MaskedNet For problems with determinism, i.e., a high proportion of zero probability states, a fully connected feedforward NN was unable to correctly predict deterministic outputs and hence Razeghi et al.
Revised: MaskedNet For problems with determinism, i.e., a high proportion of zero probability states, a fully connected feedforward NN was unable to correctly predict deterministic outputs and hence Razeghi et al.

Original: [2021] used a MaskedNet.
Revised: [2021] used a MaskedNet.

Original: The input configuration is sent to a fully connected layer with a RELU activation function to obtain a feature vector.
Revised: The input configuration is sent to a fully connected layer with a RELU activation function to obtain a feature vector.

Original: This feature vector is then sent to two sister layers: the first layer outputs a binary mask responsible for determining whether the final output is zero, and the second layer is responsible for predicting the target value of the Bucket’s function.
Revised: This feature vector is then sent to two sister layers: the first layer outputs a binary mask responsible for determining whether the final output is zero, and the second layer is responsible for predicting the target value of the Bucket’s function.

Original: The activation functions of the two final layers are the logistic function and the softplus function, respectively.
Revised: The activation functions of the two final layers are the logistic function and the softplus function, respectively.

Original: The outputs from the two sister networks are multiplied together to get the final output of the MaskedNet.
Revised: The outputs from the two sister networks are multiplied together to get the final output of the MaskedNet.

Original: The loss for the MaskedNet in NeuroBE is thus a sum of the binary crossentropy loss (from the first output layer) and the proposed
Revised: The loss for the MaskedNet in NeuroBE is thus a sum of the binary crossentropy loss (from the first output layer) and the proposed

Original: I.m.s.e loss (from the second output layer).
Revised: I.m.s.e loss (from the second output layer).

Original: Thus when a sample configuration s has µ ∗ ( s ) = 0 , the loss becomes the binary cross-entropy error, since W ( s ) = 0 , following
Revised: Thus when a sample configuration s has µ ∗ ( s ) = 0 , the loss becomes the binary cross-entropy error, since W ( s ) = 0 , following

Original: Eq.
Revised: Eq.

Original: (5) and (6).
Revised: (5) and (6).

Original: NN-Train Algorithm 3 describes the procedure NN-Train .
Revised: NN-Train Algorithm 3 describes the procedure NN-Train .

Original: Its input parameters are L, b, η where L is the number of layers, b is a constant to determine the number of hidden units, b · w (line 1), and η is another constant to determine the training sample size N (line 2, Eq.
Revised: Its input parameters are L, b, η where L is the number of layers, b is a constant to determine the number of hidden units, b · w (line 1), and η is another constant to determine the training sample size N (line 2, Eq.

Original: A major step occurs next where the algorithm generates a dataset D and splits it into the training set D Train of size N , validation set D V al of size N/ 4 and testing set D Test of fixed size (50k) (lines 3-4; see also Algorithm 2).
Revised: A major step occurs next where the algorithm generates a dataset D and splits it into the training set D Train of size N , validation set D V al of size N/ 4 and testing set D Test of fixed size (50k) (lines 3-4; see also Algorithm 2).

Original: Lines 8-12 then describe the batch training for updating the NN parameters θ using the I.m.s.e loss function (line 11, Eq.
Revised: Lines 8-12 then describe the batch training for updating the NN parameters θ using the I.m.s.e loss function (line 11, Eq.

Original: (5)), and the
Revised: (5)), and the

Original: Adam optimizer [Kingma and Ba, 2014] (line 12) with a learning rate of 0 .
Revised: Adam optimizer [Kingma and Ba, 2014] (line 12) with a learning rate of 0 .

Original: 001 and a batch-size of 256 across all benchmarks.
Revised: 001 and a batch-size of 256 across all benchmarks.

Original: At the end of each epoch, the current model is evaluated on a holdout validation set (line 14).
Revised: At the end of each epoch, the current model is evaluated on a holdout validation set (line 14).

Original: We evaluate the early-stopping criteria (line 15), which is assigned
Revised: We evaluate the early-stopping criteria (line 15), which is assigned

Original: True when either the maximum limit # epochs is reached
Revised: True when either the maximum limit # epochs is reached

Original: 1: # h ← b ∗ w 2: N ← # training samples( w, η, L, b ) {Eq. 3} 3: D ← generate-samples( X, F, N + N/ 4 + 50 k ) 4: D Train , D V al , D Test ← Split( D ) 5: Initialize NN parameters θ , p =1, early-stopping ← False 6: while p ≤ # epochs and ¬ early-stopping do 7:
Revised: 1: # h ← b ∗ w 2: N ← # training samples( w, η, L, b ) {Eq. 3} 3: D ← generate-samples( X, F, N + N/ 4 + 50 k ) 4: D Train , D V al , D Test ← Split( D ) 5: Initialize NN parameters θ , p =1, early-stopping ← False 6: while p ≤ # epochs and ¬ early-stopping do 7:

Original: D 1 , .., D k ← divide D Train to minibatch 8: for i
Revised: D 1 , .., D k ← divide D Train to minibatch 8: for i

Original: .k
Revised: .k

Original: do 9:
Revised: do 9:

Original: Let D i
Revised: Let D i

Original: = { ( s, µ ∗ norm ) } 10:
Revised: = { ( s, µ ∗ norm ) } 10:

Original: Compute { µ θ ( s ) | s ∈ D i } 11: loss D i ← L D i ( µ ∗ norm , µ θ ) {Eq. 5} 12: θ ← update θ by optimize(Adam, loss D i , θ ) 13: end for 14: loss D val ← L D val ( µ ∗ norm , µ θ ) {For stop condition} 15: early-stopping ← evaluate early-stopping( loss D Val ) 16: p ← p +17: end while 18: Unnormalize { µ θ ( s ) , µ ∗ ( s ) | s ∈ D Test } {Inverse of Eq.
Revised: Compute { µ θ ( s ) | s ∈ D i } 11: loss D i ← L D i ( µ ∗ norm , µ θ ) {Eq. 5} 12: θ ← update θ by optimize(Adam, loss D i , θ ) 13: end for 14: loss D val ← L D val ( µ ∗ norm , µ θ ) {For stop condition} 15: early-stopping ← evaluate early-stopping( loss D Val ) 16: p ← p +17: end while 18: Unnormalize { µ θ ( s ) , µ ∗ ( s ) | s ∈ D Test } {Inverse of Eq.

Original: 4} 19: ˆ ϵ ← max s ∈ D Test ( logµ ∗ ( s ) − logµ θ ( s )) 20: return µ θ , ˆ ϵ or the validation error increases for two consecutive epochs.
Revised: 4} 19: ˆ ϵ ← max s ∈ D Test ( logµ ∗ ( s ) − logµ θ ( s )) 20: return µ θ , ˆ ϵ or the validation error increases for two consecutive epochs.

Original: Once training is complete, we compute the maximum log relative error between the target and NN approximated messages over a test set (lines 18-19).
Revised: Once training is complete, we compute the maximum log relative error between the target and NN approximated messages over a test set (lines 18-19).

Original: In the next section, we use this maximum error to analyse the propagation of error in NeuroBE .
Revised: In the next section, we use this maximum error to analyse the propagation of error in NeuroBE .

Original: The NN-train procedure then returns the approximated message µ θ , along with its estimated error.
Revised: The NN-train procedure then returns the approximated message µ θ , along with its estimated error.

Original: Complexity.
Revised: Complexity.

Original: The time and space complexity for learning a single message in NeuroBE is a function of the NN and sample size.
Revised: The time and space complexity for learning a single message in NeuroBE is a function of the NN and sample size.

Original: In contrast to DBE , here the NN and sample sizes vary with the bucket’s width.
Revised: In contrast to DBE , here the NN and sample sizes vary with the bucket’s width.

Original:  ERROR ANALYSIS
Revised:  ERROR ANALYSIS

Original: We now analyse the relationship between the local errors contributed by each approximated message and the global partition function error, focusing on a simple case where the bucket tree is a chain.
Revised: We now analyse the relationship between the local errors contributed by each approximated message and the global partition function error, focusing on a simple case where the bucket tree is a chain.

Original: Definition 2 (local and global bucket errors) .
Revised: Definition 2 (local and global bucket errors) .

Original: Given a bucket
Revised: Given a bucket

Original: B , let λ be the (global) exact message generated in B , µ ∗ be the (local) exact message in B at the time of message computation, and µ = NN-train ( µ ∗ ) be its NN approximation.
Revised: B , let λ be the (global) exact message generated in B , µ ∗ be the (local) exact message in B at the time of message computation, and µ = NN-train ( µ ∗ ) be its NN approximation.

Original: Then, we define the local and global log relative errors as:
Revised: Then, we define the local and global log relative errors as:

Original: E = log µ ∗ − log µ, and,
Revised: E = log µ ∗ − log µ, and,

Original: G = log λ − log µ.
Revised: G = log λ − log µ.

Original: We use log relative error since it simplifies the analysis.
Revised: We use log relative error since it simplifies the analysis.

Original: We now show the following relationship:
Revised: We now show the following relationship:

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume a bucket-chain along an ordering d , and let B c be a bucket along the chain at position c having scope S of its bucket message.
Revised: Assume a bucket-chain along an ordering d , and let B c be a bucket along the chain at position c having scope S of its bucket message.

Original: Let E c ( s ) = log µ ∗ c ( s ) − log µ c ( s ) and let ϵ c = max s ∈ D ( S ) | E c ( s ) | .
Revised: Let E c ( s ) = log µ ∗ c ( s ) − log µ c ( s ) and let ϵ c = max s ∈ D ( S ) | E c ( s ) | .

Original: Then,
Revised: Then,

Original: G c = log λ c − log µ c ≤ n − c (cid:88) k
Revised: G c = log λ c − log µ c ≤ n − c (cid:88) k

Original: = ϵ c + k
Revised: = ϵ c + k

Original: In particular, since λ 1 = Z and µ 1 = ˆ Z ,
Revised: In particular, since λ 1 = Z and µ 1 = ˆ Z ,

Original: G 1 = log Z − log ˆ Z ≤ n − 1 (cid:88) k
Revised: G 1 = log Z − log ˆ Z ≤ n − 1 (cid:88) k

Original: = ϵ 1+ k (8)
Revised: = ϵ 1+ k (8)

Original: For the proof see the supplementary material.
Revised: For the proof see the supplementary material.

Original:  EMPIRICAL EVALUATION 5.1 EXPERIMENTAL SETUP
Revised:  EMPIRICAL EVALUATION 5.1 EXPERIMENTAL SETUP

Original: We conducted experiments comparing NeuroBE against
Revised: We conducted experiments comparing NeuroBE against

Original: WMB [Dechter and Rish, 2003, Liu and Ihler, 2012] and
Revised: WMB [Dechter and Rish, 2003, Liu and Ihler, 2012] and

Original: DBE [Razeghi et al., 2021] over several benchmarks.
Revised: DBE [Razeghi et al., 2021] over several benchmarks.

Original: We also compare the impact of the two loss functions, m.s.e and I.m.s.e, on the performance of NeuroBE .
Revised: We also compare the impact of the two loss functions, m.s.e and I.m.s.e, on the performance of NeuroBE .

Original: Finally, we illustrate how increasing sample and NN complexity impact performance.
Revised: Finally, we illustrate how increasing sample and NN complexity impact performance.

Original: i-bounds.
Revised: i-bounds.

Original: All three algorithms, WMB , DBE and NeuroBE , use the i-bound parameter ( i ).
Revised: All three algorithms, WMB , DBE and NeuroBE , use the i-bound parameter ( i ).

Original: As noted, in WMB higher i-bounds lead to more accurate bounds with more time and memory, up to their memory limit.
Revised: As noted, in WMB higher i-bounds lead to more accurate bounds with more time and memory, up to their memory limit.

Original: Algorithms DBE and
Revised: Algorithms DBE and

Original: NeuroBE are also observed to improve accuracy and time with increasing i-bounds because of the reduced number of trained buckets #NB( i ).
Revised: NeuroBE are also observed to improve accuracy and time with increasing i-bounds because of the reduced number of trained buckets #NB( i ).

Original: Hence, for a fair comparison we use an i -bound of 10 for some (easy) benchmarks, while primarily using the highest feasible i -bound of 20 dictated by WMB ’s memory bound for other (hard) benchmarks.
Revised: Hence, for a fair comparison we use an i -bound of 10 for some (easy) benchmarks, while primarily using the highest feasible i -bound of 20 dictated by WMB ’s memory bound for other (hard) benchmarks.

Original: Benchmarks Following the example of DBE , we evaluated NeuroBE on instances selected from three well-known benchmarks from the UAI repository used in Kask et al.
Revised: Benchmarks Following the example of DBE , we evaluated NeuroBE on instances selected from three well-known benchmarks from the UAI repository used in Kask et al.

Original: [2020]: grids (vision domain), pedigree (genetic linkage analysis) and DBNs.
Revised: [2020]: grids (vision domain), pedigree (genetic linkage analysis) and DBNs.

Original: We targeted diverse benchmarks (in structure and level of determinism) and aimed for different levels of hardness.
Revised: We targeted diverse benchmarks (in structure and level of determinism) and aimed for different levels of hardness.

Original: Thus, in the grids benchmark, we distinguish those problems that can be solved exactly, which we
Revised: Thus, in the grids benchmark, we distinguish those problems that can be solved exactly, which we

Original: [Figure]
Revised: [Figure]

Original: call “grid-easy”, from those that cannot be solved, called “grid-hard”.
Revised: call “grid-easy”, from those that cannot be solved, called “grid-hard”.

Original: We also distinguish benchmarks that possess determinism , namely have a high proportion of zero probabilities, since it can impact training.
Revised: We also distinguish benchmarks that possess determinism , namely have a high proportion of zero probabilities, since it can impact training.

Original: We randomly selectedinstances from Grids, with easy ones (400 variables, width 20-30) and hard ones (1600 variables, width 55 or 114),from pedigrees ( ≈ 800 variables, width ≈ 34), which posses high level of determinism and 6 from DBNs ( ≈ 40 variables, width ≈ 22), totalling 25 instances.
Revised: We randomly selectedinstances from Grids, with easy ones (400 variables, width 20-30) and hard ones (1600 variables, width 55 or 114),from pedigrees ( ≈ 800 variables, width ≈ 34), which posses high level of determinism and 6 from DBNs ( ≈ 40 variables, width ≈ 22), totalling 25 instances.

Original: As described in section 3, we apply log transformations to Grids and DBNs since they have large message function values.
Revised: As described in section 3, we apply log transformations to Grids and DBNs since they have large message function values.

Original: NN architectures and sample sizes.
Revised: NN architectures and sample sizes.

Original: Through a process of trial and error on a selected instance from each benchmark, we selected the parameters of the architectures and sample sizes as follows.
Revised: Through a process of trial and error on a selected instance from each benchmark, we selected the parameters of the architectures and sample sizes as follows.

Original: We selected L = 1 , h = 3 w, and N avg ∈ [149 k, 350 k ] for pedigrees; L = 2 , h = 3 w, and N avg ∈ [80 k, 180 k ] for DBN; L = 2 , h = w, and
Revised: We selected L = 1 , h = 3 w, and N avg ∈ [149 k, 350 k ] for pedigrees; L = 2 , h = 3 w, and N avg ∈ [80 k, 180 k ] for DBN; L = 2 , h = w, and

Original: N avg ∈ [23 k, 68 k ] , for grid-easy; and L = 1 , h = w, and
Revised: N avg ∈ [23 k, 68 k ] , for grid-easy; and L = 1 , h = w, and

Original: N avg ∈ [75 k, 150 k ] for grid-hard.
Revised: N avg ∈ [75 k, 150 k ] for grid-hard.

Original: Performance measures We evaluate the performance of
Revised: Performance measures We evaluate the performance of

Original: NeuroBE using: error = | log e Z − log e ˆ Z | where ˆ Z estimates Z .
Revised: NeuroBE using: error = | log e Z − log e ˆ Z | where ˆ Z estimates Z .

Original: When the exact Z is not available (i.e., for hard
Revised: When the exact Z is not available (i.e., for hard

Original: Grid benchmark), we use Z ∗ as a surrogate to Z , which is obtained using an advanced sampling scheme for a duration of 100 ∗ 1 hr [Kask et al., 2020].
Revised: Grid benchmark), we use Z ∗ as a surrogate to Z , which is obtained using an advanced sampling scheme for a duration of 100 ∗ 1 hr [Kask et al., 2020].

Original: 5.2 RESULTS
Revised: 5.2 RESULTS

Original: Figure 3 compares NeuroBE against WMB and DBE over the four benchmarks.
Revised: Figure 3 compares NeuroBE against WMB and DBE over the four benchmarks.

Original: The first few columns show the problem statistics for instances in the respective benchmarks (pedigree, grid-hard; grid-easy and DBN).
Revised: The first few columns show the problem statistics for instances in the respective benchmarks (pedigree, grid-hard; grid-easy and DBN).

Original: We then show results on WMB ’s error, followed by DBE ’s
Revised: We then show results on WMB ’s error, followed by DBE ’s

Original: and NeuroBE ’s performance information.
Revised: and NeuroBE ’s performance information.

Original: We omit WMB ’s time performance in
Revised: We omit WMB ’s time performance in

Original: Figure 3 since its execution takes only a minute.
Revised: Figure 3 since its execution takes only a minute.

Original: For DBE , we report the number of buckets trained by NNs, ( # NB ), followed by the average error, minimum error, standard deviation, and average time (in hours) over five runs, which is preferable due to stochasticity.
Revised: For DBE , we report the number of buckets trained by NNs, ( # NB ), followed by the average error, minimum error, standard deviation, and average time (in hours) over five runs, which is preferable due to stochasticity.

Original: For NeuroBE with both m.s.e and I.m.s.e loss functions, we also report statistics about NN architecture and sample size that varies within problem instances: the average and maximum number of training samples, N avg and N max , and maximum number of hidden units, h max .
Revised: For NeuroBE with both m.s.e and I.m.s.e loss functions, we also report statistics about NN architecture and sample size that varies within problem instances: the average and maximum number of training samples, N avg and N max , and maximum number of hidden units, h max .

Original: Pedigrees We observe immediately that, overall, NeuroBE with the I.m.s.e loss function is clearly superior to DBE and NeuroBE with m.s.e loss.
Revised: Pedigrees We observe immediately that, overall, NeuroBE with the I.m.s.e loss function is clearly superior to DBE and NeuroBE with m.s.e loss.

Original: In particular, it is ≥ 5 times more accurate than DBE for almost all instances and takes less time, since it uses far less training samples.
Revised: In particular, it is ≥ 5 times more accurate than DBE for almost all instances and takes less time, since it uses far less training samples.

Original: NeuroBE with the I.m.s.e loss function also outperforms WMB on most instances.
Revised: NeuroBE with the I.m.s.e loss function also outperforms WMB on most instances.

Original: NeuroBE with the m.s.e loss function is less accurate than both WMB and DBE for most instances.
Revised: NeuroBE with the m.s.e loss function is less accurate than both WMB and DBE for most instances.

Original: It also fails to approximate the partition function for instance
Revised: It also fails to approximate the partition function for instance

Original: [Figure]
Revised: [Figure]

Original: 5 (errors and standard deviation denoted by # in Figure 3).
Revised: 5 (errors and standard deviation denoted by # in Figure 3).

Original: Here, DBE has similar or even worse accuracy than WMB .
Revised: Here, DBE has similar or even worse accuracy than WMB .

Original: Grid-hard.
Revised: Grid-hard.

Original: The results for the grid-hard benchmark is shown in Fig. 4b.
Revised: The results for the grid-hard benchmark is shown in Fig. 4b.

Original: We used the highest possible i -bound of 20 , and we observe that NeuroBE can achieve a far lower average error and standard deviation and takes far less time than WMB and DBE , particularly with the I.m.s.e loss.
Revised: We used the highest possible i -bound of 20 , and we observe that NeuroBE can achieve a far lower average error and standard deviation and takes far less time than WMB and DBE , particularly with the I.m.s.e loss.

Original: In most cases, we see a reduction in time by a factor of two.
Revised: In most cases, we see a reduction in time by a factor of two.

Original: DBE outperforms WMB across all problem instances.
Revised: DBE outperforms WMB across all problem instances.

Original: Grid-easy.
Revised: Grid-easy.

Original: The results for the grid-easy benchmark is shown in Figure 4c.
Revised: The results for the grid-easy benchmark is shown in Figure 4c.

Original: We used a lower i -bound of 10 to facilitate the training of a relatively large number of buckets.
Revised: We used a lower i -bound of 10 to facilitate the training of a relatively large number of buckets.

Original: As expected, when an instance has a low induced-width and only a small number of buckets are approximated (e.g.,
Revised: As expected, when an instance has a low induced-width and only a small number of buckets are approximated (e.g.,

Original: ID 2), DBE obtains high accuracy.
Revised: ID 2), DBE obtains high accuracy.

Original: As the induced-width increases and more buckets are trained, NeuroBE has far higher accuracy compared with both WMB and DBE .
Revised: As the induced-width increases and more buckets are trained, NeuroBE has far higher accuracy compared with both WMB and DBE .

Original: Both the loss functions in NeuroBE show similar performance.
Revised: Both the loss functions in NeuroBE show similar performance.

Original: DBN We report results for the DBN benhcmark for two i -bounds.
Revised: DBN We report results for the DBN benhcmark for two i -bounds.

Original: Overall, the results are mixed.
Revised: Overall, the results are mixed.

Original: For i -bound = 20,
Revised: For i -bound = 20,

Original: NeuroBE achieves a higher accuracy than DBE for half of the instances with far less #training samples (but with more training time).
Revised: NeuroBE achieves a higher accuracy than DBE for half of the instances with far less #training samples (but with more training time).

Original: It is superior to WMB on instances 2, 3, and 5.
Revised: It is superior to WMB on instances 2, 3, and 5.

Original: When comparing the two loss functions in NeuroBE , I.m.s.e loss has better (or similar) performance for most instances.
Revised: When comparing the two loss functions in NeuroBE , I.m.s.e loss has better (or similar) performance for most instances.

Original: However, WMB performs better on instance 1, 4, and 6, as the induced-width is closer to the i -bound.
Revised: However, WMB performs better on instance 1, 4, and 6, as the induced-width is closer to the i -bound.

Original: For i -bound = 10, DBE and NeuroBE show better accuracy than WMB for those three instances.
Revised: For i -bound = 10, DBE and NeuroBE show better accuracy than WMB for those three instances.

Original: DBE has better accuracy when compared with NeuroBE , using more #training samples (and hence, more time).
Revised: DBE has better accuracy when compared with NeuroBE , using more #training samples (and hence, more time).

Original: NeuroBE with I.m.s.e loss is better performing compared with m.s.e loss on most instances.
Revised: NeuroBE with I.m.s.e loss is better performing compared with m.s.e loss on most instances.

Original: Overall, NeuroBE when trained with I.m.s.e loss takes more time than with m.s.e loss for the same #training samples.
Revised: Overall, NeuroBE when trained with I.m.s.e loss takes more time than with m.s.e loss for the same #training samples.

Original: In summary, NeuroBE using I.m.s.e compared against DBE is about 50 % faster while also far more accurate on pedigrees, twice as fast and 5 to 10 fold more accurate on hard grids.
Revised: In summary, NeuroBE using I.m.s.e compared against DBE is about 50 % faster while also far more accurate on pedigrees, twice as fast and 5 to 10 fold more accurate on hard grids.

Original: It is also faster and more accurate on easy grids and has a mixed but still comparable performance on DBNs.
Revised: It is also faster and more accurate on easy grids and has a mixed but still comparable performance on DBNs.

Original: The impact of loss functions.
Revised: The impact of loss functions.

Original: We observe that NeuroBE with the I.m.s.e loss shows better performance (lower average error and standard deviation) than NeuroBE with the m.s.e loss for the pedigree instances and the majority of
Revised: We observe that NeuroBE with the I.m.s.e loss shows better performance (lower average error and standard deviation) than NeuroBE with the m.s.e loss for the pedigree instances and the majority of

Original: DBN, grid-easy and grid-hard instances.
Revised: DBN, grid-easy and grid-hard instances.

Original: An F-test with a significance level of 0 .
Revised: An F-test with a significance level of 0 .

Original: on the two groups of partition function estimates (each consisting of five approximations) showed that the means are significantly different for pedigrees, in Figure 3(a).
Revised: on the two groups of partition function estimates (each consisting of five approximations) showed that the means are significantly different for pedigrees, in Figure 3(a).

Original: For the grids and DBN, there was no statistical difference between the two means.
Revised: For the grids and DBN, there was no statistical difference between the two means.

Original: However, by inspection, we see a reduction in the standard deviation for almost all instances.
Revised: However, by inspection, we see a reduction in the standard deviation for almost all instances.

Original: Impact of architecture size.
Revised: Impact of architecture size.

Original: Figure 4 shows the impact of architecture size on time and accuracy for a few problem instances.
Revised: Figure 4 shows the impact of architecture size on time and accuracy for a few problem instances.

Original: We show results for two different NN architectures and their associated sample sizes.
Revised: We show results for two different NN architectures and their associated sample sizes.

Original: As expected, we see that increasing the sample and NN sizes increases both time and accuracy for pedigrees.
Revised: As expected, we see that increasing the sample and NN sizes increases both time and accuracy for pedigrees.

Original: For grid-hard instances, we just increased the sample sizes and kept the same architecture having h = w .
Revised: For grid-hard instances, we just increased the sample sizes and kept the same architecture having h = w .

Original: We observe that the average error is reduced, as expected.
Revised: We observe that the average error is reduced, as expected.

Original: Instances from grid-easy and DBN (except ID 2) show a similar improvement in performance with a larger
Revised: Instances from grid-easy and DBN (except ID 2) show a similar improvement in performance with a larger

Original: NN and training sample size.
Revised: NN and training sample size.

Original: This trend illustrates that increasing the size of the NN (matched by a suitable increase in sample size), improves the accuracy of NeuroBE , at the cost of more time and memory.
Revised: This trend illustrates that increasing the size of the NN (matched by a suitable increase in sample size), improves the accuracy of NeuroBE , at the cost of more time and memory.

Original: A key question for future work is how to develop a policy that can facilitate gradual control of architecture and sample size increase to improve performance in an anytime way.
Revised: A key question for future work is how to develop a policy that can facilitate gradual control of architecture and sample size increase to improve performance in an anytime way.

Original:  CONCLUSION & FUTURE WORK
Revised:  CONCLUSION & FUTURE WORK

Original: In this work, we advance the earlier theme of using Neural
Revised: In this work, we advance the earlier theme of using Neural

Original: Networks to approximate the class of bucket-elimination algorithms that is at the heart to probabilistic reasoning.
Revised: Networks to approximate the class of bucket-elimination algorithms that is at the heart to probabilistic reasoning.

Original: NeuroBE can be viewed as a realization of Neuro-Dynamic
Revised: NeuroBE can be viewed as a realization of Neuro-Dynamic

Original: Programming schemes [Bertsekas and Tsitsiklis, 1996], in the context of graphical models.
Revised: Programming schemes [Bertsekas and Tsitsiklis, 1996], in the context of graphical models.

Original: That being said, it requires the training of numerous NNs per problem instance, and thus, the central aim of NeuroBE ’s design (customizing NN architectures, training samples, and the loss function to the message) is to enhance efficiency and scalability of such schemes.
Revised: That being said, it requires the training of numerous NNs per problem instance, and thus, the central aim of NeuroBE ’s design (customizing NN architectures, training samples, and the loss function to the message) is to enhance efficiency and scalability of such schemes.

Original: We presented NeuroBE and illustrated on challenging instances over three benchmarks that it can be far more accurate and requires less time compared with Deep
Revised: We presented NeuroBE and illustrated on challenging instances over three benchmarks that it can be far more accurate and requires less time compared with Deep

Original: Bucket Elimination ( DBE ).
Revised: Bucket Elimination ( DBE ).

Original: It is also superior to weighted mini-bucket ( WMB ) even when provided with the highest memory resources feasible.
Revised: It is also superior to weighted mini-bucket ( WMB ) even when provided with the highest memory resources feasible.

Original: Future Work.
Revised: Future Work.

Original: We will explore further how to improve
Revised: We will explore further how to improve

Original: NeuroBE ’s efficiency by customizing additional features of a NN and its training per bucket (e.g., varying the number of layers).
Revised: NeuroBE ’s efficiency by customizing additional features of a NN and its training per bucket (e.g., varying the number of layers).

Original: We will also explore moving from training buckets separately per single variable to training clusters of buckets within a tree-decomposition, thus training a single function per union of buckets [Dechter, 2013], potentially reducing the number of trained functions at the cost of more time for sample generation.
Revised: We will also explore moving from training buckets separately per single variable to training clusters of buckets within a tree-decomposition, thus training a single function per union of buckets [Dechter, 2013], potentially reducing the number of trained functions at the cost of more time for sample generation.

Original: Finally, we will explore parameter sharing by training multiple bucket functions simultaneously in a single problem instance and across a benchmark of instances.
Revised: Finally, we will explore parameter sharing by training multiple bucket functions simultaneously in a single problem instance and across a benchmark of instances.

Original:  ACKNOWLEDGEMENT
Revised:  ACKNOWLEDGEMENT

Original: We thank our reviewers and our lab colleagues Bobak
Revised: We thank our reviewers and our lab colleagues Bobak

Original: Pezeshki and Nick Cohen for their constructive feedback.
Revised: Pezeshki and Nick Cohen for their constructive feedback.

Original: This work was supported in part by NSF grants IIS-2008516.
Revised: This work was supported in part by NSF grants IIS-2008516.

Original: . Main Track
Revised: . Main Track

Original: TNN
Revised: TNN

Original: URL https://doi.org/10.1109/TNN.
Revised: URL https://doi.org/10.1109/TNN.

Original: ISBN 0387987800.
Revised: ISBN 0387987800.