On Recovering from Modeling Errors Using Testing Bayesian Networks
Abstract
We consider the problem of supervised learning with Bayesian Networks when the used dependency structure is incomplete due to missing edges or missing variable states.
These modeling errors induce independence constraints on the learned model that may not hold in the true, data-generating distribution.
We provide a uniﬁed treatment of these modeling errors as instances of state-space abstractions.
We then identify a class of Bayesian Networks and queries which allow one to fully recover from such modeling errors if one can choose Conditional Probability Tables (CPTs) dynamically based on evidence.
We show theoretically that the recently proposed
Testing Bayesian Networks (TBNs), which can be trained by compiling them into Testing Arithmetic Circuits (TACs), provide a promising construct for emulating this CPT selection mechanism.
Finally, we present empirical results that illustrate the promise of TBNs as a tool for recovering from certain modeling errors in the context of supervised learning.
1. Introduction
Supervised learning has become very inﬂuential recently and stands behind most real-world applications of AI today.
In supervised learning, one learns a function from labeled data, a practice that is now dominated by neural networks; see (Goodfellow et al., 2016; Hinton et al., 2006; Bengio et al., 2006; Ranzato et al., 2006).
This particular use of neural networks is an example of model-free supervised learning to be contrasted with model-based supervised learning; see, e.g., (Bishop, 2013; Gao et al., 2018; Chen et al., 2020).
The latter type feeds more information and assumptions into the learning process so should perform better in principle, let alone being more interpretable and also well-positioned to provide guarantees on the learned functions.
Yet, modelbased supervised learning has been outperformed by the model-free approach in many applications which has somewhat created a dilemma for AI (Darwiche, 2018).
A major challenge for model-based supervised learning is the availability of accurate models.
Consider Bayesian networks (Pearl, 1989), for example, which can be used for supervised learning as follows.
Given a Bayesian network structure, one can learn its parameters to optimize a distribution for some query variable as a function of the values for some evidence variables.
The used structure deﬁnes the space of functions that can be learned so if the structure is incorrect the space may fail to include the datagenerating function.
Hence, while the use of correct models can be very advantageous, the use of incorrect ones can be quite problematic.
Moreover, it may be very difﬁcult in some cases to build even approximate models as we may have very little understanding of the underlying phenomena.
One can of course aim to learn the structure of a Bayesian network in addition to its parameters.
This direction has received signiﬁcant attention but poses substantial computational difﬁculties; see (Darwiche, 2009; Koller & Friedman, 2009; Murphy, 2012) for some comprehensive treatments on learning Bayesian networks and (Elidan et al., 2001; Kim & Leskovec, 2011; Elidan & Friedman, 2001) for some works that try to infer modeling errors from data.
We address the challenge of model-based supervised learning in this paper under the assumption that we have an incorrect Bayesian network structure.
That is, the goal is to learn a function from labeled data using an incorrect structure while trying to recover from modeling errors that may have manifested.
Our approach is based on two main insights, the ﬁrst of which is motivated by earlier work on state-space abstractions (Wellman & Liu, 1994; Chang & Fung, 1990).
In particular, we show that one can cast certain modeling errors (missing states and edges) as instances of such abstractions, which allows us to cast the recovery from modeling errors as a process of recovering from state-space abstractions.
Our central contribution is a result which shows that, under certain conditions, one can fully recover from state-space abstractions if one is able to learn dynamic parameters for the Bayesian network structure; that is, if one is able to learn a collection of parameters from which a set is selected during inference time based on evidence (function input).
In other words, one has to learn multiple Bayesian networks which share the same (incomplete) structure and vary only across their parameters.
While this may sound like a theoretical exercise, another main contribution in this paper shows how to realize it practically.
This is where the second insight comes in, which is based on the recently introduced Testing Bayesian Networks (TBNs) (Choi & Darwiche, 2018; Choi et al., 2019).
These models were introduced to address the expressiveness gap between functions induced by Bayesian networks (BNs) and those induced by neural networks.
The ﬁrst class of functions are multilinear (or their quotients) while the second class are universal function approximators (e.g., piecewise multilinear for neural networks with ReLU activation functions).
A TBN is like a BN except that some of its nodes can select their CPTs dynamically based on evidence, which allows TBNs to induce universal function approximators like neural networks.
What we show is that the evidence-based
CPT selection process in TBNs can be used to implement the recovery approach that we develop theoretically.
Beyond showing how our recovery proposal can be realized practically, this also provides a theoretical basis for the utilization of TBNs in recovering from modeling errors which has so far been only shown empirically (Shen et al., 2019).
This paper is organized as follows.
We review Testing
Bayesian Networks in Section 2 and discuss the casting of certain modeling errors as state-space abstractions in Section 3.
We then present our approach for recovering from modeling errors in Section 4 and show how this approach can be implemented using Testing Bayesian Networks in Section 5.
We present empirical results in Section 6 and close with some concluding remarks in Section 7.
Proofs of theorems can be found in the supplementary material.
2. Testing Bayesian Networks
We use uppercase letters (e.g., X ) to denote variables and lowercase letters (e.g., x ) to denote their states.
We use bold uppercase letters (e.g., X ) to denote sets of variables and bold lowercase letters (e.g., x ) to denote their instantiations.
A Bayesian Network (BN) over variables X is a pair ( G, Θ) where G is a directed acyclic graph (DAG) over variables
X and Θ is a set of Conditional Probability Tables (CPTs).
Each node X ∈ X with parents U has one CPT Θ X
| U which speciﬁes a conditional distribution θ ( X | u ) for each instantiation u of parents U .
A BN represents a joint distribution over variables X (Pearl, 1989).
Testing Bayesian Networks (TBNs) were introduced recently as a generalization of BNs by allowing CPTs to be chosen dynamically based on the evidence available at inference time (Choi & Darwiche, 2018; Choi et al., 2019).
TBN nodes can be regular or testing.
Consider a node X with parents U .
If X is regular, then it has a regular CPT Θ X | U as in a BN (root nodes of a TBN must be regular).
If X is testing, it has a testing CPT Θ tX | U which speciﬁes multiple conditional distributions { θ k
( X | u ) } Nk =1 for each parent instantiation u .
These distributions are associated with intervals { I k u } Nk =1 that partition the interval [0 , 1] .
At inference time, one distribution from { θ k
( X | u )
} Nk =1 is selected based on the available evidence e .
In particular, distribution θ k
( X | u ) is selected if the posterior Pr i ( u | e i ) (explained next) belongs to the interval I k u .
This process is repeated for every parent instantiation u and the selected distributions are then assembled into a regular CPT for node
X .
When the CPT selection process concludes, the TBN is turned into a BN that is then used for inference under evidence e .
If the evidence changes, a new BN is selected and used for inference under the new evidence.
The BN selected under evidence e is constructed incrementally by selecting CPTs according to a topological order π = X 1 , X 2 , . . .
, X n of TBN nodes.
Order π deﬁnes DAGs
G 1 , . .
G n +1 where G 1 is empty and G i +1 is obtained by adding node X
i to G i (so G n +1 is the original DAG G ).
We start with the empty DAG G 1 and add root node X 1 with its regular CPT to yield BN ( G 2 , Θ 2 ) that induces distribution Pr 2 .
When adding node X i for i > 1 , we already have a BN ( G i , Θ i ) which induces distribution Pr i ( . ) and contains parents U i of X i .
If node X i is testing, we select its
CPT based on the posterior Pr i ( u i | e i ) computed using BN ( G i , Θ i ) .
Otherwise, we use the regular CPT of node X i .
To completely deﬁne the semantics of TBNs, we now turn to the deﬁnition of e i which we call the selection evidence.
There are three deﬁnitions of this evidence, all of which ensure that evidence variables E i are in DAG G
i (Choi &
Darwiche, 2018; Choi et al., 2019) deﬁned e i as the subset of evidence e pertaining to ancestors of node X i in DAG G .
This was sufﬁcient to show that TBNs are universal function approximators like neural networks.
(Shen et al., 2019) deﬁned e i as the subset of evidence e that excludes evidence at or below testing nodes that are not ancestors of X i .
It further required order π to place non-testing nodes before testing nodes if possible, 5 which maximized the selection evidence If Pr ( . ) is the distribution induced by the selected BN, then Pr i ( u i | e i ) = Pr ( u
A
B
C (a) BN/TBN * * * * P (cid:63) (¯ b ) P (cid:63) ( b ) + + λ ¯ a θ b | a θ b | ¯ a θ ¯ a θ a θ ¯ c | a θ ¯ c | ¯ a θ c | ¯ a θ c | a λ ¯ c λ c θ ¯ b | ¯ a θ ¯ b | a λ a * * * * * * * * + + (b) AC *  ? : + *  ? :
*  ? :  ? :
* *  ¯ a P ? (¯ b ) ✓  ¯ b | a ✓  b | ¯ a ✓ + b | ¯ a * ✓ ¯ a ✓ a ✓ +¯ b | ¯ a ✓  ¯ b | ¯ a + ✓ ¯ c | a ✓ ¯ c | ¯ a ✓  b | a + ✓ c | ¯ a ✓ c | a ✓ + b | a P ?
( b ) T B | a T B | ¯ a  ¯ c  c ✓ +¯ b | a  a * * * * * * * * + + (c) TAC while keeping the selected BN invariant to the used topological order π . This increased the expressiveness of TBNs as it made CPT selection more dependent on evidence e . In this work, we also require order π to satisfy this property but deﬁne e i as the subset of evidence e pertaining to DAG G i .
Our deﬁnition increases the selection evidence and TBN expressiveness even further, but it makes the selected BN sensitive to the used order π .
This additional expressiveness of TBNs is needed for recovering form a certain class of modeling errors as we show in Section 5.
As mentioned earlier, we are interested in a supervised learning setting where the goal is to learn a function that computes query Pr ( Q | e ) based on a BN ( G, Θ) (this function can be used, for example, to classify Q based on evidence e ).
This can be realized by compiling the BN into an Arithmetic Circuit (AC)
(Darwiche, 2003), whose parameters Θ are then learned using gradient descent as shown in (Darwiche, 2020); see Figure 1b.
However, since DAG G may be incomplete due to missing edges or states, we will instead compile a TBN ( G, Θ t ) into a Testing Arithmetic Circuit (TAC) (Choi & Darwiche, 2018; Choi et al., 2019) and learn its parameters Θ t using gradient descent; see Figure 1c.
This effectively learns a set of BNs, one of which is selected based on evidence e when computing the query Pr ( Q | e ) .
As we show theoretically in Section 5 and empirically in Section 6, this can help in recovering from some of the modeling errors introduced when building DAG G .
3. Modeling Errors as State-Space Abstractions
We now show that missing certain edges or states when building a Bayesian network structure can be cast and analyzed in terms of state-space abstraction.
This will aid us later when discussing recovery from such modeling errors.
To abstract the state-space of node X in a BN is to merge some states of X .
This technique was explored in (Chang & Fung, 1990; Wellman & Liu, 1994) and can be used, for example, to speed up inference.
Following these works, we call such a node X an abstracted node.
We also refer to the original states of X as elementary and the new states as superstates.
A state-space abstraction can be speciﬁed by a pair a = ( sup, elm ) where sup ( x ) is the superstate of elementary state x and elm ( x (cid:48) ) are the elementary states of superstate x (cid:48) .
We deﬁne these functions for all nodes so if node X is not abstracted, we have sup ( x ) = x and elm ( x ) = { x } .
We also extend state-space abstractions to sets of variables as follows: sup ( x 1 , . . .
, x n )
= sup ( x 1 ) , . .
, sup ( x n ) and elm ( x (cid:48) , . . .
, x (cid:48) n ) =
elm ( x (cid:48) )
× . . .
× elm ( x (cid:48) n ) .
We often write x ∈ x (cid:48) as a shorthand for x ∈ elm ( x (cid:48) ) .
We will attach a state-space abstraction a = ( sup, elm ) as a superscript to various objects as needed.
For example, x a is an abstracted state (instantiation) of variables X , G a is a DAG G with some abstracted nodes, and Pr a ( . ) is a distribution over the abstracted space of distribution Pr ( . ) .
One ideally wants to preserve consistency across distributions when abstracting nodes of a BN.
Deﬁnition 1.
Distributions Pr ( . ) and Pr a ( . ) over variables
X are said to be ‘consistent
’ iff Pr a ( x a ) =
(cid:80) x ∈ x a Pr ( x ) .
Consistency ensures that we can use the abstracted distribution Pr a to answer any query about Pr as long as the query does not reference abstracted nodes.
The key question is whether this is possible when abstracting some nodes of a
BN ( G, Θ) .
That is, can we revise CPTs Θ into some new
CPTs Θ a of the abstracted DAG G a so that the two BNs are consistent?
This would be signiﬁcant if possible as it implies that we can recover from state-space abstractions (and some modeling errors) by potentially learning the proper
CPTs Θ a .
This is generally not possible though.
To see this, consider the BN in Figure 2 and suppose node X has three states x 1 , x 2 and x 3 .
Any distribution induced by this BN must ﬁnd Y independent of U given X and V .
If we abstract node X so it has superstates x 1 and { x 2 , x 3 } , then any distribution induced by the abstracted BN must also satisfy this independence, including when X = { x 2 , x 3 } .
Yet this independence may not hold in the BN distribution: knowing the state of V and knowing only that the state of
X is in { x 2 , x 3 } may not make Y independent of U .
The following local but strong condition does, however, guarantee the sought consistency.
Deﬁnition 2.
A state-abstraction a = ( sup, elm ) is ‘harmless’ for a BN ( G, Θ) if for every node X with parents
U
we have θ ( X | u )
=
θ ( X | u (cid:63) ) when sup ( u ) =
sup ( u (cid:63) ) .
This condition says that once we know a superstate of parents U , then X becomes independent of the elementary states of U .
This somewhat generalizes the classical notion of context-speciﬁc independence (Boutilier et al., 1996).
Theorem 1.
If a is a harmless state-abstraction for BN ( G, Θ) , then there is a BN ( G a , Θ a ) which induces a distribution Pr a ( . ) that is consistent with Pr ( . ) .
The condition of Deﬁnition 2 is quite strong but the notion of consistency is also quite strong as it allows one to correctly compute any query using the abstracted BN as long as the query does not reference abstracted nodes.
As we show in
Section 4, one can signiﬁcantly weaken this condition if one is only interested in recovering speciﬁc queries, which is all that that one needs in a supervised learning context.
A modeling error that misses some states of variables can be naturally cast as a state-abstraction of these variables.
We now show that missing certain edges can also be cast in these terms.
Our formulation uses the following deﬁnition.
Deﬁnition 3.
Let G be a DAG which results from adding some edges to DAG G (cid:48) and abstracting some of its nodes.
Then G is ‘reducible’ to G (cid:48) iff for every BN ( G, Θ) there is a
BN ( G (cid:48) , Θ (cid:48) ) where distributions Pr and Pr (cid:48) are consistent.
Recall that our interest is to learn distributions through learning the parameters of a Bayesian network structure.
This deﬁnition says that any distribution (of interest) which can learned using DAG G can also be learned using DAG G (cid:48) .
[Figure] Suppose now that we have a true structure G 1 (datagenerating model) and a structure G 2 (used for learning from data) that misses some edges from G 1 .
We will next show how to construct a third structure G 3 that has the same topology as G 2 but more states than G 2 while guaranteeing that G 1 is reducible to G 3 ; that is, using G 3 does not preclude one from learning the data-generating model G 1 .
This will then show that if one can recover from having used Ginstead of G 3 (state-space abstraction) then one can recover from having used G 2 instead of G 1 (missing edges).
For a concrete example, consider Figure 3 which depicts a true structure G 1 and a structure G 2 that misses two edges in
G 1 .
We also have structure G 3 which has the same topology as G 2 except that we added more states to variables B and
C : the state-space of B was multiplied by the state-space of A and the state-space of C was multiplied by the state-space of B .
The true structure G 1 is reducible to structure G 3 .
Moreover, structure G 2 is the result of abstracting some nodes of G 3 .
Hence, if one can recover from the state-space abstraction that led to G 2 (from G 3 ) then one can recover from the missing edges that led to G 2 (from G 1 ).
For missing edges, we only treat what we call high-order edges U → X for which there exists some directed path from U to X other than edge U → X .
We next show how the auxiliary DAG G 3 is constructed as in Figure 3.
Deﬁnition 4.
To ‘harmlessly delete’ high-order edge U →
X from DAG G : remove edge U → X from G ; choose a directed path U
→ Z 1 → · · · → Z n → X ; and ﬁnally expand the state-space of each node Z i to the Cartesian product of its original state-space and the state-space of U .
Theorem 2.
If DAG G 2 is the result of harmlessly deleting high-order edges from DAG G 1 , then G 1 is reducible to G 2 .
The reduction of high-order HMMs to ﬁrst-order ones (He, 1988) is a special case of this theorem as one can obtain the latter by harmlessly deleting high-order edges from the former.
Hence, one implication of our upcoming treatment is that TBNs can be used to recover from a modeling error that uses a ﬁrst-order HMM instead of a high-order one.
(Shen et al., 2019) provided some preliminary empirical results to this effect but our upcoming results are analytic.
4. Recovering from State-Space Abstractions
We now address the question of revising the CPTs of a BN after abstracting some of its nodes, where the goal is to maintain the ability to compute some queries using the abstracted BN.
We start with the proposal of (Wellman & Liu, 1994) which we call the soft policy.
For a node X with parents U this policy deﬁnes the revised CPTs as follows:
[Equation]
If node X and its parents are not abstracted, we get θ a ( x a | u a )
= θ ( x | u ) so the CPT of X is not revised.
Consider Figure 2 and suppose that node X is the only abstracted node.
Then only the CPTs of X and its child Y are revised.
(Wellman & Liu, 1994) proposed another, less expensive CPT revision policy, which we call the average policy:
[Equation]
While these policies may appear natural, the abstracted distributions Pr a they produce are generally not consistent with Pr .
This should not be surprising for the reasons we discussed earlier:
aggregating states induces additional independencies that may not hold in the original BN.
We will address this challenge based on two main insights.
First, we will aim to only preserve Pr ( Q | e ) for some query node Q and evidence variables E ; as mandated by supervised learning.
Second, we will revise CPTs dynamically based on the available evidence e (recall how TBNs select CPTs).
We will next identify a particular class of modeling errors and queries for which these insights will lead to full recovery.
Our treatment assumes that we are recovering from having abstracted a polytree BN; that is, a BN in which two nodes can be connected through at most one (undirected) path (Pearl, 1989).
While a polytree has a restricted structure, recall that it may be the result of having missed edges so the data-generating BN does not have to be a polytree.
That is, we are treating modeling errors that lead to a polytree structure not ones in which the true BN is a polytree.
This includes, for example, learning using a ﬁrstorder HMM (polytree) when the data-generating model is a high-order HMM (not a polytree).
The queries Pr ( Q | e ) we shall preserve are ones for which evidence variables E
[Figure]
Node e πX 1,  1,2,  1,2,3,4,12,13,  3,
[Figure]
are not abstracted and the query node Q is a descendant of abstracted nodes (hence, node Q is not abstracted either).
Our approach amounts to the following.
For each node X with parents U , we identify some evidence e πX (explained later) and set the CPT of X to Pr ( x a | u a , e πX ) .
The abstracted polytree with these revised CPTs will then preserve the query Pr ( Q | e ) .
Hence, the CPTs of an abstracted polytree are evidence-speciﬁc: if we change evidence e we have to reassign these CPTs accordingly.
In Section 5, we show how this can be realized using TBNs.
To fully specify our approach, all we need now is to deﬁne evidence e πX which requires an ordering π of the polytree nodes that depends on query node Q .
There may be more than one ordering π that satisﬁes the properties we shall state, each leading to a different evidence e πY .
Yet, our main result in this section will show that any such ordering is guaranteed to preserve the query Pr ( Q | e ) .
Deﬁnition 5.
Let G be a polytree, Q be a node in G and
U be some ancestor of Q .
A child X of U will be called a ‘ Q -child’ of U iff X
=
Q
or X is an ancestor of Q .
If a node has a Q -child then it must be unique.
This follows from G being a polytree.
Figure 4a depicts a polytree where node 10 is the unique Q -child of abstracted nodes 6 and 7.
Deﬁnition 6.
Let G be a polytree and Q be a node in G .
A ‘ Q -ordering’ π of G is a total ordering of the nodes in G that places each Q -child after its siblings.
A Q -ordering always exists.
This follows because if a node
X is the Q -child of two of its parents, then X must be the only common child of these parents so the children of these parents can be ordered independently.
Consider the polytree in Figure 4a where node 14 is the query node.
The following are two Q -orderings for this polytree.
π= 1 , 2 , 3 , 5 , 6 , 7 , 4 , 8 , 12 , 9 , 13 , 11 , 15 , 10 ,π= 1 , 2 , 3 , 5 , 6 , 7 , 4 , 9 , 13 , 11 , 15 , 8 , 12 , 10 ,
We are now ready to deﬁne the evidence e πX which utilizes the following standard notation for partitioning evidence e across a polytree edge U → X (Pearl, 1989).
e + UX : evidence on the U -side of edge U → X e − UX : evidence on the X -side of edge U → X e − X : evidence on the descendants of node X e + X : evidence on the non-descendants of node X
Deﬁnition 7.
Let G be a polytree with query node Q , evidence e and Q -ordering π .
If node X has a parent U that is an ancestor of Q , we deﬁne e πX (cid:44) e + X \ { e − UY | Y is a child of U and π ( Y ) >
π ( X ) } .
If node X has multiple parents U that are ancestors of Q , then X must be the Q -child of both parents and hence ordered last among its siblings.
This leads to e πX =
e + X regardless of which parent U we pick in the above deﬁnition.
The deﬁnition of evidence e πX depends on the Q -ordering π though.
Consider the polytree in Figure 4a and the two
Q -orderings π 1 and π 2 shown earlier.
For node X =in Figure 4a, we have e π 1 X = { 1 , 2 } and e π 2 X
= { 1 , 2 , 13 } .
However, if two Q -orderings π 1 and π 2 agree on how they order the siblings of Q -children, then e π 1 X =
e π 2 X .
We are now ready to deﬁne our evidence-based policy for assigning CPTs after abstracting nodes.
Deﬁnition 8.
Consider a polytree BN with distribution
Pr .
For query Pr ( Q | e ) , Q -ordering π and state-space abstraction a , assign CPTs for each node X with parents U as follows.
If U contains abstracted nodes, then θ a ( x a | u a ) =
Pr ( x a | u a , e πX ) else θ a ( x a | u a ) =
Pr ( x a | u a ) .
The following is our main result showing that the above
CPT assignment policy will preserve the query of interest.
Theorem 3.
Consider a polytree BN with distribution Pr , a query Pr( Q | e ) and a state-space abstraction a where abstracted nodes are ancestors of Q and do not overlap with E . If Pr a is the distribution of the BN abstracted according to Deﬁnition 8, then Pr a ( Q | e ) = Pr ( Q | e ) .
For a concrete example, consider the polytree BN E ←
R → Q with the following CPTs (variables E and Q are binary, variable R has three states): [Figure] This BN induces the following query, which maps values of evidence variable E to distributions on query variable Q : [Figure]
{+Suppose that we abstract node R by merging its states rand r 
+}
so it now has two superstates r 1 and r 23 .
Using
Q -ordering π = R, E, Q we get E πE =
{} and E πQ =
{ E } so the selection evidence for node E is empty.
According to Deﬁnition 8, the abstracted BN will have one CPT for each of variables R and E : [Figure] e 0.
However, variable Q will have two CPTs, one of which is chosen depending on the available evidence on variable E :
R a
Q Θ 1 Q | R a rq 0.6 r¯ q 0.4 rq 0.5 r¯ q 0.
R a
Q Θ 2 Q | R a rq 0.6 r¯ q 0.4 rq 0.66 r¯ q 0.
CPT Θ 1 Q | R a =
Pr ( Q | R a , e ) is used under evidence e and
CPT Θ 2 Q | R a =
Pr ( Q | R a , ¯ e ) is used under evidence ¯ e .
One can verify that this abstracted BN will induce the same query as the original BN, Pr a ( Q | E ) =
Pr ( Q | E ) .
5. Evidence-Based Recovery Using TBNs
As discussed in Section 3, state-space abstraction encapsulates modeling errors in the form of missing states and edges.
Theorem 3 tells us that we can recover queries Pr ( Q | e ) from state-space abstractions and hence from these modeling errors by choosing the CPTs of a Bayesian network based on evidence e (under the stated conditions).
Focusing on a particular query is not that restrictive as it is the mark of supervised learning where the goal is to learn functions, such as the mapping from evidence e into a posterior
Pr ( Q | e ) .
On the other hand, the selection of CPTs based on evidence may seem like a theoretical exercise except that it is not.
The recently introduced TBN does exactly this (Choi & Darwiche, 2018; Choi et al., 2019).
As discussed in Section 2, a TBN is a collection of BNs that share the same structure but differ only in their CPTs.
One of these BNs is selected based on evidence and then used to answer queries where the selection process amounts to choosing the BN
CPTs.
This raises the following question: Can a TBN emulate the CPT selection policy discussed in Deﬁnitionwhich guarantees full recovery from state-space abstraction under some conditions?
The multiple CPTs associated with a node in a TBN are meant to be learned from labeled data using methods such as gradient descent.
This raises another question: Can the learned CPTs of a TBN reach the promise of Theorem 3?
We tackle the ﬁrst question next while leaving the second question to the next section.
Recall from Section 2 that a TBN has two types of nodes: regular and testing.
A testing node X with parents U has a set distributions θ 1 ( X | u ) , . . .
, θ N ( X | u ) for each parent instantiation u , in addition to a partition I 1 u , . . .
, I N u of the interval [0 , 1] .
Before answering a query under evidence e , the TBN computes a posterior p on parent instantiation u and then chooses distribution θ i ( X | u ) if p ∈
I i u .
We next discuss how this selection mechanism can be used to emulate the CPT selection process of Deﬁnition 8.
The main insight is to make each child of an abstracted node a testing node, which allows its CPT to be chosen based on evidence.
One subtlety with TBNs is that parent posteriors cannot be computed before certain CPTs have been selected.
In particular, before computing the parents posterior for node
X one needs to at least select the CPTs for testing nodes that are ancestors of X .
As discussed in Section 2, this is accomplished by using a topological ordering X 1 , . . .
, X n of TBN nodes to create a sequence of DAGs G 1 , . . .
, G n +1 , where DAG G i +1 results from adding node X
i to DAG G i .
The CPT of a testing node X
i is selected when adding node X i to DAG G i since at that point the CPTs of all nodes in G i have been selected.
Moreover, parents U i of X i are already in G
i so we can compute
Pr i ( U i | e i ) using the distribution Pr i ( . ) of BN G i .
By the time variable X n is added, we get BN G n +1 which is used to answer queries under evidence e .
Two observations are relevant to the quest of this section.
First, the evidence e i used to compute the posterior
Pr i ( U i | e i ) is the subset of evidence e pertaining to BN
G i so selecting the CPT of node X i cannot use all available evidence e .
Second, the selection of this CPT is based only on the value of posterior Pr i ( U i | e i ) .
Do these restrictions limit the ability to emulate the CPT selection process of
Deﬁnition 8?
As we show next, the ﬁrst restriction does not but the second restriction may, yet we identify a condition under which it does not.
If the topological ordering π = X 1 , . .
X n used to select the CPTs of a TBN satisﬁes the condition of Deﬁnition 6, then evidence e i will be a superset of evidence
e πX i needed by Deﬁnition 8.
Moreover, one can always construct such a topological ordering π .
This settles the ﬁrst question.
As to the second question, to fully emulate Deﬁnition 8, one may need as many intervals I 1 u i , . .
, I N u i as the number of instantiations for evidence variables E πX i which is not possible practically.
Moreover, two distinct instantiations of variables E πX i may lead to the same posterior Pr i
( U i | e i ) so a TBN cannot fully emulate Deﬁnition 8.
Even if it can, this would not be realized if we bound the number of intervals N and, hence, the number of CPTs associated with a testing node.
However, we present experiments in the next section which show that a TBN can recover queries accurately even with a relatively small N . [Table]
For the ﬁrst set of experiments, we calculate the TBN CPTs using the same procedure for selecting TBN CPTs except that we now construct the space of CPTs instead of selecting CPTs from a given space.
We ﬁrst ﬁx a number of intervals
N .
When adding node X i to BN G i , we create buckets
E 1 u i = ∅ , . . .
, E N u
i = ∅ for every instantiation u i of X
i ’s parents.
We then enumerate every possible instantiation e πX i of evidence variables E πX i .
We compute the posterior p =
Pr i ( u i | e i ) and add instantiation e πX i to bucket E k u i
if p ∈
I k u i .
After processing all evidence instantiations e πX i , we associate the following distribution with interval I k u i :
[Equation]
where η is a normalizing constant.
This method, which we call the TBN policy, divides the evidence space into N buckets E 1 u i , . .
, E N u i based on which interval the parents posterior falls into.
It then computes a weighted average of the distributions in each bucket.
If some bucket E k u i has more than one distribution, we say we have a CPT collision.
In the absence of CPT collisions, the TBN policy fully emulates the CPT selection process of Deﬁnition 8.
We can view the number of intervals N as a tradeoff between recovery quality and memory cost.
When N = 1 , the TBN policy degenerates to the soft policy (Equation 1).
As N approaches the size of the evidence space, the TBN will fully recover the query Pr ( Q | e ) if no CPT collision happens.
6. Experiments
We now empirically evaluate the techniques we proposed in
Sections 4 and 5.
We consider two tasks: model compression and supervised learning .
Both tasks try to recover a function Pr ( Q | E ) generated by a true BN ( G, Θ) but using an incomplete BN structure G (cid:48) that may be the result of a modeling error.
The recovery is accomplished using a TBN with structure G (cid:48) .
In the ﬁrst task, we handcraft TBN parameters (CPTs and intervals) based on the true BN and the policy of Section 5 (see Equation 3).
The goal is to assess how well can a TBN reach the promise of full recovery from modeling errors using a ﬁnite number of intervals.
In the second task, we assess whether learning TBN parameters, instead of handcrafting them, can potentially reach the promise of full recovery by comparing the learned function with the handcrafted and true ones.
This experiment is somewhat limited in scope since learning TBN parameters is still in its infancy and we do not yet have learning algorithms that are robust enough. This task is quite challenging as we are trying to recover from modeling errors based on data and without knowing the nature of these modeling errors.
We use the conditional KL-divergence to measure the similarity between the true function Pr ( Q | E ) and the learned or handcrafted one Pr (cid:48)
( Q | E ) :
[Equation]
6.1. Model Compression
For this experiment, we randomly generate polytree BNs and queries Pr ( Q | E ) .
We simulate a modeling error by abstracting some nodes of the polytree to yield an incomplete structure G (cid:48) .
We then parameterize G (cid:48) according to the soft policy (Equation 1) and average policy (Equation 2) to yield two BNs that we call BN SOFT and BN AVG.
We also parameterize structure G (cid:48) according to the TBN policy (Equation 3) to yield a TBN.
We ﬁnally compare the three functions Pr (cid:48)
( Q | E ) induced by the two BNs and TBN.
We do this by measuring their proximity to the ground-truth function Pr ( Q | E ) induced by the true BN model.
We use the method of (Ide & Cozman, 2002) to generate random polytree structures.
We randomly choose a leaf node to be query Q and 20% of the nodes to be evidence nodes E .
We assign 5 states to all ancestors of Q and 2 states to other nodes.
CPTs are generated by randomly assigning one state a probability ≥ .
8 and assigning the remaining probabilities uniformly.
To generate the incomplete structure G , we abstract all ancestor of query node Q by reducing their cardinality to 2 .
When parameterizing this structure to yield [Table] [Table] a TBN, we make every child of an abstracted node a testing node and use N evenly-spaced intervals for selecting a CPT.
Table 1 reports the conditional KL-divergence against the true BN.
As we can see, the TBN yields much better recovery compared to both BN baselines and is close to the ground-truth BN when the number of intervals N is large enough.
For N = 2 , the TBN improves the KL-divergence by 10-20% compared to BN SOFT.
For N = 50 , the TBN signiﬁcantly improves the recovery quality by 50-90%.
Figure 5 shows the quality of TBN recovery for up to N = 100 .
6.2. Supervised Learning with Missing States
The setting for this task is similar to the previous task except that we now learn BN/TBN parameters from data using an incomplete structure G (cid:48) (with abstracted nodes).
First, we randomly generate polytree BNs and queries Pr ( Q | E ) as discussed earlier.
Next, we generate labeled datasets (cid:104) E , Q (cid:105) from the BNs (true models), each including 100 × n examples where n is the number of distinct evidence instantiations.
Finally, we compile the incomplete structure G (cid:48) and query into an AC (Arithmetic Circuit) and into a TAC (Testing Arithmetic Circuit) using the P Y TAC system (Darwiche, 2020).
These circuits are computation graphs, in the form of tensor graphs, which compute the query based on a BN (AC) or a TBN (TAC), therefore facilitating the learning of their parameters using gradient descent. [Table]
We conduct two experiments for this task.
In the ﬁrst, we compare learned TBNs with handcrafted ones.
In the second, we compare learned TBNs with learned BNs.
When learning TBN parameters, we also learn the interval boundaries instead of using equally-spaced intervals.
Table 2 reports the results of the ﬁrst experiment for N = 8 .
When the network size is small, the learned TBN does not recover the query as well as the handcrafted one, but the learned TBN improves with increasing network size and dominates the handcrafted TBN with 25 or more nodes.
Recall that for the handcrafted TBN, only children of abstracted nodes are testing so only these children have dynamic CPTs; all other nodes have ﬁxed CPTs copied from the true BN.
For the learned TBN, we make every node testing except for root nodes as we assume that we do not know the nature of modeling errors.
As a result, the learned
TBN has more degrees of freedom than the handcrafted one.
Table 3 depicts the results of the second experiment, where we use a number of intervals N ∈ { 2 , 4 , 6 , 8 } and report the best function Pr (cid:48)
( Q | E ) learned by a TBN.
Our learning algorithm is not optimal or robust enough so the quality of learned functions was sometimes better for a smaller N .
Yet, as shown by Table 3, a TBN with learned parameters holds the promise of recovering from modeling errors as in this case it did lead to 20-50% reduction in the conditional KL-divergence from the true function.
That is, learning a set of distributions based on an incomplete structure (and then using one of them based on evidence) can allow one to learn better functions compared to learning a single distribution which is the case when learning a BN.
6.3. Supervised Learning with Missing Edges
We ﬁnally consider recovery from modeling errors in the form of missing edges.
The true BN in this experiment is not a polytree, but becomes a polytree due to missing edges.
We assume the data is generated by an n -th order HMM (true model) but learn using a 1st-order HMM.
We use HMMs with ten hidden variables H 1 , . . .
, H 10 and ten sensors
E 1 , . . .
, E 10 and learn the function Pr ( H 10 | E 1 , . . .
, E 9 ) .
We experiment with 2nd-order and 3rd-order HMMs as the true model, each with variables of cardinality 2 - 4 (six combinations).
The dataset size was 512 for cardinality 2 and 20 , 000 for cardinalities 3 and 4 .
We use a uniform initial distribution.
For the emission distribution Pr ( E t | h t ) , we use a random p ≥ 0 .
95 for e t = h t and uniformly distribute 1 − p over e t (cid:54) =
h t .
For the transition distribution
Pr ( H t | h t − 1 , . . .
, h t − n +1 ) , we pick a state h t randomly and assign it a probability p ≥ 0 .
We repeat this to assign a probability ≥ .
8 ∗ (1 − p ) for a second state and so on.
Table 4 reports the quality of learned functions using a BN (static CPTs) and a TBN (dynamic CPTs).
Again, we see that the TBN can help in recovering from this modeling error, as it did lead to gains up to 77% in the conditional KL-divergence in this case.
(Shen et al., 2019) reported related experiments but using more restricted transition distributions.
We obtain broader results as we use more than two CPTs per testing node.
We now also have a theoretical basis for justifying the ability of TBNs to recover from such modeling errors.
7. Conclusion
We investigated the recovery from some modeling errors when learning probabilistic queries based on Bayesian network structures.
These modeling errors induce independence assumptions that may not hold in the data-generating function, which contributes to bias.
We cast these modeling errors in terms of state-space abstractions and provided conditions under which one can fully recover from these errors, assuming one can choose the CPTs of a Bayesian network dynamically based on evidence.
We then showed how Testing Bayesian Networks can be used to realize the recovery proposal which revealed their promise as a tool for recovering from modeling errors.
In future work, we plan to weaken the conditions under which full recovery can be attained, cast more modeling errors in terms of state-space abstractions and improve the learning of TBNs to further approach the promise of theoretical recovery results.
Acknowledgements
This work has been partially supported by grants from
NSF IIS-1910317, ONR N00014-18-1-2561, and DARPA
N66001-17-2-4032.