Learning Tractable Probabilistic Models from Inconsistent Local Estimates
Abstract
Tractable probabilistic models such as cutset networks which admit exact linear time posterior marginal inference are often preferred in practice over intractable models such as Bayesian and Markov networks.
This is because although tractable models, when learned from data, are slightly inferior to the intractable ones in terms of goodness-of-fit measures such as log-likelihood, they do not use approximate inference at prediction time and as a result exhibit superior predictive performance.
In this paper, we consider the problem of improving a tractable model using a large number of local probability estimates, each defined over a small subset of variables that are either available from experts or via an external process.
Given a model learned from fully-observed, but small amount of possibly noisy data, the key idea in our approach is to update the parameters of the model via a gradient descent procedure that seeks to minimize a convex combination of two quantities: one that enforces closeness via KL divergence to the local estimates and another that enforces closeness to the given model.
We show that although the gradients are NPhard to compute on arbitrary graphical models, they can be efficiently computed over tractable models.
We show via experiments that our approach yields tractable models that are significantly superior to the ones learned from small amount of possibly noisy data, even when the local estimates are inconsistent.
Introduction
A major issue with probabilistic graphical models (PGMs) [3, 8] such as Bayesian and Markov networks is that in most real-world domains, exact inference is computationally infeasible.
As a result, approximate inference algorithms are often used in practice, but these algorithms can be inaccurate and may exhibit high variability.
Thus, even if a PGM performs well on goodness-of-fit measures such as test set log-likelihood, its predictive performance, because of the use of approximate inference methods, can be quite poor.
To combat this problem the field of tractable probabilistic models (TPMs) or probabilistic circuits (PCs) [9, 11, 20] seeks to learn probabilistic models that admit polynomial time exact inference algorithms.
TPMs are more biased or less expressive than general probabilistic models (because they enforce tractability) and as a result may yield a slightly inferior (model) fit.
But, the hope is that because no approximations are used at prediction time the slight drop in goodness-of-fit measures is offset by superior predictive performance.
This hope is often realized in practice, and demonstrated in numerous empirical studies in literature (for example, see [18, 20]).
To date, algorithms for learning the structure and parameters of tractable probabilistic models such as cutset networks [17], sum-product networks [14] and probabilistic sentential decision diagrams [9] assume access to either full data which has no missing values or almost full data in which only a few variables have missing values.
Often in the real world, however, the following scenario is quite common.
The learning algorithm has access to a small amount of full, possibly noisy data and a large
(NeurIPS number of local marginal probability estimates that are derived from some combination of local data, imperfect domain knowledge and local, independent predictive models.
For instance:
• Due to privacy concerns in application domains such as social networks [7, 22], only limited global data is available.
But local statistics, such as information about a person’s contacts/connections can be retrieved easily.
• In lazy learning of generative models [18, 26, 27], we derive sufficient statistics needed for inducing a probabilistic model at test time (when a query is made) from various sources such as local classifiers for each statistic or via a query made to a large database.
• In active learning [23], the learning algorithm interactively solicits a user for labels or in general local marginal probability distributions for certain variables that the user is an expert at given observations.
A hallmark of all the scenarios just outlined is that the local marginal estimates are often inconsistent , namely, there may not exist a joint probability distribution that is consistent with the local estimates.
For example, in the active learning case, consider two pairwise marginal estimates P A ( x 1 , x 2 ) and P B ( x 2 , x 3 ) over three random variables
X 1 , X 2 , X 3 derived by querying two users A and B respectively.
A joint probability distribution P ( x 1 , x 2 , x 3 ) over the three variables that is consistent with the two estimates exists iff P A ( x 2 ) =
P B ( x 2 ) where P A ( x 2 ) =
(cid:80) x 1 P A ( x 1 , x 2 ) and P B ( x 2 ) =
(cid:80) x 3 P B ( x 2 , x 3 ) .
Unfortunately, because of factors such as precision errors and user bias, P A ( x 2 ) will not be equal to P B ( x 2 ) .
In this paper, we focus on the following learning problem over TPMs.
We assume that the learner has access to: (1) a TPM learned from a small dataset having few or no missing variables, and (2) a set containing local estimates where each estimate is a marginal probability distribution over a small subset of variables given observations.
Contributions.
We propose to express the above learning task as the following minimization problem: update the parameters of the given tractable model such that a linear combination of the following two KL distances is minimized: (1) the distance between the distribution represented by the original parameters and the one represented by the updated parameters; and (2) the sum of the distances between the given local, possibly inconsistent estimates and the ones computed using the updated distribution.
We derive a gradient-based method for solving this optimization task.
Since the gradients require computation of marginal distributions over subsets of variables, they are NP-hard in general on arbitrary probabilistic models but can be computed efficiently on tractable models [24].
This shows the virtue of using tractable models in our learning settings.
We performed a controlled empirical evaluation of our proposed method using 20 popular datasets that have been used in numerous studies on tractable models [10].
Our results show that our approach that leverages local estimates yields significant improvements in both generative and predictive performance over the original model learned from small amount of training data, even when the local estimates are inconsistent.
Moreover, since the optimization problem is smooth, our procedure is guaranteed to reach a local optima under mild conditions.
Related Work
Vomlel [25] studied the problem of integrating probabilistic knowledge bases where a joint probability distribution is constructed from low-dimensional probability distributions (local estimates).
Vomlel used a classic optimization method called iterative proportional fitting procedure (IPFP) [6] and proposed a variant called the generalized expectation maximization algorithm (GEMA) for solving this problem.
Vomlel provided convergence proofs for these methods; showing that IPFP converges when the local estimates are consistent and GEMA converges even if the local estimates are inconsistent.
Unfortunately, Vomlel’s approach has high computational complexity (is exponential in the treewidth of the graph defined over the local estimates) and is not practical.
The method proposed in this paper does not have this limitation.
Peng and Ding [13] proposed two polynomial time approximations for
IPFP and applied them to Bayesian networks.
However, their preliminary experimental study demonstrates that the error due to their approximations is quite high and convergence is not guaranteed if the local estimates are inconsistent.
In contrast, our proposed method makes very few approximations and leverages tractable inference to yield a practical scheme.
Notation and Background
We use the following notation.
Bold upper-case letters (e.g., X , Y , U , etc.) are used to denote sets of discrete random variables while bold lower-case letters (e.g., x , y , etc.)
denote an assignment of values to all variables in the corresponding set denoted by bold upper case letters (thus x
and y denote an assignment to all variables in the set X and Y respectively).
We use upper-case letters (e.g.,
X , U , V , etc.)
to denote the variables.
For simplicity of exposition, we assume that all variables are binary taking values from the set { 0 , 1 } .
A lower case letter (e.g., x , y , etc.) denotes an assignment of a value to the corresponding variable denoted by the upper case letter.
For simplicity of exposition, we present our algorithm and experimental results on a specific type of tractable model called cutset networks [17] which combines Bayesian networks and AND/OR graphs [5], noting that the algorithm in this paper can be easily extended to other tractable models such as sum-product networks [14], thin junction trees [1], and arithmetic circuits [2].
3.1 Bayesian Networks
Bayesian networks (BNs) [3, 12] use directed acyclic graphs (DAGs) and conditional probability tables (CPTs) to compactly represent joint probability distributions over a set of random variables.
Each node in the DAG denotes a random variable and is associated with a conditional probability distribution of the corresponding variable given its parents in the DAG.
Formally, a Bayesian network is a triple ⟨ X , G, F ⟩ where X = { X 1 , . . .
, X n } is a set of random variables; F = { F 1 , . .
. , F n } is a set of conditional probability tables (CPTs); and G ( V, E ) is a directed acyclic graph
such that each vertex V i ∈ V is associated with the variable X i ∈
X and E is the set of directed edges.
Each node
V i is associated with the CPT F i =
P i ( X i |
U i ) that represents the conditional probability distribution of the variable X i given its parents
U i ⊆
X \ X
i .
Let x = ( x 1 , . . .
, x n ) be an assignment of values to all variables in the set X and let u i ∼ x denote the value assignment to all variables in the set U i that is compatible with x , namely u i is the projection of x on U i .
We will parameterize the Bayesian network using a set of parameters Θ where each θ x
i u
i ∈ Θ is equal to the conditional probability P
( X i =
| U i = u i ) .
Note that since
P ( X i = 1 |
U
i =
u i )
+ P
( X i =
| U i = u i )
= 1 , we do not need to have a separate parameter for
P ( X i =
| U i = u i ) .
Under this parameterization, the probability distribution represented by the
Bayesian network can be written as:
[Equation]
The two main reasoning tasks over Bayesian networks are most-probable explanation ( MPE ) and posterior marginal inference ( MAR ).
The former seeks to find an assignment of values to all variables given evidence such that the probability is maximized.
Formally, let E ⊂
X be a set of evidence variables, let Z = X \ E , and e be an assignment to all variables in E , then the MPE task is given by argmax z
P ( z , e ) .
The MAR task seeks to find the marginal probability distribution over each non-evidence variable given e , namely compute P ( z j | e ) for each variable Z j ∈ Z .
Both tasks are known to be NP-hard in general but can be solved efficiently in practice on Bayesian networks having small treewidth using bucket (or variable) elimination and propagation algorithms [4].
Specifically, given a Bayesian network having treewidth w , the time and space complexity of these exact inference algorithms is bounded by O ( n 2 w +1 ) and O ( n 2 w ) respectively.
3.2 Cutset Networks
Cutset networks (CNs) [15] are tractable probabilistic models that combine two well-known classes of tractable models: AND/OR graphs [5] and tree Bayesian networks.
Formally, a CN defined over a set of variables X ( X may include latent variables) is defined recursively using the following three conditions: (1) A tree Bayesian network over X is a CN; (2) An OR node labeled by a variable
X i ∈
X such that | X | > 1 with two child CNs, each defined over the set X \ { X i } is a CN.
We follow the convention that the left child of the OR node labeled by X i represents conditioning over
X i =
and the right child represents conditioning over X i = 1 .
The edges from the OR node to its child nodes are labeled with probability values in R + such that they sum to 1 ; and (3) Let ( X 1 , X 2 ) be a partition of X such that | X | > 1 .
Then, an AND node with two child CNs, one defined over Xand the second defined over X 2 is a CN.
X
X
X 3 X 4 X
T
X 7 X 6 X
T
X 5 X 7 X
T
X 4 X 6 X
T
X 4 X 5 X 7 T
X
X 2 X 3 X
T
X 8 X 2 X
T
P ( X 1 =1)=0 . 4 P ( X 1 =0)=0 .
P ( X 2 =0 | X 1 =0)=0 . P ( X 2 =1 | X 1 =0)=0 . P ( X 6 =0 | X 1 =1)=0 . P ( X 6 =1 | X 1 =1)=0 .
Fig.
1 shows a CN defined over eight variables.
In general, a full assignment x yields a rooted sub-graph in a CN.
Following our notation for Bayesian networks, let Θ denote the set of parameters of the cutset network, such that θ x
i
u i ∈ Θ is equal to the conditional probability P
( X i =
| U i = u i ) .
Given an assignment x , when X i is an OR node, u i denotes the assignment along the path from the root of the CN to X i .
Alternatively, when X i belongs to a tree Bayesian network in the CN, u i denotes the assignment formed by composing the assignment along the path from the root of the CN to the tree Bayesian network with the assignment to the parent of X i (if X i has a parent in the network).
Given this parameterization, the probability distribution associated with the cutset network is given by
[Equation]
A distinguishing feature of CNs is that when they have no latent variables, both MAR and MPE inference can be performed over them in time that scales linearly with the size of the network.
This can be accomplished by converting the CN in linear time to an AND/OR graph [5] or an arithmetic circuit [2] and performing two-passes over these circuits.
When latent variables are present, CNs admit linear time MAR inference only while MPE inference is intractable in general.
Our Approach
In this section, we define the optimization problem for learning the parameters of a cutset network from local estimates and present a gradient-based algorithm for solving it.
4.1 The Learning Problem
The high-level goal of this paper is to improve the prediction quality and model fit of an existing model that is induced using global information or full data alone by leveraging local information .
The latter is often available in practice from various sources such as prior knowledge, expert domain knowledge, external sensors or processes and (local) data over a small subset of features.
A distinguishing property of local information is that it is typically available in large quantities and thus more robust as compared to global information/full data.
However, because of various issues such as user bias (e.g., when information is obtained from different experts), locality and privacy constraints, local information is often noisy and inconsistent.
Therefore, our specific goal is to learn cutset networks such that they optimally combine both local and global information, regularizing the two appropriately based on background knowledge.
We begin by describing required notation and simplifying assumptions.
We assume that we have access to local information that can be summarized via pairwise distributions.
Note that our algorithm can be easily extended to arbitrary (non-pairwise) local marginal distributions and we make the assumption for clarity of presentation only.
Let D denote the KL divergence (distance) between two probability distributions defined over the same set of variables.
Namely, D ( A , B ) =
(cid:80) x
A ( x ) log A ( x ) − (cid:80) x
A ( x ) log B ( x ) where A and B are two distributions defined over X .
The first term equals the negative entropy of the distribution A and the second term equals the negative cross-entropy between
A and B .
Let E denote a subset of pairs of random variables, namely E ⊆ { ( X j , X k ) | X j , X k ∈ X and j < k } .
Definition 1.
We say that a set of local estimates {P jk ( X j , X k ) | ( X j , X k
) ∈ E} is consistent iff there exists a probability distribution P such that each estimate P jk
( X j , X k ) is a marginal probability distribution of P .
Otherwise, we say that the set is inconsistent .
Given a set of local estimates and data, the learning problem is given by:
Given: A cutset network structure with parameters Θ representing a probability distribution R , a set of pairwise local distributions P jk ( X j , X k ) where ( X j , X k ) ∈ E and a fully observed dataset X = { x (1) , . . .
, x ( m ) } .
To do: Find an assignment of values to all parameters in Θ such that the negative log-likelihood of X w.r.t.
R Θ is minimized and the KL distance between P jk ( X j , X k ) and R Θ ( X j , X k ) equals zero for all ( X j ,
X k ) ∈ E .
Mathematically, we can express it as:
[Equation]
Unfortunately, if the set {P jk ( X j , X k ) | ( X j , X k ) ∈ E} is inconsistent (see Def. 1), then the above constrained optimization problem is infeasible.
1 To address this issue, we propose to use a linear combination of the objective and constraints (which is akin to Lagrange relaxation).
Mathematically, argmin Θ
[Equation]
where λ 1 ≥ 0 and λ 2 ≥ 0 are hyperparameters (technically, we only need one hyperparameter; we use two for convenience) that model the relative importance of the local and global statistics respectively.
Here, D ( P jk ( X j , X k ) ,
[Equation]
4.2 Simplifying the Learning Problem
We simplify the optimization problem given in Eq.
(4) by making the following observation.
Since (cid:80) x j ,x k P jk ( x j , x k ) log P jk ( x j , x k ) is constant, we can remove it from the objective function and replace each term D ( P jk ( x j , x k ) , R Θ ( x j , x k )) in Eq.
(4) by − (cid:80) x j ,x k P jk ( x j , x k ) log R Θ ( x j , x k ) .
Negating the objective and making the substitutions described above yields the following maximization problem:
[Equation]
The optimization problem given in Eq.
(5) is not concave in the parameters Θ but it is smooth. Therefore, it can be solved using an iterative, gradient ascent algorithm.
However, a drawback of this algorithm is that the gradient of the second term in Eq.
(5) requires us to go over the whole data at each iteration yielding a slow algorithm.
Although, stochastic gradient descent or mini-batch approaches can be used to address this issue, their convergence is quite slow in practice.
Therefore, in order to reduce the computational complexity, we propose the following moment-matching approach.
Let Q denote the distribution associated with the cutset network having the same structure as R .
Thus, there is a one-to-one correspondence between the parameters of Q and R .
Let Π denote the set of parameters of Q .
Thus, given a parameter π x i , u i ∈ Π , there is a corresponding parameter θ x i , u i in Θ .
Let the set of parameters Π be learned from data X by maximizing the log-likelihood.
Since the parameters of cutset networks are conditional probability distributions, given Q learned from data, we can use negative cross-entropy between parameters of Q and R in lieu of the second term (log-likelihood) of Eq.
Mathematically,
[Equation]
4.3 Solving the Learning Problem via Gradient Ascent
We propose to solve the optimization task given in Eq.
(6) using gradient ascent methods.
To this end, we derive the gradients w.r.t.
each parameter θ x i , u i .
The partial derivative of the second term w.r.t.
θ x i , u i is straight-forward and given by
[Equation]
The partial derivative of the first term of Eq.
(6) is more involved and we summarize it in the following proposition.
(Proofs are given in the appendix.)
Proposition 2.
The partial derivative of
[Equation]
w.r.t.
θ x i , u i is given by
[Equation]
Formal Algorithm
Next, we formally present an algorithm that leverages the gradient equations given in Eqs.
(7)–(8) (see Algorithm 1) for solving the learning problem given in Eq.
The algorithm, which we call learning cutset networks with local inconsistent statistics (LCN-LIS), takes as input training dataset
X , local statistics P jk ( X j , X k ) , two hyperparameters λ 1 and λ 2 (real numbers) and an integer bound T on the number of iterations.
It begins by learning a cutset network Q from the dataset X (step 2) and initializes R to have the same structure as Q (step 3).
In steps 4–19, the algorithm runs the gradient ascent steps.
The gradient ascent begins by setting all parameters to a random number between 0 and 1 .
Then, at each iteration t , for each pair ( X j , X k ) ∈
E and its possible assignments ( x j , x k ) , it sets X j = x j and X k = x k as evidence and runs a two-pass inference algorithm over the cutset network [5] to compute the required conditional probabilities R Θ t ( u i , X i = 1 | x j , x k ) and R Θ t ( u i , X i =
| x j , x k ) (step 9).
The algorithm then updates the gradient for each parameter θ x i , u i (steps 10–13; also see Eqs.
(7)–(8)) given the assignment X j = x j and X k = x k .
In steps 15–17, the algorithm updates the parameters using the gradient estimates g x i , u i and learning rate α .
The algorithm terminates the gradient ascent on convergence or when the bound T on the number of iterations is reached.
At termination, the algorithm returns R with parameters Θ t .
The main virtue of our algorithm is that it has polynomial computational complexity.
The time (and space) complexity of step 9 is O ( | Θ | ) using a two-pass algorithm over the CN that calculates the conditional probabilities of the parameters given evidence (see [2, 5]).
The time complexity of updating the gradients (steps 10-13) is also O ( | Θ | ) . Thus, the time complexity of steps 9–13 is O ( | Θ | ) . Since these steps can be executed a maximum of O ( |E| × T ) times (step 8 and step 6 respectively), where T is the bound on the number of iterations, the overall complexity is O ( |E| × T × | Θ | ) . Remarks.
Note that a feasible solution to the optimization problem given in Eq. (6), and thus the one returned by Algorithm 1 filters inconsistency in the local estimates {P jk ( X j , X k ) | ( X j , X k ) ∈ E} Set and
R Θ t Let: δ + = R Θ t
u i , X i = 1 | x j , x k ) θ
tx i u i δ − =
R Θ t u i X i =
| j x k ) 1 − θ tx u i ) g x i
[Equation]
i = g x
[Equation] [Equation]
+
[Equation]
[Equation]
[Equation]
, x k )( δ + − δ − ) + λ 2 (cid:18) π x i , u i θ tx i , u i − 1 − π x i , u i 1 − θ tx i , u i (cid:19) end end for each parameter θ x i , u
i ∈
Θ do θ t +1 x i , u i =
θ
tx i , u i + α × g x i , u i // α : rateend t + until convergence or t ≥ T ; return R with parameters Θ t 21 end because it yields a globally consistent model R Θ .
Unlike previously proposed techniques for solving the optimization task in Eq.
(3) such as the iterative proportional fitting procedure (IPFP) [6, 25] which will not converge when the local estimates are inconsistent 2 , Algorithm 1 will converge to a local optimum (because the objective is smooth).
To summarize, we derived and presented a gradient-based algorithm for learning the parameters of cutset networks in presence of local estimates (see Algorithm 1) and showed that the algorithm requires time and space that scales linearly with the number of given local statistics.
Note that
Theorem 2 can be easily extended to Bayesian and Markov networks.
However, the problem is that computing the terms in the numerator in Eq.
8 will be NP-hard in general on Bayesian and
Markov networks.
This highlights another virtue of tractable models, local information, even if it is inconsistent can be efficiently integrated into a tractable model.
Experiments
We performed a detailed, controlled experimental study to evaluate the impact of using inconsistent local statistics on the quality of the learned model.
Specifically, we used the following controls: (1) the accuracy of the cutset network Q learned from data X in Algorithm 1; (2) the strength/level of inconsistency in the local statistics P jk ( X j , X k ) ; (3) the cutset network architecture; and (4) the number of evidence variables or observations available at test time (to test discriminative performance).
We used 20 benchmark datasets that have been widely used in previous studies [20, 21] to evaluate our new approach.
The number of variables in these datasets vary from 16 to 1556, and all variables are binary.
We ran Algorithm 1 for a maximum of 48 hours or 1000 iterations (namely T = 1000 ) or convergence,whichever was earlier.
We used 5 -fold cross-validation to select the values of the hyperparameters λ 1 and λ 2 .
[Table]
Local estimates: For each dataset, we learned a mixture of cutset networks [16, 17] and used it as the true model P .
We generated local statistics from P as follows.
Since
P is a tractable model [Table] we can efficiently (in linear time) compute P jk ( X j , X k ) for all X j , X k ∈ X .
To make them inconsistent [Table] we added a value ϵ that is randomly sampled from a normal distribution with 0 mean and standard deviation σ .
We experimented with five values of σ : { 0 . 001 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 } .
Note that after adding ϵ , we have to normalize the distributions to ensure that they are valid.
Data model: We used 10% of the randomly chosen examples in the training set to learn Q .
Thus, the dataset used by Algorithm 1 has 90% fewer examples than the one used for learning P .
We did this to ensure that Q , learned from a much smaller amount of data, differs significantly from the true model P , which in turn will help us evaluate how local information improves the quality from an inferior starting point.
We further controlled the quality of Q using a parameter h which we call the perturb rate .
h lies between 0 and 100 , and given a value for h , we replaced h % of the parameters in Q with a random number.
We normalized Q to ensure that it is a valid probability distribution.
We used three types of cutset network architectures: (1) cutset networks with depth 0 which are equivalent to Chow-Liu trees (CLTs); (2) cutset networks with no latent variables (CNs); and (3) mixtures of cutset networks (MCNs).
The latter is a state-of-the-art model [19].
We learned both discriminative and generative cutset networks.
In discriminative networks, we set L % of random variables as evidence E and learn a probability distribution over the variables X \ E given an assignment e to the evidence variables.
We used 4 values for L : { 0 , 20 , 50 , 80 } .
When L = 0 , we get a generative model while the remaining models are discriminative.
As our evaluation criteria, we used negative cross-entropy between the true P and the learned model.
The higher the negative cross-entropy, the better the model.
6.1 Results
Improved Model Quality.
For lack of space, we present plots showing the impact of the level of noise in Q , controlled by h ; and the level of inconsistency in the pairwise estimates P jk
( X j , X k ) , controlled by σ on one randomly chosen dataset.
We present the plots for the remaining datasets in [Figure] [Figure] the supplementary material.
Figure 2 presents the negative cross-entropy scores achieved by CLTs,
CNs [Figure] and MCNs as a function of the perturb rate ( h ) for a randomly chosen dataset ‘movie’
[Figure]
and for different values of L (the percentage of evidence variables).
These plots help us evaluate the impact that the quality of Q has (namely the impact of the starting point) on the model learned by
Algorithm 1.
We observe that as the perturb rate increases, there is a substantial drop in the negative cross-entropy between P and Q .
However, the negative cross entropy between P and R remains relatively flat.
This shows that the use of local statistics significantly improves the model quality, especially when the model based on global data alone is inaccurate.
Figure 3 shows the negative cross-entropy scores achieved by CLTs, CNs and MCNs as a function of the standard deviation σ ∈ { 0 . 001 , 0 . 01 , 0 .
05 , 0 . 1 , 0 . 2 } of the Gaussian noise that is applied to the local statistics P jk
( X j , X k ) for the dataset ’pumsb*’, and for different values of L (the percentage of evidence variables).
The figure clearly shows that when the local statistics contain more noise, the performance degrades as expected.
When the σ is large ( σ > 0 . 2 ), there is a non-zero chance that the final model R will be worse than Q .
Thus, based on our experimental results, our proposed algorithm is more likely to yield a better model than the one based on global information alone when the Gaussian noise has standard deviation ≤ 0 . 1 .
Improved Generative and Discriminative Performances.
Tables 1 and 2 show the generative and discriminative ( L = 50 and L = 80 ) performances of Q and R respectively.
We present the results for L = 20 are shown in the supplement (see Table 3).
We used σ = 0 .
1 and h = 0 to generate the experimental data given in the tables.
Each value in the tables is an average over 5 runs.
To avoid clutter, we do not report the standard deviation because it was fairly small over all the runs.
These results help us analyze the impact of the cutset network architecture and the number of evidence variables on our evaluation criteria.
We observe that, on average, using local inconsistent statistics improves the negative cross entropy of each architecture by 17-23%.
MCNs are the best performing [Table] model overall and Chow-Liu trees (CLTs) are significantly worse.
There was no significant difference in the amount of improvement as we increased the number of evidence nodes.
This suggests that our approach is equally useful for both discriminative and generative models.
Conclusion
In this paper, we presented a new method for learning the parameters of cutset networks in presence of inconsistent local estimates.
Unlike conventional algorithms which use full i.i.d.
data during the learning process, we proposed a novel approach that uses noisy local information to learn a more accurate and robust model.
The key advantage of using local estimates is that they are often readily available as compared to full i.i.d.
data.
We also showed via experiments on benchmark datasets that our new algorithm greatly improves the quality of the initial model learned from i.i.d.
data, even when the local estimates are inconsistent and noisy.
Future work includes: (1) developing new structure learning algorithms that use local inconsistent statistics; (2) applying our approach to other tractable models such as sum-product networks and probabilistic sentential decision diagrams; (3) improving human feedback to improve explanation quality; (4) developing lazy learning algorithms; etc.
Acknowledgments and Disclosure of Funding
This work was supported in part by the DARPA Explainable Artificial Intelligence (XAI) Program under contract number N66001-17-2-4032, the DARPA Perceptually-Enabled Task Guidance (PTG)
Program under contract number HR00112220005 and by the National Science Foundation grants
IIS-1652835 and IIS-1528037.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
In the supplement.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
In the supplementary material 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] In the Supplement (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] In the section on Experiments and the Supplement (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
Standard deviation was very small and so error bars were not provided.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
In the supplement 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Proof of Proposition
The partial derivative of
[Equation]
w.r.t.
θ x i , u i is given by
[Equation]
Proof.
[Equation]
Using the sum rule of probability theory, R θ ( x j , x k ) equals = (cid:88)
[Equation]
x
[Equation]
(11) =
[Equation]
u [Equation] ( u i ) R θ ( x j , x k | u i , x i ) (12) Since R θ ( x i | u i ) =
θ x i , u i if X i = 1 and R θ ( x i | u i ) =
1 − θ x i , u i if X i = 0 , we have: R θ ( x j , x k ) =
(cid:88) u i θ x i , u i R θ ( u i ) R θ ( x j , x k | u i , X i = 1) + (cid:88) u i (1 − θ x i , u i ) R θ ( u i ) R
θ ( x j , x k | u i , X i = 0) (13)
Differentiating the Equation above (Eq.
w.r.t.
θ x i , u i , we get: ∂
R θ ( x j , x k ) ∂θ x i , u i = R θ ( u i ) R θ ( x j , x k | u i , X i = 1) −R θ ( u i )
R θ ( x j , x k | u i , X i = 0) (14)
Since
R θ ( u i ) R θ ( x j , x k | u i , X i = 1) = R θ ( x j , x k , u i , X i = 1)
θ x i , u i (15) and
R θ ( u i ) R θ ( x j , x k | u i , X i = 0) = R θ ( x j , x k , u i , X i = 0) 1 − θ x i , u i (16) we can rewrite Eq.
(14) as:
[Equation]
[Equation]
Substituting Eq.
(17) in Eq.
(10) and using the definition of conditional probability
[Table]
we get:
[Equation]
B Additional Experimental Results
[Table]
Table 3 has the 20% evidence results to supplement the discriminative performance results presented in the main paper (with 50% and 80% evidences)
[Table]
Next, we present the plots for all the 20 datasets.
We varied the perturb rate h and standard deviation σ as described in the main paper.
Figures 4 – 8 and Figures 9 – 13 show the respective results.
[Figure]
a t i v e C r
[Figure]
n t r o
[Figure]
y
[Figure]
t i
[Figure]
g a t i v e C r o ss [Figure] n t r o p y f
r o m P
[Figure] [Figure]
N e g a t
i v e C r
o
ss E n t r o p
y f
r o m P
[Figure]
a t i v
[Figure]
C r
[Figure]
n t r
[Figure]
y f
[Figure] [Figure]
e g a t
i
v e C r o
[Figure]
E n t r o p y f r o
[Figure]
g a t i v e C r
[Figure]
ss
[Figure]
n t r o p
y f r
[Figure] [Figure]
a t i v e C r
[Figure]
f
[Figure]
o
[Figure]
e g a t
i
v e C r o ss E n t r o p
y f r
[Figure]
m
[Figure] [Figure]
N e g a t
i v
[Figure]
C r o ss
[Figure]
n t r o p
y f
r o m
[Figure]
t i v
[Figure]
C r
[Figure]
n t r o p
y f
[Figure] [Figure]
e g a t
i
v e C r o ss E n t r o p
y f
r o m P
[Figure]
N
[Figure]
t
i
[Figure]
C
[Figure]
t
[Figure]
y
[Figure]
[Figure]
a t i v e C r o ss E n t r o p
y f
r o
[Figure]
P
[Figure]
a t i v e C
[Figure]
t
[Figure] [Figure]
N e g a t
i v e C r
o
ss E n t r o p
y f
r o m P
[Figure] [Figure]
N
[Figure]
g a t i v e C r o ss E n t r o p y f r o
[Figure]
a t i v e C r
[Figure]
n t r o p
y f r
[Figure] [Figure]
N e g a t
i v e C r
o
ss E n t r o p
y f
r o m P
[Figure]
g a t i v e C r
[Figure] [Figure]