L EARNING K OLMOGOROV M ODELS FOR B INARY R ANDOM V ARIABLES
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a framework for learning a Kolmogorov model , for a collection of binary random variables.
More speciﬁcally, we derive conditions that link (in the sense of implications in mathematical logic) outcomes of speciﬁc random variables and extract valuable relations from the data.
We also propose an efﬁcient algorithm for computing the model and show its ﬁrst-order optimality , despite the combinatorial nature of the learning problem.
We exemplify our general framework to recommendation systems and gene expression data.
We believe that the work is a signiﬁcant step toward interpretable machine learning
I NTRODUCTION
Machine learning and artiﬁcial intelligence method have permeated a large number of areas (Marr,
Sept 2016).
These methods are based on machine learning models , which consist of learning an input-output mapping for a given dataset.
Despite the plethora of such models (e.g., matrix factorization (Koren et al., 2009), SVD-based models (Koren, 2008), deep neural networks (LeCun et al., 2015), and models inspired from physics (Stark, 2016b))
, they lack interpretability : they are not capable of offering insight about the data, nor the underlying process.
The lack of interpretablity may have serious consequences in mission-critical systems, ethics, and validation of computeraided diagnosis (Doshi-Velez & Kim, 2017).
While there is no consensus around the deﬁnition of interpretability, causality (Lipton, 2016) is a vital component: it refers to causal relations within the data and insight about the underlying data-generating process.
We thus adopt a generalized version of causality using implication in mathematical logic as our ‘deﬁnition’ of interpretable models .
To this end, we propose learning a so-called Kolmogorov Model (KM) associated with a set of binary
Random Variables (RVs).
In addition to prediction, the interpretability of the model (as deﬁned above) enables learning causal relations .
1 We derive a sufﬁcient condition under which the realization of one RV’s outcome (causally) implies the outcome of the other.
In the context of recommendation systems, causal relations identify groups of items, for which a user liking one item implies that he/she likes all other items in the group.
In gene expression analysis, the same relations identify groups of DNA locations for which the expression of a gene in one of them implies its expression in all other locations.
The foundation of our approach is to model binary RVs as elementary events on a Kolmogorov space, by an inner product of two vectors (formally stated in Section 2.1), which is based on established results from classical probability theory.
To our best knowledge, this speciﬁc formulation is novel in the context of learning representation.
The inner product of our formulation is reminiscent of factorization methods such as, matrix factorization (Koren, 2008), non-negative matrix factorization (Lee & Seung, 2001), SVD (Cai et al., 2010), and physics-inspired techniques, non-negative models (Stark, 2016a).
It should be noted that the inner product components of these methods is usually based on strong intuition about the data, provided and validated by a human expert.
In contrast, the proposed KM is a fully automated approach, deeply rooted in probability theory, which ﬁnds an interpretable model (as deﬁned above) of the data without any human intervention.
Thus, both the model and the causal relations within the data have a strong mathematical basis.
Moreover, in most of the existing approaches, we may need to have one pipeline for learning the representation and another one for mining these relations.
However, our approach requires only a single pipeline for both tasks.
This is very appealing on a conceptual level, and it can drastically simplify coding and validation.
Our method also generalizes
K-means and some of its variants.
Detailed discussions of the relation between our proposed method and these prior works is in Appendix A.3.
In an abstract sense, we formulate a KM learning problem as a coupled combinatorial program , decompose it into two subproblems using the Block-Coordinate Descent (BCD) method, and obtain provably optimal solutions for both.
For the ﬁrst one, we exploit the structure of linear programs on the unit simplex and use low-complexity (yet optimal) Frank-Wolfe algorithm (Frank & Wolfe, 1956).
To bypass the inherent complexity of the second subproblem (combinatorial and NP-hard), we propose a semideﬁnite relaxation, and show its quasi-optimality in recovering the optimal solution of the combinatorial subproblem.
Finally, we show the convergence of our algorithm to a stationary point of the original problem.
We propose a simple algorithm for mining the causal relations.
All the proofs and additional discussions are available in Authors (Oct 2017).
S YSTEM M ODEL
Notation: Lowercase letter a , uppercase bold letter A , and calligraphic letter A denote vectors, matrices, and sets, respectively.
[ A ] i,j and A T denote element ( i, j ) and the transpose of A .
supp ( a ) denotes the support set of a .
The inequality x ≤ y holds element-wise.
I denotes the identity matrix, 1 and 0 the all-one and all-zero vectors (of appropriate dimension).
e n is the n th elementary basis,
P = { p ∈ R D + | 1 T p = 1 } the unit simplex, and { n } := { 1 , · · · , n } .
2. P ROBLEM F ORMULATION
Consider a double-indexed set of binary Random Variables (RVs) , X u,
i ∈
A = { 1 , 2 } , taken from a dataset D = { ( u, i ) | ( u, i ) ∈
U × I} .
Each RV is deﬁned on a sample space Ω , consisting of elementary events Ω = { ω d | 1 ≤ d ≤ D } .
We denote by P [ X u,i =
z ] , z ∈ A , the probability that RV X u,i takes the value z ∈ A ; see example in Section 2.2.
Since X u,i is binary, it is fully characterized by considering one outcome.
Thus, we write the Kolmogorov Model (KM) for X u,i as
[Equation]
Thus, each RV X u,i is associated with (characterized by) a Probability Mass Function (PMF) θ u , u ∈ U , and an indicator vector ψ i , i ∈ I .
The model follows from established results in classical probability theory (Kolmogorov, 1957),(Gray, 2009) (formalized in Appendix A.2).
Notice that the model in (1) can approximate with arbitrarily small accuracy the measure corresponding to P [ · ] for a large enough D .
Problem 1 (Problem Statement) Let p u,i denote the empirical value of P [ X u,i = 1] .
We assume that { p u,i } are known for elements of a training set K ⊆ D , where K = { ( u, i ) | ( u, i ) ∈
U × I} .
Given samples coming from the model in (1), we wish to infer the parameters of underlying probability distribution: ﬁnd parameters of the KM, i.e., { ψ i , θ u } that best describe { p u,i | ( u, i ) ∈ K} .
The resulting problem is a fully parametric statistical inference task.
For tractability, we address it using the minimum mean-squared error as a point estimator.
The corresponding optimization problem is
[Equation]
Q ) :
[Equation]
We present our solution to this coupled non-convex conspiratorial optimization in the next section.
The obtained solution to ( Q ) can be used for prediction on a test set, as well as extracting causality structures within training set (see Section 4).
Proposed Approach: While the proposal to model binary RVs as elementary events on a Kolmogorov space is based on established results, the speciﬁc learning formulation (Problem 1) is novel.
Because this model is rooted in probability theory, (1) deﬁnes the outcome of a RV in the strict Kolmogorov sense, and the resulting causal relations (Section 4) also hold from a strict analytical perspective.
Note that causal relations (a.k.a. association rules ) may still be extracted using existing methods,
However, these relations are not based on causality and the formal relations that (mathematically) follow from the KM in (1), but rather on intuitions/heuristics, which may yield different relations.
Naturally, we wish the explore further in this work the causal relations that arise from the proposed model.
Additionally, unlike existing methods, the prediction and causal relations mining are done in ’one-shot’, thereby simplifying the implementation/validation; see Appendix A.3.
2. I LLUSTRATIVE E XAMPLE : R ECOMMENDATION S YSTEMS
In this context, U and I denote the set of users and items respectively, and X u,i models the preference of user u for item i , ( u, i ) ∈ K .
Thus, P [ X u,i = 1] (or P [ X u,i = 2] ) is the probability that user u likes (or dislikes) item i .
Moreover, θ u determines the proﬁle/taste of user u , ψ i is related to item i (depending on genre, price, etc.), and the elementary events denote movie genres (e.g., ω 1 = “Action”, ω
2 = “SciFi”, etc.).
The training set , consisting of an empirical probability that user u likes item i , p u,i . We obtain this probability using p u,i (cid:44)
[ R ] u,i /R max , where [ R ] u,i ∈ N denotes the rating that user u has provided for item i , and R max the maximum rating (Stark, 2015).
For instance, if user u rates item i with a score of [ R ] u,i = 7 (where the maximum rating is 10 ), then p u,i = 0 .
Other approaches may be used to obtain p u,i depending on the speciﬁc application.
As a concrete illustrative example, consider a 10 -star “recommendation system”, having 2 users anditems.
We then ﬁnd the D -dimensional
( D = 3 ) KM factorization to obtain , { ψ (cid:63)i } 2 i =1 and { θ (cid:63)u } 2 u =1 .
D is the size of the Kolmogorov space Ω , the number of elementary events, and the dimension of the factorization (selected via cross-validation to minimize the test error).
Solving ( Q ) results in ﬁnding { ψ (cid:63)i } 2 i =1 and { θ (cid:63)u } 2 u =1 .
An example result is given below:
[Equation]
[Equation] 
[Equation]
[Table]
[Equation]
 
To showcase the model’s intuition, note that p 1 , 1 , the probability that user 1 likes movie 1 , is represented as ψ T 1 θ 1 .
It is thus expressed as convex/stochastic mixture of movie genres , since elementary events are movie genres in this scenario.
We underline that this high degree of interpretability is not artiﬁcially enforced but rather context-dependent.
More generally, a KM represents a set of observed outcomes for RVs as (context-dependent) mixtures of elementary events.
The approach consists of learning a (hidden) latent model by jointly optimizing the PMF and binary indicator vectors.
Thus, interpreting elements of the indicator vectors as movie genres requires a context-dependent map , from the elements of ψ (cid:63)i to movie genres.
Another way to ﬁnd such an interpretation is when this context-dependent map is known apriori. In this setting, each item is already tagged with its movie genres, and ψ i need not be optimized;
Alas, having this context-dependent map comes at the expense of a loss in training/test (see Appendix A.4).
However, recall that the primary interest of this work is not this facet of interpretability, but rather by that of the causal relations.
P ROPOSED A LGORITHM
To approach a solution for problem (2), we use the block-coordinate descent (BCD) method to split ( Q ) into two sub-problems.
Here, we derive our solution approach to each.
We ﬁrst reﬁne the current
PMF estimation θ u , and then that of the indicator ψ i .
Given { ψ ( n ) i } at iteration n , we pose the PMF reﬁnement, θ -step, as
[Equation]
We then pose the indicator vector reﬁnement, ψ -step, as ( Q 2 ) : ψ ( n +1)
i
[Equation]
[Table]
u
[Table]
u
[Table]
( k ) u = e j (cid:63)
[Table]
j (cid:63)
[Table] 1 ≤ j ≤ D [ ∇ f ( θ ( k ) u )]
j
[Table]
( k +1) u = (1 − α ( k ) u ) θ ( k ) u + α ( k ) u
[Table]
( k )
[Table] [Table] Moreover, U K and I K are deﬁned as K = { ( u, i ) |
u ∈ U K ⊆ U , i ∈
I K ⊆ I} . Recall that having globally optimal solutions for both ( Q 1 ) and ( Q 2 ) is necessary to show the convergence of BCD (Tseng, 2001) - a challenging task due to the NP-hardness of ( Q 2 ) . 3. θ -S TEP : R EFINE PMF E STIMATE
We use the Frank-Wolfe (FW) algorithm (Frank & Wolfe, 1956) to solve ( Q 1 ) as a succession of
Linear Programs (LPs) over the unit simplex.
While LP solvers generally have similar complexity as quadratic program solvers, solving an LP reduces to searching for the minimum index - which is computationally efﬁcient, when the LP is over the unit simplex.
We summarized this FW variant for solving ( Q 1 ) ; see also Jaggi (2013)[Algorithm 1].
Here, we drop the BCD iteration index, n , and just keep the FW iteration number, k , for notation simplicity.
We ﬁrst determine the descent direction : d ( k ) u ∈ argmin s ∈P (cid:16) ∇ f ( θ ( k ) u ) (cid:17) T s .
The constraint s ∈ P greatly simpliﬁes the above
LP and yields: d ( k ) u
= e j (cid:63) , j (cid:63) ∈ argmin 1 ≤ j ≤ D [ ∇ f ( θ ( k ) u )]
j ; see Proposition 2 (Appendix A.6).
The solution follows from LPs over the unit probability simplex.
Thus, ﬁnding the descent direction reduces to searching over the D -dimensional gradient vector (done in O ( D ) ).
Then, the current value is updated using a simple step size rule, α ( k ) u = k/ ( k + 1) .
Table 1 shows the θ -step solution, and
Proposition 3 in Appendix A.6 characterizes its convergence.
3. ψ - STEP : R EFINE I NDICATOR E STIMATE
To address the NP-hard nature of ( Q 2 ) , we propose a solution based on Semi-Deﬁnite Relaxation and
Randomization (SDR) , and establish its quasi-optimality .
We use the results of Ma et al.
(2002)[Sec
IV-C]) and a series of reformulations to rewrite ( Q 2 ) in the following equivalent form (see Authors (Oct 2017) for all the derivations): [Equation] i
[Equation]
[ X i ] k,k = 1 , ∀ k , rank ( X i )
=where X i = x i x Ti , ˜ S
i = (cid:34) (1 /
[Equation]
S i
[Equation]
, x i =
[Equation]
z
[Equation]
w
[Equation]
(cid:21) , z i =
ψ
i − 1 , w i ∈ {− 1 , +1 } is an auxiliary variable, and
˜ t i (cid:44) ( v i − (1 / 2) S i 1 ) .
The problem is then relaxed into a convex program,
X
[Equation]
X (SDR) i may be solved using generic solvers.
Then, a randomization procedure (Ma et al., 2002) extracts an approximate (binary) solution for ( Q 2 ) ; see Table 2.
This evidently raises the issue of the sub-optimality gap for SDR.
Based on the results of Tan & Rasmussen (2001) and Luo et al.
(2010), we show in Proposition 4 (see Appendix A.6) that SDR (Table 2) is optimal (asymptotically in D ) in recovering the binary solution of ( Q 2 ) .
We highlight that the performance bound in Proposition 4 compares the quality of both the approximate binary solution offered by SDR (with respect to the optimal binary solution of ( Q 2 ) ), as well as their respective cost functions.
The asymptotic optimality is empirically validated in Appendix A.7.
3. A LGORITHM D ESCRIPTION
The BCD-based algorithm alternates between reﬁning the indicator and PMF vectors; see Algorithm 1.
Lemma 2 (Appendix A.6) shows its convergence to a stationary point of ( Q ) .
Algorithm 1 Iterative computation of KMs // Randomly Initialize { θ (1) u ∈ P} for n = 1 , 2 , ... do
Compute S ( n ) i and t ( n ) i using (4)
Call ˆ ψ ( n ) i =
SDR( S ( n ) i , t ( n ) i , M rnd ), ∀
i ∈
I K
Compute Q ( n ) u and r ( n ) u using (3) //
Initialize FW with { θ ( n − 1) u } , from previous iteration
Call θ ( n ) (cid:63) u
= FW( Q ( n ) u , r ( n ) u , (cid:15) ), for all u ∈ U K end for
I NTERPRETABILITY VIA C AUSAL R ELATION 4. C AUSAL R ELATIONS
Once a solution is found using Algorithm 1, here, we propose a method to ﬁnd causal relations among the RVs.
More speciﬁcally, we compare the support set of each pair of RVs from the training set, X u,i and X u,j , and check if there is any ‘overlap’ between their support set.
Intuitively, this condition means that some of the elementary events (see Section 2.1) of one RV might be ‘contained’ in the elementary events of another.
Consequently, the RVs are mutually related by causality, and the outcome of one determines that of the other.
This insight is formalized here.
Proposition 1 (Inclusion of Support Set) Consider two random variables X u,i and X u,j (belonging to the training set) whose KM are given by the model in (1) .
If supp ( ψ j ) ⊆ supp ( ψ i ) , then the following two causal relations hold:
[Equation]
Proof: See Authors (Oct 2017) for the proof.
Stated plainly, when the support set condition holds, the ﬁrst outcome of X u,i implies the same outcome for X u,j , and the second outcome for X u,j implies the second one for X u,i , thereby implying a mutual causal relation among them (since X u,i inﬂuences X u,j and vice-versa).
Note that our above deﬁnition of causality and causal relations is different than conventional ones in Pearl (2009)[Chap 2.8].
Moreover, our deﬁnition is distinct from Granger causality, due to the mutual coupling among the RVs in question.
We present next a special case of Proposition 1.
When ψ i = 1 , then supp ( ψ i ) =
{ D } , and supp ( ψ j ) ⊆ supp ( ψ i ) holds, for any choice of ψ j , ∀ j ∈ I K , where I K deﬁned in Section 3.
Corollary 1 (Maximally Supported RVs) Let { ψ i , θ u } ( u,i ) ∈K denote the KM associated with { P [ X u,i = 1] } ( u,i ) ∈K .
We deﬁne M as set of RVs for which the support set of the indicator vector is one, i.e., M =
{ i | ψ i = 1 } .
Then, the condition supp ( ψ j ) ⊆ supp ( ψ i ) (Proposition 1) holds trivially ∀ j ∈
I K .
Thus, the causal relations in (6) hold, for every i ∈ M .
For maximally supported RVs, the realization of one outcome, X u,i = 1 , determines that of all RVs of the set { X u,j = 1 | ∀
j ∈
I K } .
Example 1 (Causal Relations in Recommendation Systems)
In addition to prediction, recommendation systems are designed to accurately mine for association rules : if a user likes item i will he/she like another item j ?
Thus, the causal relations of Proposition 1 and Corollary 1 will be quite powerful, as we shall see.
In the example of Section 2.2, note that supp ( ψ 1 ) ⊆ supp ( ψ 2 ) .
Then, Propositionyields: if user 1 (or user 2 ) likes movie 2 implies he/she also likes movie 1 .
Moreover, X 1 , 2 and X 2 ,
X u,
X u, X u,
X u, X u,
X u, X u,
X u,
X u, a) b) c) a) Check support set condition between X u, 1 , and X u, 2 , X u, 3 ( X u, 1 is a maximally supported RV here)
b)Check support set condition between X u, 2 , and X u, 1 , X u, 3 c) Check support set condition between X u, 3 , and
X u, 1 , X u, are maximally supported RVs, since ψ 2 = 1 . Thus, Corollary 1 reads: If any user likes item 2 , then this causally implies that he/she likes all other items in the training set. Thus, our approach provides association rules that follow from causal relations between different RVs.
Consequently, these relations are stricter (as they are rooted in Kolmogorov probability) than other methods for mining association rules, which are not based on causality.
4. C AUSAL R ELATIONS M INING (CRM)
We provide an efﬁcient algorithmic approach to automatically mine the above relations.
In a nutshell, the algorithm does a pairwise check of the support set condition (Proposition 1), for pairs of RVs
X u,i and X u,j , ∀
( i, j ) ∈
I
K × I K .
The method is illustrated in Figure 1, for the illustrative example of Section 2.2.
The above causal relations can be modeled conveniently using the adjacency matrix A ∈ B |I
K |×|I K | , deﬁned as
[Equation]
Stated differently, [ A ] i,j = 1 if the support of X u,j is contained in that of X u,i .
Note that when A is sparse (resp. dense) is an indication for a dataset in which few (resp.
many) casual relations exist.
To quantify the sparsity level, we deﬁne an inﬂuence score, β i = |I
K | − 1 (cid:80) j ∈I K j (cid:54) =
i
[ A ] i,j , which measures the normalized number of pairs X u,i and X u,j , satisfying the support set condition.
These steps are summarized in the Algorithm 2 (illustrated in Figure 1).
Moreover, its complexity is dominated by the pairwise search over I K (see Step 1), which comes at O ( |I K | 2 −
|I K | )
≈ O ( |I K | 2 ) operations.
Check the support set condition, via a pairwise search to check for pairs ψ i and ψ j satisfying supp ( ψ j ) ⊆ supp ( ψ i ) , ∀ ( i, j ) ∈ I K × I K ,
i
(cid:54) =
j (done over the training set K ) 2.
Build the adjacency matrix A , in (7) , and compute the inﬂuence score
β i , ∀ i ∈
I K 3.
Find all pairs ( i, j ) such that a i,j = 1 .
For each of these pairs it holds that (Proposition 1), X u,i and X u,j are causally related, i.e., X u,
i
= 1 implies
X u,j = X u,j implies
X u,i =(8)
Identify (if possible) maximally supported RVs (Corollary 1), M =
{ i | ψ i = 1 } .
For each of them, the relations in (8) hold for all i ∈
I K
S PECIAL C ASES AND A PPLICATIONS
We highlight relevant special cases and applications of our approach.
Special Case: Unsupervised Learning Setting.
Note that Algorithm 1 is equally applicable to an unsupervised learning task (no prediction needed).
Moreover, if the training set has no missing data, i.e., K = D , then an alternate solution to ( Q ) may be obtained using the binary matrix factorization (MF) (Slawski et al., 2013) method.
However, it is not applicable when factorizing a training set K (where K ⊂ D ).
Unlike Algorithm 1, binary MF does not offer prediction.
Application: Analysis of Gene Expression.
The ability to analyze gene expression data is critical to DNA research (Zhang et al., 2009).
This task can be formulated in our proposed framework as follows: P [ X u,i = 1] denotes the probability that the genes being studied are expressed in
[Figure] sample u ∈ U at location i ∈ I of the DNA.
In this setting, the set Kolmogorov elementary events, { ω 1 , · · · , ω D } , represents the various genes that might be involved.
Consequently, our approach models the probability that a set of genes is expressed as a convex mixture of D various genes .
We recall that this intuition follows from our model and is not enforced explicitly.
More importantly, the causal relations can identify groups of DNA locations for which the expression of a gene (or its absence) in a given location implies its presence (or its absence) for all other locations in the group.
We will numerically show the usefulness of these relations to DNA analysis, in Section 7.2.
P RACTICAL C ONSIDERATIONS
Overﬁtting: Regularization parameters (to mitigate overﬁtting) can be included without any changes to the solution method.
An (cid:96) 2 − regularization can be included in ( Q 1 ) :
f ( θ u )
= θ
Tu ( Q u + λ u I D ) θ u − 2 θ Tu r u + γ u , where the regularizer λ u ≥ 0 is absorbed into a “new” matrix ( Q u + λ u I D ) .
Note that an (cid:96) 1 − regularization for θ u would not work, since θ u ∈ P .
Similarly, an (cid:96) 1 -regularization for ( Q 2 ) is: g ( ψ i ) =
ψ Ti S
i ψ i − 2( v i − ( µ i / 2) 1 ) T ψ i + γ i , where the regularizer µ i ≥ 0 is absorbed into the linear term, since µ i (cid:107) ψ i (cid:107) 1 = µ i 1 T ψ i , for ψ
i binary.
Optimality Gap: We recall that the proposed SDR method was shown to be quasi-optimal in providing approximate binary solutions to ( Q 2 ) .
Thus, the relaxation does not affect the interpretability, in the sense that Proposition 1 and Corollary 1 still hold.
While the derivations pertaining to causal relations (Section 4) assume globally optimal solutions to ( Q ) - an NP-hard problem, Algorithmguarantees locally optimal ones.
Thus, a bound on the gap between these solutions is needed.
We highlight this issue as an interesting topic for further investigation.
6. L IMITATIONS
Computational Complexity: The computational complexity of Algorithm 1 is dominated by the solution in (5), which is ≈ O ( D 4 . 5 ) operations per iteration of Algorithm 1 (recalling the negligible cost of the FW method).
Notice that the additional complexity compared to matrix factorization (its extensions), for which the complexity is O
( D 3 ) (e.g., D ≤ 16 in all numerical results).
Moreover, we are already investigating complexity reduction techniques leveraging the structure of the SDP, and distributed solutions to ( P ) to enable parallelization .
Finally, the computational complexity of Algorithm 2 consists mainly of step 1, which has ≈ O ( |I K | 2 ) operations.
Learning a non-stationary distribution: The proposed method assumes that distributions of the
RVs in the training set are stationary: Indeed, scenarios with time-varying distributions are a limitation (and interesting future directions).
However, in learning it is quite common to assume that the datagenerating distribution is stationary.
N UMERICAL R ESULTS 7. A PPLICATION TO R ECOMMENDATION S YSTEMS
Experimental Setup: We evaluate the performance of Algorithm 1 under the following.
We opted to not have a stopping criterion based on the prediction error of the algorithm (on a validation set), since we are primarily interested in the model that we learn in the training phase.
Nonetheless, as we have reported in Appendix A.7, there is not loss in the predictive performance of the model.
Hereafter, “Prop.” refers to the proposed method.
Setting 1: An artiﬁcial training dataset, where p u,i ∈ K
= { U = 20 } ×
{ I = 40 } , where { p u,i } are i.i.d.
and uniformly chosen on the unit interval.
We benchmark against a variant on Algorithm 1, where the ψ -step for Q 2 is replaced by an exhaustive search .
As the data is artiﬁcial (unsupervised learning setting), we include the Binary MF in Slawski et al.
(2013)[Algorithm 2].
Setting 2: The training set K , is chosen as the MovieLens 100K, with U = 943 users and I =items, split into 80% for training and 20% for testing.
Let { ˆ ψ i } , { ˆ θ u } the output of Algorithm 1, after 5 iterations.
We benchmark against matrix factorization (MF) (Koren et al., 2009), non-negative
MF (NMF) (Lee & Seung, 2001), SVD++ (Koren, 2008) (ensuring the dimension of the factorization, k , is close to D ), non-negative models (NNM) Stark (2015), and the K -means (K-M) algorithm.
The implementation and results use the MyMediaLite package (Gantner et al., 2011).
Training Performance for Unsupervised Learning (Setting 1): Fig 2a shows the resulting normalized training RMSE =
( (cid:80) ( u,i ) ∈K | p u,i − ˆ θ Tu ˆ ψ i | 2 / |K| ) 1 / 2 .
We observe that the monotone convergence in Lemma 2 is validated numerically, and that the training error decays with increasing model size, D .
Note that the training performance of Algorithm 1 is indistinguishable from its exhaustive search variant.
Moreover, Algorithm 1 converges to solution whose performance is similar to Binary MF, with a few iterations (except for D = 8 where Algorithm 1 outperforms Binary MF).
Training Performance for Supervised Learning (Setting 2): Recall that Binary MF is not applicable here, due to the supervised learning setting.
The same conclusions hold when testing Algorithmon the ML100K (Figure 2b).
We underline that while NNMs yield better training performance over
Algorithm 1, the latter will have better test performance (since NNMs are indeed deﬁned by relaxing
KMs).
In Table 4 (Appendix A.7), we empirically verify that the test performance of the proposed method outperforms the benchmarks in Setting 2.
Interpretability via Causal Relations (Setting 2): We numerically evaluate the relations of Algorithm 2.
The inﬂuence score for each item in the training set, β i , is shown in Figure 2c where we displayed items with ‘high’ inﬂuence score , β i ≥ 0 .
Note that each of these high inﬂuence items is causally related to at least half of the items in the training set.
This conﬁrms the effectiveness of Algorithm 2 for ﬁnding causal relations, and that a sparse adjacency matrix is uncommon.
We next identify the set of items corresponding to maximally supported RVs,
M = { 119 , 814 , 1188 , 1190 , 1290 , 1393 , 1462 , 1486 , 1494 , 1530 , 1590 , 1638 } .
For each of these items, a user liking one given item, implies he/she likes all other items in the training set.
Interestingly, these results remain the same when D = 24 , thereby suggesting that procedure for mining causal relations is quite stable.
7. A PPLICATION TO G ENE E XPRESSION
Experimental Setup: Following the problem statement in Section 5, we show the usefulness
Algorithm 2 for gene expression.
We ﬁrst deﬁne another setting.
Setting 3: We use the REGED0 dataset 3 .
Element ( u, i ) in the input matrix, [ L ] u,i represents the level of gene expression for sample u at location i of the DNA, and is reported as integer between 0 and L max .
Similarly to recommendation systems (Section 2.2), the training set is obtained as p u,i :=
[ L ] u,i /L max .
Thus, p u,i denotes the (empirical) probability that the gene is expressed in sample u at location i .
4 After running Algorithm {+
 ,+} we use Algorithm 2 to mine causal relations.
As this is unsupervised learning setting, we include the binary MF (Slawski et al., 2013) method.
Results: While Fig 3a shows the training performance for several values of D , Figure 3b plots the corresponding inﬂuence score that is obtained with our method.
Fig 3a reveals a huge gap ( ≈ 3 × less) between the training error of Algorithm 1 and Binary MF (unlike Figure 2a where both algorithms yield similar performance).
This drastic degradation in the performance of binary MF compared to the small artiﬁcial of Figure 2a may be attributed to increasing the data/problem size (though we were unable to empirically verify this claim).
Thus, we opted to use the solution of Algorithm 1 as [Figure] a basis for ﬁnding causal relations.
We identiﬁed 10 DNA locations corresponding to the highest β i as: { 813 , 250 , 774 , 706 , 380 , 49 , 477 , 162 , 740 , 702 } .
For instance, the highest inﬂuence score of .
1612 at DNA location 813 allowed to identify a set, S , of 161 different locations which are causally related to DNA location 813 .
More speciﬁcally, the expression (or absence) of a given gene in location 813 implies its expression (or absence) in all other DNA locations in S .
While similar relations are possible using gene analysis methods, the above causal relations follow from our rigorous mathematical framework.
We also ran experiments on the ML 1M dataset (10x larger that ML 100K) and observed that main conclusions were the same.
For lack of space, we instead opted to include different datasets such as the REGED0 dataset (5x larger than ML 100K) to exemplify another application of the approach.
However, we note that a large-scale implementation of the method may be challenging at this stage.
As mentioned earlier, we are currently investigating complexity reduction methods before running experiments involving large datasets.
Rather, the current work is intended a proof of concept of the usefulness of such an approach.
C ONCLUSION
We have proposed a framework for learning a Kolmogorov model, associated with a collection of binary random variables.
Interpretability of the model (as deﬁned by logical implication and causality) was harnessed by deriving causal relations, i.e., by ﬁnding sufﬁcient conditions that bind outcomes of certain random variables.
We also proposed an algorithm for computing a Kolmogorov model, a combinatorial non-convex problem, and showed its convergence to a stationary point of the problem using results from block-coordinate descent.
The combinatorial nature of the problem was addressed using a semi-deﬁnite relaxation.
We also proposed an efﬁcient algorithm to mine for the causal relations inherent to our model.
Our results suggest that increased interpretability and improved prediction, do not cause a signiﬁcant increase in complexity.
We highlight several key issues for future work, e.g., complexity reduction for (5) (leveraging that its dual is piece-wise linear), and a sufﬁcient condition for identiﬁability (by adapting that of Fu et al.
A S UPPLEMENTARY M ATERIAL FOR L EARNING K OLMOGOROV M ODELS FOR B INARY R ANDOM V ARIABLES A. D EFINITIONS A. K OLMOGOROV M ODEL FOR A R ANDOM V ARIABLE
Let (Ω , M , µ ) be a ﬁnite probability space.
Here Ω is the sample space, the event class M is the power set of Ω , and µ assigns probabilities to the sets in M .
Let X u,i denote a doubly-indexed set of random variables on the given probability space, having output alphabet in A , where elements of
A are indexed by z , i.e., A ( z ) represents the z th element in A .
Also, let P [ X u,i =
A ( z )] ∈ [0 , 1] denote the probability that outcome A ( z ) occurs, for 1 ≤ z ≤ |A| .
We let Ω = { ω d | 1 ≤ d ≤ D } , where { ω d } the set of all D -elementary events.
Since X u,i is binary, i.e., A = { 1 , 2 } , we write the
KM for X u,i as,
[Equation]
where θ u is a Probability Mass Function (PMF) vector on the unit simplex, P , and { ψ i, 1 , ψ i, 2 } ∈ B D are binary indicator vectors representing the support of its probability measure.
Moreover, the last equality follows from ψ i, 1 + ψ i, 2 = 1 , which in turn follows from the outcomes of each RV summing to one.
Since X u,i is binary, it is fully characterized by considering one outcome,
[Equation]
A. R ELATED W ORK
Matrix Factorization Methods: Note that, ( Q ) can be re-written as a low-rank matrix factorization problem, over the set of binary and stochastic matrices (see Appendix A.5.
Thus, the proposed approach is connected to factorization methods: Matrix Factorization (MF) (Koren et al., 2009),
Nonnegative Matrix Factorization (Lee & Seung, 2001), SVD (Cai et al., 2010) (and their many variants/extensions) have gained widespread applicability, covering areas in sound processing, (medical) image reconstruction, recommendation systems and prediction problems (Davenport & Romberg, 2016).
These techniques model elements of the training set as inner product of two arbitrary vectors :
Despite their success, the performance is inherently tied to the validity of that model , and consequently the extent to which these assumption hold.
However, this inner product does not represent a RV (in a mathematical sense), when viewed in the context of the proposed model (see Section 2.1).
Consequently, the analytical guarantees of Section 4, which underpin the causal relations, do not hold for general factorization methods.
Nonnegative Sparse MF: Hoyer (2004) numerically showed that the two low-rank components that are offered by sparse nonnegative MF (a variant of MF) provide insights on the data they model.
While our proposed method also provides insight among the data (via the causal relations), the proposed KM and the resulting causal relations are based on established axioms in probability.
Unlike the relations offered by nonnegative sparse MF which are shown empirically for a handful of examples, the causal relations (Section 4) hold analytically.
Binary MF: We underline that most MF methods also rely on BCD methods (alternating minimization), for which globally optimal solutions to each subproblem are needed for convergence.
Thus, these methods cannot be directly applied to ( Q ) , due to the binary constraints on ψ i .
The authors are unaware generic solution approaches for ( Q ) .
Nonetheless, the Binary MF method (Slawski et al., 2013) can solve ( Q ) when the problem is feasible and the training set spans the entire data set, i.e.,
K = D .
However, the method operates in the unsupervised learning setting only, without providing prediction (see Section 5).
This indeed limits the applicability of binary MF to practical scenarios, since real-world data will have missing/erroneous data.
Clustering Methods: Consider a special case of ( Q ) , where ψ i is constrained to have one nonzero element.
The resulting problem becomes the well-known K -means clustering (Lloyd, 2006).
The K-means algorithms (and its variants K-medoids, fuzzy K-means and K-SVD), have become pervasive in an abundance of applications such as clustering, classiﬁcation, image segmentation,
DNA analysis, online dictionary learning, source coding, etc.
Our approach generalizes K-means , by allowing for overlapping clusters.
While a similar generalization of the classical K-means algorithm was considered in (Whang et al., 2015), the number of points per cluster is determined explicitly.
Nonnegative Models: Non-Negative Models (NNMs) (Stark, 2016a) are recent attempts at interpretable models.
For reasons of computational tractability (Stark, 2016a), NNMs are deﬁned by relaxing ψ i in (1).
However, this relaxation impairs the highly interpretable nature of the model in (1), making causal relations less accurate .
A. K NOWN CONTEXT - DEPENDENT MAP
Consider the case where the elementary events correspond to each of the movie genres, i.e., Ω = { ω 1 , · · · , ω D } = { “Action” , · · · , “SciFi” } .
We refer to this as a known context-dependent map : P [ X u,i = 1] =
ψ
Ti θ u = t
Ti θ u is expressed as a convex/stochastic mixture of movie genres.
These models are a ’holy grail’ for recommendation systems, due to the direct interpretation that they provide.
In this case, t i is a binary genre tag vector with [ t i ] m = 1 , ∀ m ∈ { D } , if movie i belongs to movie genre m , and zero otherwise.
Now consider a genie-aided setting, where the set of movie genre tags { t i } , are known (and consequently { ψ i } are known as well).
Then, one can revisit the optimization problem for determining the KM, ( Q ) , where the indicator vectors are given, { ψ i =
t i } , and the optimization is performed over the PMF vectors only { θ u } :
[Equation]
However, numerical result reveal that the training and test performance of these representations is quite poor.
We tested the performance of this highly interpretable model, by extracting the tag vectors { t i } from the ML 100K dataset, and using them to optimize the corresponding PMF vectors.
We set
D = 19 to match the total number of movie genres for the ML 100K dataset.
As seen in Table 3, the
Training RMSE
[Table]
known context-dependent map have poor performance.
This suggests the existence of a trade-off between the having this content-dependent map, and training/test performance.
A. P ROBLEM F ORMULATION IN M ATRIX F ORM
Let Ψ =
[ ψ 1 , · · · , ψ I ] , Ψ ∈ B D
× I denote the aggregate indicator matrix (containing all the individual indicator vectors), Θ =
[ θ 1 , · · · , θ U ] , Θ ∈ R D × U + the aggregate matrix of PMF vectors, and [ P ] ( u,i ) =
p ( u,i ) , ∀ ( u, i ) ∈
U
×
I , P ∈ R
U × I + the aggregate matrix of known probabilities.
Then, ( Q ) can be written in equivalent matrix form,
[Equation]
where ◦ denotes the Hadamard product, and M ∈ B
U
× I is a mask matrix having M u,i = 1 , ∀ ( u, i ) ∈
K . A. M AIN R ESULTS
Below, we summarized the results used in the paper; see Authors (Oct 2017) for the proofs.
We use following known result to ﬁnd the descent direction for the FW method (the proof is known).
Proposition 2 Consider the following Linear Program (LP),
[Equation]
Its optimal solution is given by x
[Equation]
= e j (cid:63) , where j (cid:63) = argmin 1 ≤ j ≤ n c T e j
Thus, the solution reduces to searching over the vector c .
[Equation]
We show the convergence of the FW algorithm (Table 1).
Proposition 3 Let θ (cid:63)u be the optimal solution to ( Q 1 ) .
Then the sequence of iterates { θ ( k ) u } satisﬁes (Jaggi, 2013)[Theorem 1],
[Equation]
Proof: The linear convergence rate for all FW variants, was proved in Jaggi (2013)[Theorem 1].
Quasi-optimality of SDR:
The question was studied extensively in the context of binary detection for multi-antenna communication (Tan & Rasmussen, 2001).
Interestingly, ( Q 2 ) can be recast as a noiseless binary detection problem, where SDR has been to be optimal.
The results is formalized below.
Proposition 4 Let g ( ψ (cid:63)i ) and g ( ˆ ψ i ) denote the optimal solutions to the binary QP in ( Q 2 ) , and
its
SDR after randomization (Table 2), respectively.
The approximation quality is deﬁned as (Luo et al., 2010),
[Equation]
It holds that η = 1 , with probability 1 − exp −O
( D ) , asymptotically in D .
Thus, the relaxation is quasi-optimal.
[Equation]
Proof: See (Authors, Oct 2017).
Lemma 2 Let t n (cid:44) E ( { ψ ( n ) i } , { θ ( n ) u } ) , n = 1 , 2 , ... be the sequence of iterates, resulting from the updates in Algorithm 1.
Then, { t n } is non-increasing in n , and converges to a stationary point of ( Q ) in (2) , almost surely.
[Equation]
Proof: The convergence is shown in (Authors, Oct 2017).
A. A DDITIONAL N UMERICAL R ESULTS
Prediction Performance (Setting 2): Since the range of the predicted variable is different for MF/NMF/SVD++, and KM/NNM, we use the normalized test RMSE, i.e., NRMSE =
η ( (cid:80) ( u,i ) ∈
¯ K | [ R ] ( u,i ) − ˆ R u,i | 2 / | ¯ K| ) 1 / 2 where ¯ K is the test set, and η = ( R max − R min ) − 1 = 1 / 4 is the normalization for MF/NMF/SVD++.
For KMs/NNMs the same metric reduces to NRMSE =
(cid:16)(cid:80) (
[Equation] [Equation]
K|
[Equation]
The best values for λ u and µ i , were picked from a coarse two-dimensional grid by cross-validation, using a held-out validation set.
The Normalized
RMSE results are shown in Table 4.
We observe a signiﬁcant gap between KMs, and well known collaborative ﬁltering methods, especially as D increases.
Moreover, the drop in performance for
NNMs for increasing D may be due to over-ﬁtting.
Asymptotic optimality of SDR for ψ -step solution: Table 5 is a numerical validation of Proposition 4 where we computed the error rate of SDR (compared to the exhaustive search), aggregated over all iterations.
We observe that the approximation error decreases, with increasing D (following
Proposition 4).
[Table] [Table]