Efﬁciently Factorizing Boolean Matrices using Proximal Gradient Descent
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Addressing the interpretability problem of NMF on Boolean data, Boolean Matrix
Factorization (BMF) uses Boolean algebra to decompose the input into low-rank
Boolean factor matrices.
These matrices are highly interpretable and very usefulin practice, but they come at the high computational cost of solving an NP-hardcombinatorial optimization problem.
To reduce the computational burden, wecontinuously relax BMF using a novel elastic-binary regularizer, from which wederive a proximal gradient algorithm.
Through an extensive set of experiments,we demonstrate that our method works well in practice: On synthetic data, weshow that our algorithm converges quickly, recovers the ground truth precisely, andestimates the simulated rank robustly.
On real-world data, we improve upon thestate of the art in recall, loss, and runtime, and a case study from the medical domainconﬁrms that our results are easily interpretable and semantically meaningful.
Introduction
Discovering groups in data and expressing them in terms of common concepts is a central problem inmany scientiﬁc domains and business applications, including cancer genomics [24], neuroscience[14] and recommender systems [19, 9].
This problem is often addressed using variants of matrixfactorization , a family of methods that decompose the target matrix into a set of typically lowrank factor matrices whose product approximates the input well.
Prominent examples of matrixfactorization are Singular Value Decomposition (SVD) [11], Principal Component Analysis (PCA)[11] and Nonnegative Matrix Factorization (NMF) [32, 22, 23].
These methods differ in how theyconstrain the matrices involved: SVD and PCA require orthogonal factors, while NMF constrains thetarget matrix and the factors to be nonnegative.
SVD, PCA, and NMF achieve interpretable results—unless the data is Boolean.
In this case, whichis ubiquitous in the real world, their results are hard to interpret directly, because the input domaindiffers from the output domain, thus requiring post-processing to extract useful information.
Boolean
Matrix Factorization (BMF) addresses this problem by seeking two low-rank Boolean factor matriceswhose Boolean product is close to the Boolean target matrix [25].
The output matrices, now lying inthe same domain as the input, are interpretable and useful, but they come at the computational cost ofsolving an NP-hard combinatorial optimization problem [31, 25, 26].
To make BMF applicable inpractice, we need efﬁcient approximation algorithms.
There are many ways to approximate BMF—for example, by exploiting its underlying combinatorialor spatial structure [25, 6, 5], using probabilistic inference [38, 37, 36], or solving the related BiClustering problem [29, 30].
Although these approaches achieve impressive results, they fall shortwhen the input data is large and noisy.
Hence, we take a different approach to overcome BMF’scomputational barrier.
Starting from an NMF-like optimization problem, we derive a continuousrelaxation of the original BMF problem that allows intermediate solutions to be real-valued.
Inspired by the elastic-net regularizer [44], we introduce the novel elastic binary ( E LB ) regularizer to regularizetowards Boolean factor matrices.
We obtain an efﬁcient-to-compute proximal operator from our
E LB regularizer that projects relaxed real-valued factors towards being Boolean, which allowsus to leverage fast gradient-based optimization procedures.
In stark contrast to the state of theart [16, 17, 18], which requires heavy post-hoc post-processing to actually achieve Boolean factors,we ensure a Boolean outcome upon convergence, by gradually increasing the projection strengthusing a regularization rate .
We combine our relaxation, efﬁcient proximal operator, and regularizationrate into an Elastic Boolean Matrix Factorization algorithm (E LBMF ) that is scalable to large data,results in accurate reconstructions, and does so without relying on heavy post-processing procedures.
Not conﬁned to BMF, however, E LB and its rate can regularize, e.g., binary MF or bi-clustering [18].
In summary, our main contributions are as follows:
We introduce the E LB regularizer.
We overcome the computational hardness of BMF leveraging a novel relaxed BMF problem.3.
We efﬁciently solve the relaxed BMF problem using an optimization algorithm based on
proximal gradient descent.
The remainder of the paper proceeds as follows.
In Sec. 2, we formally introduce the BMF problemand its relaxation, deﬁne our E LB regularizer and its proximal point operator, and show how toensure a Boolean outcome upon convergence.
We discuss related work in Sec. 3, validate our methodthrough an extensive set of experiments in Sec. 4, and conclude with a discussion in Sec. 5.
Theory
Our goal is to factorize a given Boolean target matrix into at least two smaller, low-rank Booleanfactor matrices, whose product comes close to the target matrix.
Since the factor matrices are Boolean,this product follows Boolean algebra, i.e., it is identical to the standard outer product on a ﬁeld whereaddition obeys 1 + 1 = 1 .
We deﬁne the Boolean product between two Boolean matrices as
[Equation]
where U ∈ { 0 , 1 } n × k , V ∈ { 0 , 1 } k × m , and U ◦ V ∈ { 0 , 1 } n × m
This gives rise to the BMF problem.
Problem 1 (Boolean Matrix Factorization)
For a given target matrix A ∈ { 0 , 1 } n × m , a givenmatrix rank N (cid:51) k ≤ min { n, m } , and let A ⊕ B be logical exclusive or, discover the factor matrices
U ∈ { 0 , 1 } n × k and V ∈ { 0 , 1 } k × m that minimize
[Equation]
While beautiful in theory, we cannot solve this problem exactly for all
but the smallest matrices, as itis NP-complete [26].
In practice, we have to rely on approximations.
To do this, we relax the Booleanconstraints of Eq.
(2) to allow non-negative, non-Boolean ‘intermediate’ factor matrices during theoptimization, allowing us to use linear algebra rather than Boolean algebra.
In other words, we solvethe non-negative matrix factorization (NMF) [32]
[Equation]
subject to U ∈ R n × k + and V ∈ R k × m
+ .
We can solve this problem efﬁciently, e.g., via a Gauss-Seidelscheme.
Although efﬁcient, using plain NMF, however, disregards the Boolean structure of ourmatrices and produces factor matrices from a different domain, which are consequently hard tointerpret and potentially very dense.
To beneﬁt from efﬁcient optimization and still arrive at Booleanoutputs, we allow real-valued intermediate solutions and regularize them towards becoming Boolean.
To steer our optimization towards Boolean solutions, we penalize non-Boolean solutions usinga regularizer.
This idea has been explored in prior work.
There exists the l 1 -inspired P RIMPregularizer [16], which is − κ [ −| 1 − 2 x | + 1] for values inside [0 , 1] and ∞ otherwise, and the l 2inspired bowl-shaped regularizer [43], which is λ ( x 2 − x ) 2 / 2 everywhere on the real line.
Although both have been successfully applied to BMF, both also have undesirable properties: The P RIMPregularizer penalizes well inside the interval [0 , 1] but is non-differentiable on the outside, while thebowl-shaped regularizer is differentiable and penalizes well outside the interval [0 , 1] but is almostﬂat on the inside.
Hence, both regularizers are problematic if used individually.
Combining them,however, yields a regularizer that penalizes non-Boolean values well across the full real line.
Tocombine l 1 - and l 2 -regularization, we use the elastic-net regularizer ,
[Equation]
which, however, only penalizes non-zero solutions [44].
To penalize non-Boolean solutions, wecombine two elastic-net regularizers into our (almost W-shaped) E LB regularizer ,
[Equation]
where X ∈ { U , V } .
In Fig.
1, we show all three regularizers in the range of [ − 1 , 2] , for λ
= κ = 0 .
We see that only the E LB regularizer penalizes non-Boolean solutions across the full spectrum, andsummarize our regularized relaxed BMF as follows.
Problem 2 (Elastic Boolean Matrix Factorization)
For a given target matrix A ∈ { 0 , 1 } n × m anda given matrix rank N (cid:51) k ≤ min { n, m } , discover the factor matrices U ∈ R n × k + and V ∈ R k × m +that minimize
[Equation]
Although this is a relaxed problem, it is still non-convex.
Therefore, we cannot straightforwardlysolve it.
However, the problem is suitable for the Gauss-Seidel optimization scheme.
That is, wealternatingly ﬁx one factor matrix to optimize the other.
By doing so, we generate a sequence
[Equation]
of simpler-to-solve sub-problems, until convergence.
Now, each sub-problem is again a sum oftwo f ( X ) + R ( X ) functions, where f is the loss (cid:107) · (cid:107) 2 F , and R ( X ) is the regularizer.
This allowsus to follow a proximal gradient approach, i.e., we use Proximal Alternating Linear Minimization(PALM)
In a nutshell, we minimize a sub-problem by following the gradient ∇ f of f , tothen use the proximal operator for R to nudge its outcome towards a Boolean solution.
That is, forthe gradients ∇ U f = UVV (cid:62) − AV (cid:62) and ∇ V f
= U (cid:62) UV − U (cid:62) A , we compute the step
[Equation] [Equation]
where η represents the step size, which we compute in terms of Lipschitz constant, rather than relyingon a costly line-search [7].
To further improve the convergence properties, we make use of an inertialterm that linearly combines X t with X t − 1 before applying Eq.
(8) (see [34] for a detailed description).
We now derive the proximal operator for the E LB regularizer, before discussing how we ensure thatthe factor matrices are Boolean, and summarizing our approach as an algorithm.
2.1 Proximal Mapping
Let X ∈ { U , V } , to solve the sub-problem from Eq.
[Equation]
κλ
[Equation]
=
[Equation]
+
[Equation] [Equation]
we need a proximal operator [33] that projects values towards a regularized point.
Formally in
Appendix A, here in a nutshell: this operator is the solution to
[Equation]
As Eq.
(10) is coordinate-wise solvable, we can reduce this equation to a scalar proximal operator
[Equation] [Equation]
[Figure]
Input: Matrix A ∈ { 0 , 1 } n × m , rank k ∈ N
Output: Factors U ∈ { 0 , 1 } n × k , V ∈ { 0 , 1 } k × m initialize U , V uniformly at random for t = 1 , 2 , . . .
until convergence do
U ← arg reduce U (cid:96) κλ t (A , U , V) V ← arg reduce
V (cid:96) κλ t
(A , U , V) end
Although this scalar function may seem simple, due to R , it is non-convex, which—in general—rulesout the existence of unique minima.
We can, however, exploit the W-shape of R .
That is, fromthe perspective of a ﬁxed X ij , the regularizer is locally convex after a case distinction.
Startingwith the case of X ij = x, x ≤ 1 / 2 , we can simplify Eq.
(11) to 1 2 ( x − y )
+ λ (cid:48) / 2 y 2 + κ | y | , lettingλ (cid:48)
= 2 λ .
Setting its derivative to 0 , we get to y =
x
− κ sign( y ) .
Asserting that a least-squaressolution will always be of the same sign, we substitute the sign of x with sign( y ) .
Repeating thesesteps analogously for x > 1 / 2 , we obtain our proximal operator:
Deﬁnition 1 (Proximal Operator)
Given regularization coefﬁcients κ and λ , our proximal operatorfor matrix X is prox R ( X ) =
[prox κλ ( X ij )] ij , where prox κλ is the scalar proximal operator
[Equation]
Although not strictly necessary, to improve the empirical convergence rate, we would like to constrainour factor matrices to be non-negative.
As our proximal operator does not account for this, we imposenon-negativity by using the alternating projection procedure to combine the non-negativity proximaloperator [33] with Eq.
(12) into max { 0 , prox κλ ( x ) } .
2.2 Ensuring Boolean Factors
Our proximal operator only nudges the factor matrices towards becoming Boolean.
We, however,want to ensure that our results are Boolean.
To this end, the state-of-the-art method P RIMP reliesheavily on post-processing, performing a very expensive joint two-dimensional grid search to guessthe ‘best’ pair of rounding thresholds, which are then used to produce Boolean matrices.
Althoughthis tends to work in practice, it is an inefﬁcient post-hoc procedure—and thus, it would be highlydesirable to have Boolean factors already upon convergence.
To achieve this without rounding orclamping, we revisit our regularizer, which binarizes more strongly if we regularize more aggressively.
Consequently, if we regularize too aggressively, we converge to a suboptimal solution, and if weregularize too mildly, we do not binarize our solutions.
To prevent subpar solutions and still binarizeour output, we start with a weak regularization and gradually increase its strength.
Considering Eq.
(12), we see that a higher regularization strength increases the distance over which ourproximal operator projects.
Thus, if we set the l 1 -distance controlling κ too high, we will immediatelyleap to a Boolean factor matrix, which will terminate the algorithm and yield a suboptimal solution.
Regulating the l 2 -distance controlling λ is a less delicate matter.
Hence, we gradually increase λ toprevent a subpar solution and achieve a Boolean outcome, using a regularization rate λ t = λ · ν tthat gradually increases the proximal distance at a user-deﬁned rate ν t ≥ 0 ∀ t ≥ 0 .
In case E LBMFhas stopped without convergence, we bridge the remaining integrality gap by projecting the outcomeonto their closed Boolean counterpart, using our proximal operator (see Fig. 2).
We summarize the considerations laid out above as E LBMF in Alg. 1.
The computational complexity of
E LBMF is bounded by the complexity of computing the gradient, which is identical to the complexityof matrix multiplication.
Therefore, for all practical purposes, E LBMF is sub-cubic O (cid:0) n 2 . 807 (cid:1) using
Strassen’s algorithm.
Related Work
Matrix factorization is a well-established family of methods, whose members, such as SVD, PCA, or
NMF, are used everywhere in machine learning.
Almost all matrix factorization methods operateon real-valued matrices, however, while BMF operates under Boolean algebra.
Boolean Matrix
Factorization originated in the combinatorics community [28] and was later introduced to the datamining community [25], where many cover-based BMF algorithms were developed [6, 5, 25, 27].
In recent years, BMF has gained traction in the machine learning community, which tends totackle the problem differently.
Here, relaxation-based approaches that optimize for a relaxedbut regularized BMF [16, 15, 43] are related to our method, but they differ especially in theirregularization.
Hess et al.
[16] introduce a regularizer that is only partially differentiable andrely heavily on post-processing to force a Boolean solution, and Zhang et al.
[43] regularize onlyweakly between 0 and 1 .
In contrast, our regularizer penalizes well across the full spectrum andyields a Boolean outcome upon convergence.
Building on a thresholding-based BMF formulation,
Araujo et al.
[3] also considers relaxations to beneﬁt from gradient-based optimization.
Otherrecent approaches build on probabilistic inference .
Rukat et al.
[38, 37, 36], for example, combine
Bayesian Modeling and sampling into their logical factor machine.
A similar direction is taken by
Ravanbakhsh et al.
[35], who use graphical models and message passing, and Liang et al
[24], whocombine MAP-inference and sampling.
A different, geometry-based approach lies in locating densesubmatrices by ordering the data to exploit the consecutive-ones property [41, 39].
Since BMF isessentially solving a bipartite graph partitioning problem, it is also closely related to Bi-Clusteringand Co-Clustering [29, 18].
Neumann and Miettinen [30] use this relationship to efﬁciently solve
BMF by means of a streaming algorithm.
Although there are many different approaches to BMF, itsbiggest challenge to date remains scalability [26].
Experiments
We implement E LBMF in the Julia language and run experiments on 16 Cores of an AMD EPYC7702 and a single NVIDIA A100 GPU, reporting wall clock time.
We provide the source code,datasets, synthetic dataset generator, and other information needed for reproducibility.
1 We compare
E LBMF against six methods: four dedicated BMF methods (A SSO [25], G RECOND [6], O R M [37],and P RIMP [16]), one streaming Bi-Clustering algorithm S OFA [30], one elastic-net-regularized NMFmethod leveraging proximal gradient descent (NMF [32, 22, 23]), and one interpretable Booleanautoencoder (B INAPS [10]).
Since NMF outputs non-negative factor matrices, rather Boolean matrices,we cannot compare against NMF directly, so we clamp and round its solutions to the nearest Booleanoutcome.
To fairly compare against B INAPS , we task it with autoencoding the target matrix as areconstruction, given the matrix ranks from our experiments as the number of latent dimensions.
Weperform three sets of experiments.
First, we ascertain that E LBMF works reliably on synthetic data.
Second, we verify that it generally performs well on real-world data.
And third, we illustrate that itsoutputs are semantically meaningful through an exploratory analysis of a biomedical dataset.
4.1 Performance of E LBMF on Synthetic Data
In the following experiments, we ask three questions: ( 1 ) Does E LBMF converge?; ( 2 ) How well does
E LBMF recover the information in the target matrix?
; and ( 3 ) Does E LBMF estimate the underlying Boolean matrix rank correctly?
To answer these questions, we generate synthetic data with knownground truth as follows.
Starting with an all-zeros matrix, we randomly create rectangular, nonoverlapping, consecutive areas of ones called tiles , each spanning between 5 and 10 rows and columns,thus inducing matrices with varying densities.
We then add noise by setting each cell to 1 , uniformlyat random, with varying noise probabilities.
How does E LBMF converge?
To study how our method converges to a Boolean solution, wequantify relevant properties of the sequence of intermediate solutions (cf.
Eq.
First, to understandhow quickly and stably E LBMF converges to a Boolean solution, we quantify the Boolean gap ,(cid:88)
X ∈{ t ,V t } | X | − 1 (cid:88) x ∈ X min {| x | , | x − 1 |} .
[Figure]
hasconverged keepsonchanging
Iteration t
F li pp e d B it s (c)
Second, to understand when we can safely round intermediate almost-Boolean solutions withoutlosing information, we compute for the rounded intermediate reconstruction B , the fraction of bitsthat ﬂip from iteration t to iteration t + 1 as the Hamming process
[Equation]
As shown in Fig.
2a, we achieve an almost-Boolean solution without any rounding after around 250epochs, continuing until we reach a Boolean outcome.
This is also the point at which the roundedintermediate solution and its relaxation are almost identical, as illustrated in Fig.
2b.
Considering therounded solution sequence in Fig.
2c, we observe that E LBMF goes through an erratic bit-ﬂippingphase in the beginning, followed by only minor changes in each iteration until iteration t = 100 .
Afterwards, E LBMF has settled on a solution—under our regularization rate.
When using constantregularization instead, we continue to observe bit ﬂips until the end of the experiment (Fig. 2c).
Theconstant regularization ﬁts the data slightly better than the regularization rate (Fig. 2b), which isunsurprising, given that its factors are less constrained than the Boolean factors we use otherwise.
However, under constant regularization, the Boolean gap hardly decreases over time (Fig. 2a).
Inother words, our regularization works well, and it allows us to safely binarize almost-convergedfactors that are (cid:15) -far from being Boolean by means of, e.g., our proximal operator.
How well does E LBMF recover the information in the target matrix?
Having ensured thatour method converges stably and quickly, we would like to assess whether it also converges to ahigh-quality factorization.
To this end, we generate synthetic 40 × 30 matrices containing 5 randomtiles, under additive noise levels between 0% (no noise) and 50% .
We then compute the fraction ofones in target
A that is covered by the reconstruction B
= U ◦ V , i.e., the recall (higher is better)
[Equation]
To ensure that we ﬁt the signal in the data, we additionally report the recall regarding the generating,noise-free ground-truth tiles A ∗ , denoted as recall ∗ .
Finally, to rate the overall quality of thereconstruction including zeros , we compute the Hamming similarity (higher is better) between thetarget matrix and its reconstruction
[Equation]
We run each method on our synthetic datasets, targeting a matrix rank of 5 .
To account for randomﬂuctuations, we average over 10 randomly drawn sets of 5 ground-truth tiles per 10% increment innoise probability.
In Fig.
3, we show similarity, recall, and recall ∗ .
We observe that in the noiselesscase ( 0% ), all methods except B INAPS recover the 5 ground-truth tiles with high accuracy, butonly A SSO and E LBMF do so with perfect recalls.
Starting with as little as 10% noise, both recallsof A SSO , G RECOND , S OFA , NMF , and B INAPS deteriorate quickly, while the similarity and bothrecalls of P RIMP and E LBMF remain high.
In fact, E LBMF and P RIMP perform similarly across theboard—which is highly encouraging, as unlike P RIMP , E LBMF does not require post-processing.
For A SSO and G RECOND , recall and similarity drop considerably, but they exhibit a slightly higherrecall ∗ .
This means that these methods are robust against noise, but they fail to recover the remaininginformation.
Starting low, O R M’s recalls increase jointly with the noise level, suggesting that cleandata is problematic for O R M.
[Figure]
r it y [Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
R eca ll (b) 0 0 .
Noise
R eca ll ( G r ound - T r u t h )
E LBMF P RIMP
NMF A SSO
G RECOND B INAPS
O R M S OFA (
[Figure] [Figure]
[Figure] [Figure]
[Figure]
AIC
Does E LBMF estimate the underlying Boolean matrix rank correctly?
When the target rank isknown, we can immediately apply E LBMF to factorize the data.
In the real world, however, the targetmatrix rank might be unknown.
In this case, we need to estimate an inexact appropriate choice fromthe data, and we use synthetic data to ensure that E LBMF does so correctly.
Since a higher matrix rank usually also means a better ﬁt, selecting the best rank according to recall,loss, or similarity leads to overﬁtting—unless we properly account for the growth in model complexity.
There are many model selection criteria that penalize complex models, such as AIC [1], Bai & Ng’scriteria [4, 2], Nuclear-norm regularizing [21, 13], the information-theoretic Minimal Description
Length principle (MDL) [12], or (Decomposed)
Normalized Maximum Likelihood [20, 42].
Following common practice and motivated by its practical performance in preliminary experiments, wechoose MDL.
That is, we select the minimizer of sum the log binomial l ( X ) =
log (cid:0) | X | (cid:107) X (cid:107) 1 (cid:1) of theerror matrix and the rows and columns of our factorization (assumed to be i.i.d.)
[Equation]
To validate whether E LBMF recovers the right rank, we synthetically generate a ground-truth 400 ×
300matrix of rank 10 with 10% additive noise.
In Fig.
4, we show MDL, AIC, and Bai & Ng’s ﬁrstcriterion for each rank up to 30 , ﬁnding that our method precisely discovers the true matrix rank.
4.2 Performance of E LBMF on Real-World Data
Having ascertained that E LBMF works well on synthetic data, we turn to its performance in thereal world.
Here, we use nine publicly available datasets and anonymized 2 from different domains.
[Figure]
ORG
GROUPLENS .
ORG /
[Table]
/ MOVIELENS KAGGLE .
COM / DATASETS / NETFLIX - INC / NETFLIX - PRIZE - DATA
G e no m e s
T C GA G B M L GG
C e r e b e ll u m S
t r
i ng P a t e n t s
M ov i e l e n s
N e t ﬂ i x 0 .0 .0 .0 .
R e l a ti v e L o ss (a)
G e no m e s
T C GA G B M L GG
C e r e b e ll u m S
t r
i ng P a t e n t s
M ov i e l e n s
N e t ﬂ i x
T i m e [ s ]
B INAPS S OFA
A SSO G RECOND
NMF P RIMP
O R M E LBMF (b)
To cover the biomedical domain , we extract the network containing empirical evidence of proteinprotein interactions in homo sapiens from the S TRING database.
From the G RAND repository, wetake the gene regulatory networks sampled from Glioblastoma (GBM) and Lower Grade Glioma(LGG) brain cancer tissues, as well as from non-cancerous Cerebellum tissue.
The TCGA datasetcontains binarized gene expressions from cancer patients, and we further obtain the single nucleotidepolymorphism (SNP) mutation data from the 1 k Genomes project, following processing steps fromthe authors of B INAPS [10].
In the entertainment domain , we use the user-movie datasets Movielensand Netﬂix , binarizing the original 5 -star-scale ratings by setting only reviews with more than 3 .
5stars to 1 .
Finally, as data from the innovation domain , we derive a directed citation network betweenpatent groups from patent citation and classiﬁcation data provided by PatentsView .
For each datasetwith a given number of groups, such as cancer types or movie genres, we set the matrix rank k to33 (TCGA), 28 (Genomes), 136 (Patents), 20 (Movielens), and 20 (Netﬂix).
When the number ofsubgroups is unknown, we estimate the rank using MDL and E LBMF , resulting in 100 (GBM), 32(LGG), 100 (String), and 450 (Cerebellum).
As we can achieve a high similarity with an all-zeros reconstruction (in the case of sparse data), or aperfect recall with an all-ones reconstruction, we also report the relative loss (lower is better),(cid:107)
A (cid:107) 1 (cid:107)
A − U ◦ V (cid:107) 1 , between the target matrices and their reconstructions.
In Fig.
5, we show recall and runtime of A SSO ,
G RECOND , O R M, S OFA , B INAPS , NMF , and E LBMF , applied to all datasets.
We see in Fig.
5a that
E LBMF often has the lowest loss, outperforming A SSO , G RECOND , O R M, S OFA , B INAPS , and NMF .
In many cases, E LBMF also has a loss lower than P RIMP .
We observe the largest gap between thetwo on the Cerebellum dataset, where P RIMP ’s grid search procedure fails to ﬁnd suitable thresholds.
This is a highly impressive result, as E LBMF does not necessitate heavy post-processing, which isrequired by P RIMP .
In Fig.
5b (log scale), we see how post-processing degrades runtime, observingthat P RIMP is always much slower than E LBMF .
For example, P RIMP does not ﬁnish on the Netﬂixdata, where only E LBMF and B INAPS succeeded.
Although similarly fast as E LBMF , O R M resultsin a signiﬁcantly higher loss.
Notably, without much extra care, NMF ’s performance is surprisinglygood: it often outperforms the dedicated BMF methods A SSO , G RECOND , O R M, and S OFA .
4.3 Exploratory Analysis of Gene Expression Data with E LBMF
Knowing that E LBMF performs well quantitatively, we ask whether its outputs are also interpretable.
To this end, we take a closer look at the TCGA data, which contains the expression levels of 20 530genes from 10 459 patients, who are labeled with 33 cancer types.
Since we are interested in retaininghigh gene expression levels only, we set expression levels to one if their Z -scores fall into the top 5%quantile, and to zero otherwise [24].
We run E LBMF on this dataset, targeting a rank of 33 .
To learn whether our method groups patients meaningfully, we visualize the target matrix and itsreconstruction.
As the target matrix is high-dimensional, we embed both the target matrix and thereconstruction into a two-dimensional space using t-SNE [40] as illustrated in Fig.
6, where each
−
[Figure] [Figure]
−
[Figure]
5 −
[Figure]
(b) 0 5 10 15 20
Index of Estimated Group
I nd e x o f C a n ce r T yp e 0 .0 .0 .0 .
(c)
While the cancer types are hard to differentiate in the embedding of
, they separated into easily distinguishable clusters in the embedding of our reconstruction (b).
In (c), we show the renormalized mutual information between estimated groups and cancer types, and observe that there is an almost 1 -to- 1 correspondence between our estimated groups and the cancer types.
color corresponds to one cancer type.
In Fig.
6a, we show that when embedding the target matrixdirectly, the cancer types are highly overlapping and hard to distinguish without the color coding.
Incontrast, when embedding our reconstruction, depicted in Fig.
6b, we see a clean segmentation intoclusters that predominantly contain a single cancer type.
To better understand these results, we quantify the association between our 33 estimated groups andthe ground-truth cancer types by computing the mutual information matrix.
This matrix is noticeablysparse (Fig. 6c), leading to a clean segmentation.
Upon closer inspection with E NRICHR [8], theassociations we discover turn out to be biologically meaningful.
For example, we ﬁnd that E LBMFassociates a group of 356 genes to patients with thyroid carcinoma .
This group is associated withthyroid hormone generation and thyroid gland development , and statistically signiﬁcantly so—evenunder a strict False Discovery Control, with p -values as low as 2 .
574 × 10 − 8 and 6 .
530 × 10 − 6 .
Conclusion
We introduced E LBMF to efﬁciently factorize Boolean matrices using an elegant and simple algorithmthat, unlike its closest competitors, does not rely on heavy post-processing.
E LBMF considers a novelrelaxed BMF problem, which allows intermediate solutions to be real-valued.
It solves this problem byleveraging an efﬁciently computable proximal operator, derived from the innovative E LB regularizer,and using a regularization rate to obtain Boolean factors upon convergence.
Experimentally, we haveshown that E LBMF works well in practice.
It operates reliably on synthetic data, outperforms thediscrete state of the art, is at least as good as the best relaxations on real data, and yields interpretableresults even in difﬁcult domains—without relying on post-processing.
Limitations Although E LBMF works overall, it has two bottlenecks.
First, by randomly initializingfactors, we start with highly dense matrices, thus prohibiting efﬁcient sparse matrix operations.
Thisis not ideal for sparse datasets that are too large to ﬁt into memory, and future research on sparseinitialization of Boolean factors will beneﬁt not only E LBMF but also many other methods.
Second,the larger the datasets, the higher the cost of computing gradients, and future work might adaptstochastic gradient methods for E LBMF to mitigate this problem.
Broader Impact
E LBMF is a method for factorizing Boolean matrices, which permits interpretable,rather than black box data analysis.
As such, it can help to make explicit any biases that may bepresent in the data.
Just like any other method for interpretable data analysis, the insights they yieldus can be used for good (e.g., insight into cancer-causing mutations) or for bad, and the ultimatequality of the outcome also depends on the behavior of the user.
In our experiments we focus on goodapplications, and only consider anonymized open data.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Sec. 5.(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec. 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Sec. 2 and appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes] See Sec. 4, appendix, andunder dropbox.com/sh/gaoftnf0iyh27xq/AADRlhKxTV1S9LovwwnxVTCYa.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sec. 4 and appendix.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)?
[Yes] See appendix.(d)
Did you include the total amount of compute and the type of resources used (e
g., type of GPUs,internal cluster, or cloud provider)?
[Yes] See Sec. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Sec. 4.
(b) Did you mention the license of the assets?
[Yes] See Sec. 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Sec. 4,
appendix, and under dropbox.com/sh/gaoftnf0iyh27xq/AADRlhKxTV1S9LovwwnxVTCYa.
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[Yes] See Sec. 4.
All datasets are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
informa-
tion or offensive content?
[Yes] See Sec. 4.
All datasets have been anonymized prior to their
publication.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
[N/A]
AppendixA Derivation of the Proximal Operator
In this section, we derive our proximal operator (given in Eq.
(12), repeated below)
[Equation]
x
[Equation]
if x ≤x − κ sign( x − 1) + λ otherwise , (13) from the E LB (Eq.
(4), repeated below)
R ( X ) =
(cid:88) x ∈ X min { r ( x ) , r ( x − 1) } .
Starting with the deﬁnition [33] of the general proximal operator
[Equation]
we observe that this proximal operator is coordinate-wise solvable.
This allows us to derive a scalar proximal 482 operator, which we then apply to each value in the matrix independently.
Substituting the regularizer R with itsscalar version, we obtain a scalar proximal operator (Eq.
[Equation]
which is non-convex, has no unique minima, and, therefore, is not straightforwardly solvable.
We can, however, 485 separate this function into two locally convex V-shaped functions, which are straightforwardly solvable.
Byasserting that its least-squares solution is either less than 1 / 2 (if x ≤ 1 / 2 ), or greater than 1 / 2 (if x > 1 / 2 ), we canaddress each case independently, and merge the outcome into a single piecewise proximal operator.
Case I
In the ﬁrst case, we address the operator for x ≤ 1 / 2 .
For this, we start by simplifying our scalarproximal operator Eq.
(11), by substituting r ( y ) with its deﬁnition, and get
[Equation]
for λ (cid:48) / 2 = λ .
Then, we take its partial derivative for y
[Equation]
which we set to zero, obtaining
[Equation]
=
[Equation]
y
[Equation]
By asserting that we can obtain a better least-squares solution if y has the same sign as x , we can substitute thesign of x with sign( y ) , and get
[Equation]
which concludes the ﬁrst case.
Case II Analogously, we now repeat the steps from above for x > 1 / 2 .
Again, we start by simplifying
Eq.
(11), substituting r ( y − 1)
[Equation]
for λ (cid:48) / 2 = λ .
By taking its partial derivative for y
[Equation]
and setting it to zero, we obtain
[Equation]
Then, asserting that the least-squares solution does not get worse by using the same sign for y − 1 and x − 1 , wecan substitute the sign of x − 1 with sign( y − 1) , and get [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Combining Case I & Case II
Combining the cases above yields our piecewise proximal operator (see
Eq.
(12))prox
[Equation]
sign
[Equation]
x ) if x ≤x − κ sign(
[Equation]
− 1)
+ λ otherwise .
Alternative Proximal Operator Considering Eq.
(11), we notice that the term y − x is squared, whichmeans that there are multiple solutions to this equation.
We derive the alternative operator analogously to the 506 steps above, however, by switching the positions of y and x in f .prox
[Equation]
if
[Equation] [Equation]
otherwise .
Since this operator is denominated by λ (cid:48) − 1 , we need to ensure that λ (cid:48) (cid:54) = 1 .
Because our original proximaloperator in Eq.
(12) is denominated by λ (cid:48) + 1 , and since λ (cid:48) is usually positive, we are not required to take extraprecautions.
Since this is more convenient, we select Eq.
(12) as our proximal operator, rather than taking extra 510 precautions when using Eq.
[Table]
Patents 10100 .
B Reproducibility
In this addendum, we explain ( 1 ) how to obtain the datasets, ( 2 ) how we binarized them, ( 3 ) how to obtain thesource code, and ( 4 ) how we parameterized the algorithms, starting with the dataset descriptions.
Broadly speaking, our datasets cover three domains: the biomedical domain , the entertainment domain , and the 515 innovation domain .
To cover the biomedical domain , we extract the network containing empirical evidence ofprotein-protein interactions in homo sapiens from the S TRING database.
For this, we remove all interactions 517 in the protein-protein network, for which the empirical-evidence sub-score of the S TRING database is zero,retaining only the protein-protein interactions discovered experimentally.
From the G RAND repository, we take 519 the gene regulatory networks sampled from Glioblastoma (GBM) and Lower Grade Glioma (LGG) brain cancertissues, as well as from non-cancerous Cerebellum tissue.
These networks all stem from a method, P ANDA , which 521 extracts gene-regulatory networks from data.
The weights are between − 20 and 20 , where − 20 corresponds toa low interaction likelihood, and 20 corresponds to a high interaction likelihood.
To binarize, we set everything 523 to zero except the weights whose Z -scores are in the upper 5% quantile, retaining the information about likelyinteractions.
The TCGA dataset contains gene expressions from cancer patients.
To binarize these logarithmic 525 expression rates ( log 10 ( x + 1) ), we again only set those weights to one whose Z -scores lie in the upper 5%quantile [24], retaining high gene expressions.
We further obtain the single nucleotide polymorphism (SNP) 527 mutation data from the 1 k Genomes project, and follow the data retrieval steps from the authors of B INAPS [10],which immediately produces a binary dataset.
In the entertainment domain , we use the user-movie datasets 529 Movielens and Netﬂix .
Since we are only interested in recommending good movies, we binarize the original5 -star-scaled ratings, by setting reviews with more than 3 .
5 stars to one, and everything else to zero.
Finally,as data from the innovation domain , we derive a directed citation network between patent groups from patent 532 citation and classiﬁcation data provided by PatentsView .
We binarize this weighted network simply by settingevery non-zero weight to one, retaining all edges in the network.
For each of these publicly 3 available dataset,we give its dimensionality, density, and the matrix rank used in our experiments in Table 1.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
In our experiments in Sec. 4, we compare E LBMF against six methods: three dedicated BMF methods (A SSO [25],
G RECOND [6], and P RIMP [16]), one streaming Bi-Clustering algorithm S OFA [30], one elastic-net-regularized 537 NMF method leveraging proximal gradient descent (NMF [32, 22, 23]), and one interpretable Boolean autoencoder (B INAPS [10]).
The code for A SSO , G RECOND , P RIMP , S OFA , and B INAPS was written by their respective 539 authors and is publicly available.
We implement NMF and E LBMF in the Julia programming language andprovide their source code for reproducibility.
On TCGA , Genomes , Movielens , Netﬂix , and Patents , we set the 541 l 2 -regularizer λ = 0 .
001 , the l 1 -regularizer κ = 0 .
005 , and the regularization rate to ν t = 1 .
0033 t .
On GBM ,
LGG , and Cerebellum , we set the l 2 -regularizer λ = 0 .
001 , the l 1 -regularizer κ = 0 .
001 , and the regularizationrate to ν t = 1 . 0015 t .
We run NMF , E LBMF , P RIMP for at most 1 500 epochs on each dataset.
In the case that
E LBMF reaches its maximum number of iterations without convergence, we bridge the remaining integrality gap 545 simply by applying our proximal operator (see Fig. 2).
To obtain a good reconstruction for P RIMP , we use agrid-width of 0 . 01 .
To obtain a binary solution from NMF , we ﬁrst clamp
and then round its factor matrices uponconvergence.
We set A SSO ’s threshold, gain for covering, and penalty for over-covering each to 1 .
To achieve a 548 better performance with A SSO , we parallelize A SSO on 16 CPU cores.
Further, because A SSO ’s runtime scaleswith the number of columns, we reconstruct the transposed target whenever it has more columns than rows (see
[Figure]
we show similarity (a) and recall (b) w.r.t.
the target matrix, and recall ∗ w.r.t.
the noise-free ground-truth tiles (c), for B INAPS , A SSO , G RECOND , O R M, S OFA ,
NMF , P RIMP , and E LBMF .
C Additional Results on the Performance of E LBMF on Synthetic Data
In this section, we provide additional results on our synthetic experiments in Sec. 4.
Supplementary to our synthetic experiments in Fig.
3, we report their standard deviations, depicted in Fig.
7 555 (a)-(c) as the shaded region around each line representing the mean.
In Fig.
7a, we see that the standard deviationof the similarity score is low, except for A SSO and NMF in the highest noise regime, where they exhibit higher 557 variance.
In the other two ﬁgures, we observe that the standard deviation in Fig.
7b is lower than in Fig.
7c, for
A SSO , G RECOND , NMF , and S OFA .
Overall in Fig.
7, we see that E LBMF , P RIMP , and B INAPS perform most 559 consistently.
[Figure]
Density S i m il a r it y
(a) 0 . Density
R eca ll (b) 0 .
Density
R e l a ti v e L o ss
B INAPS S OFA
G RECOND A SSO
NMF P RIMP
O R M MP E LBMF (c)
In our synthetic experiments in Sec. 4, we simulated data by generating non-overlapping tiles.
In the experiments 561 without tile overlap, we perform rejection sampling.
That is, before we place the next randomly drawn tileinto our matrix, we check for overlap with past placements.
If we detect an overlap, we reject, redraw, and 563 repeat, until we placed the desired number of tiles into our matrix.
In the following we allow the overlap oftiles, depicting results on harder to separate data.
For this, we do not check for overlap, thus inserting each draw 565 into the matrix.
To this end, we generate synthetic matrices as described in Sec. 4, however, this time, allowingtiles to overlap arbitrarily.
In Fig.
7 we show similarity (d), recall (e), and recall ∗ (f) for the overlapping case,observing a similar behavior to Fig.
7 and Fig.
3 across the board.
Figures (a)-(f) show that E LBMF , which does 568 not use any post-processing, achieves highly competitive results for overlapping and non-overlapping tiles, onpar with the strongest competitor, which relies heavily on post-processing.
D Additional Results on the Performance of E LBMF on Synthetic Data withVarying Densities
Here, we provide additional synthetic experiments under a ﬁxed noise level but varying densities.
To this end, we generate synthetic matrices as described in Sec. 4, however, this time, we ﬁx the noise level 574 to 0 .
and vary the width and height of the ground-truth squared tiles from 3 2 to 12 2 , resulting in densitiesbetween 0 . 0375 to 0 .
6 , before noise.
In Fig. 8, we show the similarity, recall, and loss of B INAPS , A SSO , G RECOND , O R M, S OFA , NMF , P RIMP ,
E LBMF , and this time also for MP [37].
We can see that the increasing density affects the performance of all 578 methods, however, it does not affect the performance of all.
methods equally .
A SSO , G RECOND , NMF , P RIMP ,
S OFA , B INAPS , O R M, and E LBMF all improve in recall and loss with increasing density.
The reason is that very 580 tiny tiles in the beginning are very hard to distinguish from noise.
The methods A SSO , G RECOND , and NMF areagain highly competitive, however, still outperformed by P RIMP and E LBMF .
Closely trailing B INAPS , MP, on 582 the other hand, is consistently in the last place, unable to recover the ground-truth tiles, regardless of the densitylevel.
With increasing density, O R M gets slightly worse, before its performance increases dramatically, such that 584 it ﬁnishes on a recall, similarity, and loss similar to S OFA .
The similarity, Fig.
8a, however, is consistently closeto 1 for A SSO , G RECOND , NMF , P RIMP , and E LBMF , but drops with increasing density for S OFA , B INAPS ,
O R M, and MP.
This is because a 0 -heavy reconstruction is getting progressively worse for increasingly dense 587 matrices.
E Additional Results on the Performance of E LBMF on Real-World Data
In this section, we provide additional results on our real-world experiments in Sec. 4.
In particular, we want to answer the question whether A SSO , S OFA , NMF , P RIMP , B INAPS , and E LBMF are 591 sparsity-inducing.
For this, we additionally show similarity scores in Fig.
Since our data is sparse (see
Table 1), we can expect that sparsity-inducing methods would result in a high similarity, even under a high 593 loss.
There, we observe that A SSO , G RECOND , O R M, S OFA , s NMF , P RIMP , and E LBMF yield reconstructionsthat are highly similar to the target, which means that they are sparsity-inducing methods.
Although P RIMP 595 exhibits a high loss (Fig. 5a) on, for example, Cerebellum , it also has a high similarity (Fig. 9).
Hence, P RIMPstill recovers a sparse and interpretable result.
Since B INAPS covers many zeros with its patterns, it results in G e no m e s T C GA G B M
L GG
C e r e b e ll u m S
t r
i ng P a t e n t s M ov i e l e n s N e t ﬂ i x 0 .0 .0 .0 . S i m il a r it y
S OFA B INAPS
G RECOND A SSO
NMF P RIMP
O R M E LBMF dense reconstructions of sparse targets, thus exhibiting lower similarity and high loss.
That, however, does not 598 mean patterns
[Figure]
pseudo
Algorithm
Input:
Target Matrix
A ∈ { 0 , 1 } n × m
Rank k ∈ N , l 1 Regularizer Coefﬁcients κ ∈ R , l 2 Regularizer Coefﬁcients λ ∈ R ,
Regularization Rate ν t ∈ N → R , optional Inertial Parameter β ∈ R +
Output: Factors U ∈ { 0 , 1 } n × k , V ∈ { 0 , 1 } k × m
U
= U 1 ← rand( n, k ) V
= V 1 ← rand( k, m ) for t = 1 , 2 , . . .
until convergence do λ t ← λ · ν t
U ← U t − 1 + β (U t − 1 − U t − 2 ) ∇ U f = UVV (cid:62) − AV (cid:62)
L ← (cid:107) VV (cid:62) (cid:107)
U ← prox κL − 1 ,λ t L − 1 (cid:0)
U − L − 1 ∇ U f (cid:1)
U t ← U
V ← V t − 1 + β
(V t − 1 − V t − 2 )
∇ V f
= U (cid:62) UV − U (cid:62) A
L ← (cid:107) U (cid:62) U (cid:107)
V ← prox κL − 1 ,λ t L − 1 (cid:0) V − L − 1 ∇ V f (cid:1)
V t ← V end if U or V not Boolean (i.e., if the above was aborted early (cf.
Fig. 2)) let
λ (cid:48) ∈
R be huge
U ← (cid:98) prox 0 .
5 ,λ (cid:48) (U) (cid:101)
V ← (cid:98) prox 0 .
5 ,λ (cid:48)
(V) (cid:101) end return U , V