Efficiently Factorizing Boolean Matrices using Proximal Gradient Descent
Abstract
Addressing the interpretability problem of NMF on Boolean data, Boolean Matrix
Factorization (BMF) uses Boolean algebra to decompose the input into low-rank
Boolean factor matrices.
These matrices are highly interpretable and very useful in practice, but they come at the high computational cost of solving an NP-hard combinatorial optimization problem.
To reduce the computational burden, we propose to relax BMF continuously using a novel elastic-binary regularizer, from which we derive a proximal gradient algorithm.
Through an extensive set of experiments, we demonstrate that our method works well in practice: On synthetic data, we show that it converges quickly, recovers the ground truth precisely, and estimates the simulated rank exactly.
On real-world data, we improve upon the state of the art in recall, loss, and runtime, and a case study from the medical domain confirms that our results are easily interpretable and semantically meaningful.
Introduction
Discovering groups in data and expressing them in terms of common concepts is a central problem in many scientific domains and business applications, including cancer genomics [23], neuroscience [13] and recommender systems [18].
This problem is often addressed using variants of matrix factorization , a family of methods that decompose the target matrix into a set of typically lowrank factor matrices whose product approximates the input well.
Prominent examples of matrix factorization are Singular Value Decomposition (SVD) [10], Principal Component Analysis (PCA)
[10] and Nonnegative Matrix Factorization (NMF) [31, 21, 22].
These methods differ in how they constrain the matrices involved: SVD and PCA require orthogonal factors while NMF constrains the target matrix and the factors to be nonnegative.
SVD, PCA, and NMF achieve interpretable results—unless the data is Boolean, which is ubiquitous in the real world.
In this case, their results are hard to interpret directly, because the input domain differs from the output domain, thus requiring post-processing to extract useful information.
Boolean
Matrix Factorization (BMF) addresses this problem by seeking two low-rank Boolean factor matrices whose Boolean product is close to the Boolean target matrix [24].
The output matrices, now lying in the same domain as the input, are interpretable and useful, but they come at the computational cost of solving an NP-hard combinatorial optimization problem [30, 24, 25].
To make BMF applicable in practice, we need efficient approximation algorithms.
There are many ways to approximate BMF—for example, by exploiting its underlying combinatorial or spatial structure [24, 6, 5], using probabilistic inference [37, 35, 36], or solving the related BiClustering problem [28, 29].
Although these approaches achieve impressive results, they fall short when the input data is large and noisy.
Hence, we take a different approach to overcome BMF’s computational barrier.
Starting from an NMF-like optimization problem, we derive a continuous relaxation of the original BMF formulation that allows intermediate solutions to be real-valued.
Inspired by the elastic-net regularizer [43], we introduce the novel elastic binary ( E LB ) regularizer to 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
regularize towards Boolean factor matrices.
We obtain an efficient-to-compute proximal operator from our E LB regularizer that projects relaxed real-valued factors towards being Boolean, which allows us to leverage fast gradient-based optimization procedures.
In stark contrast to the state of the art [15, 16, 17], which requires heavy post-hoc post-processing to actually achieve Boolean factors, we ensure a Boolean outcome upon convergence by gradually increasing the projection strength using a regularization rate .
We combine our relaxation, efficient proximal operator, and regularization rate into an Elastic Boolean Matrix Factorization algorithm (E LBMF ) that scales to large data, results in accurate reconstructions, and does so without relying on heavy post-processing procedures.
E LB and its rate are, however, not confined to BMF and can regularize, e.g., binary MF or bi-clustering [17].
In summary, our main contributions are as follows:
We introduce the E LB regularizer.
We overcome the computational hardness of BMF leveraging a novel relaxed BMF problem.
We efficiently solve the relaxed BMF problem using an optimization algorithm based on proximal gradient descent.
The remainder of the paper proceeds as follows.
In Sec. 2, we formally introduce the BMF problem and its relaxation, define our E LB regularizer and its proximal point operator, and show how to ensure a Boolean outcome upon convergence.
We discuss related work in Sec. 3, validate our method through an extensive set of experiments in Sec. 4, and conclude with a discussion in Sec. 5.
Theory
Our goal is to factorize a given Boolean target matrix into at least two smaller, low-rank Boolean factor matrices, whose product comes close to the target matrix.
Since the factor matrices are Boolean, this product follows the algebra of a Boolean semi-ring, i.e., it is identical to the standard outer product on a field where addition obeys 1 + 1 = 1 .
We define the product between two Boolean matrices U ∈ { 0 , 1 } n × k and V ∈ { 0 , 1 } k × m on a Boolean semi-ring ( { 0 , 1 } , ∨ , ∧ ) as
[Equation]
where U ∈ { 0 , 1 } n × k , V ∈ { 0 , 1 } k × m , and U ◦ V ∈ { 0 , 1 } n × m
This gives rise to the BMF problem.
Problem 1 (Boolean Matrix Factorization)
For a given target matrix A ∈ { 0 , 1 } n × m , a given matrix rank N ∋ k ≤ min { n, m } , and let A ⊕ B be logical exclusive or, discover the factor matrices
U ∈ { 0 , 1 } n × k and V ∈ { 0 , 1 } k × m that minimize
[Equation]
While beautiful in theory it is NP-complete [25].
Thus, we cannot solve this problem exactly for all but the smallest matrices.
In practice, we hence have to rely on approximations.
Here, we relax the
Boolean constraints of Eq.
(2) to allow non-negative, non-Boolean ‘intermediate’ factor matrices during the optimization, allowing us to use linear algebra rather than Boolean algebra.
In other words, we solve the non-negative matrix factorization (NMF) problem [31]
[Equation]
subject to U ∈ R n × k + and V ∈ R k × m
+ .
In contrast to the original BMF formulation, we can solve this problem efficiently, e.g., via a Gauss-Seidel scheme.
Although efficient, using plain NMF, however, disregards the Boolean structure of our matrices and produces factor matrices from a different domain, which are consequently hard to interpret and potentially very dense.
To benefit from efficient optimization and still arrive at Boolean outputs, we allow real-valued intermediate solutions and regularize them towards becoming Boolean.
To steer our optimization towards Boolean solutions, we penalize non-Boolean solutions using a regularizer.
This idea has been explored in prior work.
There exists the l 1 -inspired P RIMP regularizer [15], which is − κ [ −| 1 − 2 x | + 1] for values inside [0 , 1] and ∞ otherwise, and the l 2inspired bowl-shaped regularizer [42], which is λ ( x 2 − x ) 2 / 2 everywhere on the real line.
Although both have been successfully applied to BMF, both also have undesirable properties: The P RIMP regularizer penalizes well inside the interval [0 , 1] but is non-differentiable on the outside, while the bowl-shaped regularizer is differentiable and penalizes well outside the interval [0 , 1] but is almost flat on the inside.
Hence, both regularizers are problematic if used individually.
Combining them, however, yields a regularizer that penalizes non-Boolean values well across the full real line.
To combine l 1 - and l 2 -regularization, we use the elastic-net regularizer ,
[Equation]
which, however, only penalizes non-zero solutions [43].
To penalize non-Boolean solutions, we combine two elastic-net regularizers into our (almost W-shaped) E LB regularizer ,
[Equation]
where X ∈ { U , V } .
In Fig.
1, we show all three regularizers in the range of [ − 1 , 2] , for λ
= κ = 0 .
We see that only the E LB regularizer penalizes non-Boolean solutions across the full spectrum, and summarize our regularized relaxed BMF as follows.
Problem 2 (Elastic Boolean Matrix Factorization)
For a given target matrix A ∈ { 0 , 1 } n × m and a given matrix rank N ∋ k ≤ min { n, m } , discover the factor matrices U ∈ R n × k +
[Equation]
V ∈ R k × m + that minimize
[Equation] [Equation]
Although this is a relaxed problem, it is still non-convex and therefore, we cannot solve it straightforwardly.
The problem, however, is suitable for the Gauss-Seidel optimization scheme.
That is, we alternatingly fix one factor matrix to optimize the other.
By doing so, we generate a sequence
[Equation]
of simpler-to-solve sub-problems, until convergence.
Now, each sub-problem is again a sum of two f ( X ) + R ( X ) functions, where f is the loss ∥ · ∥ 2 F , and R ( X ) is the regularizer.
This allows us to follow a proximal gradient approach, i.e., we use Proximal Alternating Linear Minimization (PALM)
In a nutshell, we minimize a sub-problem by following the gradient ∇ f of f , to then use the proximal operator for R to nudge its outcome towards a Boolean solution.
That is, for the gradients ∇ U f = UVV ⊤ − AV ⊤ and ∇ V f =
U ⊤ UV − U ⊤ A , we compute the step
[Equation]
where η represents the step size, which we compute in terms of Lipschitz constant, rather than relying on a costly line-search [7].
To further improve the convergence properties, we make use of an inertial term that linearly combines X t with X t − 1 before applying Eq.
(8) (see [33] for a detailed description).
We now derive the proximal operator for the E LB regularizer, before discussing how we ensure that the factor matrices are Boolean, and summarizing our approach as an algorithm.
2.1 Proximal Mapping
To solve the sub-problem
[Equation]
from Eq.
(7) for X ∈ { U , V } , we need a proximal operator [32] that projects values towards a regularized point.
Formally in Appendix A, here in a nutshell: this operator is the solution to prox R ( X ) =
arg min Y
[Equation] [Equation]
−    0 .0 .0 .0 .
x
R ( x )
Regularizer
Bowl
P RIMP
E LB
Algorithm 1: E LBMF
Input: Matrix A ∈ { 0 , 1 } n × m , rank k ∈ N
Output: Factors U ∈ { 0 , 1 } n × k , V ∈ { 0 , 1 } k × m initialize U and V uniformly at random for t = 1 , 2 , . . .
until convergence do
U ← arg reduce U ℓ κλ t (A , U , V) V ← arg reduce
V ℓ κλ t (A , U , V) endreturn U , V
As Eq.
(10) is coordinate-wise solvable, we can reduce this equation to a scalar proximal operator
[Equation]
Although this scalar function may seem simple, due to R , it is non-convex, which—in general—rules out the existence of unique minima.
We can, however, exploit the W-shape of R .
That is, from the perspective of a fixed X ij , the regularizer is locally convex after a case distinction.
Starting with the case of X ij = x, x ≤ 1 / 2 , we can simplify Eq.
(11) to 12 ( x − y )
+ λ ′ / 2 y
+ κ | y | , letting λ ′ = 2 λ .
Setting its derivative to 0 , we get to y =
x
− κ sign( y ) .
Asserting that a least-squares solution will always be of the same sign, we substitute the sign of x with sign( y ) .
Repeating these steps analogously for x > 1 / 2 , we obtain our proximal operator:
Definition 1 (Proximal Operator)
Given regularization coefficients κ and λ , our proximal operator for matrix X is prox R ( X ) =
[prox κλ ( X ij )] ij , where prox κλ is the scalar proximal operator
[Equation]
Although not strictly necessary, to improve the empirical convergence rate, we would like to constrain our factor matrices to be non-negative.
As our proximal operator does not account for this, we impose non-negativity by using the alternating projection procedure to combine the non-negativity proximal operator [32] with Eq.
(12) into max { 0 , prox κλ ( x ) } .
2.2 Ensuring Boolean Factors
Our proximal operator only nudges the factor matrices towards becoming Boolean.
We, however, want to ensure that our results are Boolean.
To this end, the state-of-the-art method P RIMP relies heavily on post-processing, performing a very expensive joint two-dimensional grid search to guess the ‘best’ pair of rounding thresholds, which are then used to produce Boolean matrices.
Although this tends to work in practice, it is an inefficient post-hoc procedure—and thus, it would be highly desirable to have Boolean factors already upon convergence.
To achieve this without rounding or clamping, we revisit our regularizer, which binarizes more strongly if we regularize more aggressively.
Consequently, if we regularize too aggressively, we converge to a suboptimal solution, and if we regularize too mildly, we do not binarize our solutions.
To prevent subpar solutions and still binarize our output, we start with a weak regularization and gradually increase its strength.
Considering Eq.
(12), we see that a higher regularization strength increases the distance over which our proximal operator projects.
Thus, if we set the l 1 -distance controlling κ too high, we will immediately leap to a Boolean factor matrix, which will terminate the algorithm and yield a suboptimal solution.
Regulating the l 2 -distance controlling λ is a less delicate matter.
Hence, we gradually increase λ to prevent a subpar solution and achieve a Boolean outcome, using a regularization rate
[Equation]
that gradually increases the proximal distance at a user-defined rate.
In case E LBMF has stopped without convergence, we bridge the remaining integrality gap by projecting the outcome onto their closed Boolean counterpart, using our proximal operator (see Fig. 2).
We summarize the considerations laid out above as E LBMF in Alg. 1.
The computational complexity of
E LBMF is bounded by the complexity of computing the gradient, which is identical to the complexity of matrix multiplication.
Therefore, for all practical purposes, E LBMF is sub-cubic O (cid:0) n 2 . 807 (cid:1) using
Strassen’s algorithm.
Related Work
Matrix factorization is a well-established family of methods, whose members, such as SVD, PCA, or
NMF, are used everywhere in machine learning.
Almost all matrix factorization methods operate on real-valued matrices, however, while BMF operates under Boolean algebra.
Boolean Matrix
Factorization originated in the combinatorics community [27] and was later introduced to the data mining community [24], where many cover-based BMF algorithms were developed [6, 5, 24, 26].
In recent years, BMF has gained traction in the machine learning community, which tends to tackle the problem differently.
Here, relaxation-based approaches that optimize for a relaxed but regularized BMF [15, 14, 42] are related to our method, but they differ especially in their regularization.
Hess et al.
[15] introduce a regularizer that is only partially differentiable and rely heavily on post-processing to force a Boolean solution, and Zhang et al.
[42] regularize only weakly between 0 and 1 .
In contrast, our regularizer penalizes well across the full spectrum and yields a Boolean outcome upon convergence.
Building on a thresholding-based BMF formulation,
Araujo et al.
[3] also considers relaxations to benefit from gradient-based optimization.
Other recent approaches build on probabilistic inference .
Rukat et al.
[37, 35, 36], for example, combine
Bayesian Modeling and sampling into their logical factor machine.
A similar direction is taken by
Ravanbakhsh et al.
[34], who use graphical models and message passing, and Liang et al [23], who combine MAP-inference and sampling.
A different, geometry-based approach lies in locating dense submatrices by ordering the data to exploit the consecutive-ones property [40, 38].
Since BMF is essentially solving a bipartite graph partitioning problem, it is also closely related to Bi-Clustering and Co-Clustering [28, 17].
Neumann and Miettinen [29] use this relationship to efficiently solve
BMF by means of a streaming algorithm.
Although there are many different approaches to BMF, its biggest challenge to date remains scalability [25].
Experiments
We implement E LBMF in the Julia language and run experiments on 16 Cores of an AMD EPYC 7702 and a single NVIDIA A100 GPU, reporting wall clock time.
We provide the source code, datasets, synthetic dataset generator, and other information needed for reproducibility.
1 We compare
E LBMF against six methods: four dedicated BMF methods (A SSO [24], G RECOND [6], O R M [35], and P RIMP [15]), one streaming Bi-Clustering algorithm S OFA [29], one elastic-net-regularized NMF method leveraging proximal gradient descent (NMF [31, 21, 22]), and one interpretable Boolean autoencoder (B INAPS [9]).
Since NMF outputs non-negative factor matrices, rather Boolean matrices, we cannot compare against NMF directly, so we clamp and round its solutions to the nearest Boolean outcome.
To fairly compare against B INAPS , we task it with autoencoding the target matrix as a reconstruction, given the matrix ranks from our experiments as the number of latent dimensions.
We perform three sets of experiments.
First, we ascertain that E LBMF works reliably on synthetic data.
Second, we verify that it generally performs well on real-world data.
And third, we illustrate that its outputs are semantically meaningful through an exploratory analysis of a biomedical dataset.
4.1 Performance of E LBMF on Synthetic Data
In the following experiments, we ask three questions: ( 1 ) How does E LBMF converge?; ( 2 ) How well does E LBMF recover the information in the target matrix?
; ( 3 ) How consistently does E LBMF
Iteration t
B oo l ea n G a p
Rate
Constant   0.
Iteration t
L o ss
G a p hasconverged keepsonchanging    5 ,10 ,15 ,20 ,
Iteration t
F li pp e d B it s reconstructs low- or high-density target matrices?; and ( 4 ) Does E LBMF estimate the underlying
Boolean matrix rank correctly?
To answer these questions, we generate synthetic data with known ground truth as follows.
Starting with an all-zeros matrix, we randomly create rectangular, nonoverlapping, consecutive areas of ones called tiles , each spanning a randomly chosen number of consecutive rows and columns, thus inducing matrices with varying densities.
We then add noise by setting each cell to 1 , uniformly at random, with varying noise probabilities.
How does E LBMF converge?
To study how our method converges to a Boolean solution, we quantify relevant properties of the sequence of intermediate solutions (cf.
Eq.
First, to understand how quickly and stably E LBMF converges to a Boolean solution, we quantify the Boolean gap , (cid:88)
X ∈{ U t ,V t } | X | − 1 (cid:88) x ∈ X min {| x | , | x − 1 |} .
Second, to understand when we can safely round intermediate almost-Boolean solutions without losing information, we calculate for the reconstruction B from rounded intermediate factors, the cumulative Hamming process as the sum of fraction of bits that flip from iteration t to iteration t +
[Equation]
1 , and the loss gap as the difference between the relaxed loss and the loss from the rounded B .
As shown in Fig.
2, we achieve an almost-Boolean solution without any rounding after aroundepochs, continuing until we reach a Boolean outcome.
This is also the point at which the rounded intermediate solution and its relaxation are almost identical, as illustrated by the loss gap.
Considering the Hamming process on the right, we observe that E LBMF goes through an erratic bit-flipping-phase in the beginning, followed by only minor changes in each iteration until iteration t = 100 .
Afterwards,
E LBMF has settled on a solution—under our regularization rate.
When using constant regularization instead, we continue to observe bit flips until the end of the experiment.
Under constant regularization, the Boolean gap hardly decreases over time.
Far from Boolean, the constant regularization thus also never closes the loss gap, which is unsurprising, given that its factors are less regularized.
In other words, our regularization works well, and it allows us to safely binarize almost-converged factors that are ϵ -far from being Boolean by means of, e.g., our proximal operator.
How well does E LBMF recover the information in the target matrix?
Having ensured that our method converges stably and quickly, we would like to assess whether it also converges to a high-quality factorization.
To this end, we generate synthetic 40 × 30 matrices containing 5 random tiles each spanning 5 to 10 rows and columns, under additive noise levels between 0% (no noise) and 50% .
We then compute the fraction of ones in target
A that is covered by the reconstruction
B = U ◦ V , i.e., the recall (higher is better)
[Equation]
To ensure that we fit the signal in the data, we additionally report the recall regarding the generating, noise-free ground-truth tiles A ∗ , denoted as recall ∗ .
Finally, to rate the overall quality of the
[Figure] [Figure]
[Figure] [Figure]
reconstruction including zeros , we compute the Hamming similarity (higher is better) between the target matrix and its reconstruction
[Equation]
We run each method on our synthetic datasets, targeting a matrix rank of 5 .
To account for random fluctuations, we average over 10 randomly drawn sets of 5 ground-truth tiles per 10% increment in noise probability.
In Fig.
3, we show similarity, recall, and recall ∗ .
We observe that in the noiseless case ( 0% ), all methods except B INAPS recover the 5 ground-truth tiles with high accuracy, but only A SSO and E LBMF do so with perfect recalls.
Starting with as little as 10% noise, both recalls of A SSO , G RECOND , S OFA , NMF , and B INAPS deteriorate quickly, while the similarity and both recalls of P RIMP and E LBMF remain high.
In fact, E LBMF and P RIMP perform similarly across the board—which is highly encouraging, as unlike P RIMP , E LBMF does not require post-processing.
For A SSO and G RECOND , recall and similarity drop considerably, but they exhibit a slightly higher recall ∗ .
This means that these methods are robust against noise, but they fail to recover the remaining information.
Starting low, O R M’s recalls increase jointly with the noise level, suggesting that clean data is problematic for O R M. Reporting the standard deviations as the shaded region, we see little variance across all similarities—except for A SSO and NMF in the highest noise regime.
The deviation of both recalls are, however, inconsistent for most methods, except for B INAPS , P RIMP , and E LBMF .
Overall, the performance characteristics of E LBMF are among the most reliable.
How robust is E LBMF regarding varying matrix densities?
Now, to understand whether E LBMF performs consistently well on low or high density matrices.
To this end, we generate synthetic matrices like before, this time using fixed noise of 0 . 2 and varying the width and height of the ground-truth squared tiles from 3 2 to 12 2 , resulting in densities between 0 . 0375 to 0 .
6 , before noise.
In Fig.
4, we show the similarity, recall, and loss of B INAPS , A SSO , G RECOND , O R M, S OFA ,
NMF , P RIMP , and E LBMF .
We can see that the increasing density affects the performance of all
Rank k
A I C    7 . 6 · 107 . 8 · 108 · 108 . 2 · 108 . 4 · 108 . 6 ·
Rank k
M D L    9 .
Rank k
B a i &
N g methods
Does E LBMF estimate the underlying Boolean matrix rank correctly?
When a target rank is known, we can immediately apply E LBMF to factorize the data.
In the real world, however, the target rank might be unknown.
In this case, we need to estimate an appropriate choice from the data, and we use synthetic data to ensure that E LBMF does so correctly.
Since a higher matrix rank usually also means a better fit, selecting the best rank according to recall, loss, or similarity leads to overfitting—unless we properly account for the growth in model complexity.
There are many model selection criteria that penalize complex models, such as AIC [1], Bai & Ng’s criteria [4, 2], Nuclear-norm regularizing [20, 12], the information-theoretic Minimal Description
Length principle (MDL) [11], or (Decomposed)
Normalized Maximum Likelihood [19, 41].
Following common practice and motivated by its practical performance in preliminary experiments, we choose MDL.
That is, we select the minimizer of sum the log binomial l ( X ) =
log (cid:0) | X | ∥
X ∥ 1 (cid:1) of the error matrix and the rows and columns of our factorization (assumed to be i.i.d.)
[Equation]
To validate whether E LBMF recovers the right rank, we synthetically generate a 400 × 300 matrix of ground-truth rank 10 with 10% additive noise.
In Fig.
5, we show AIC, MDL, and Bai & Ng’s first criterion for each rank up to 30 , finding that our method precisely discovers the right rank.
4.2 Performance of E LBMF on Real-World Data
Having ascertained that E LBMF works well on synthetic data, we turn to its performance in the real world.
Here, we use nine publicly available datasets 2 from different domains.
To cover the biomedical domain , we extract the network containing empirical evidence of protein-protein interactions in Homo sapiens from the S TRING database.
From the G RAND repository, we take the gene regulatory networks sampled from Glioblastoma (GBM) and Lower Grade Glioma (LGG) brain cancer tissues, as well as from non-cancerous Cerebellum tissue.
The TCGA dataset contains binarized gene expressions from cancer patients, and we further obtain the single nucleotide polymorphism (SNP) mutation data from the 1 k Genomes project, following processing steps from the authors of B INAPS [9].
In the entertainment domain , we use the user-movie datasets Movielens and Netflix , binarizing the original 5 -star-scale ratings by setting only reviews with more than 3 .
5 stars to 1 .
Finally, as data from the innovation domain , we derive a directed citation network between patent groups from patent citation and classification data provided by PatentsView .
For each dataset with a given number of groups, such as cancer types or movie genres, we set the matrix rank k to 33 (TCGA), 28 (Genomes),(Patents), 20 (Movielens), and 20 (Netflix).
When the number of subgroups is unknown, we estimate the rank that minimizes MDL using E LBMF , resulting in 100 (GBM), 32 (LGG), 100 (String), and 450 (Cerebellum).
[Figure]
StringPatents
MovielensNetﬂix
[Figure]
RECOND
[Figure]
As we can achieve a high similarity with an all-zeros reconstruction (in the case of sparse data), or a perfect recall with an all-ones reconstruction, we also report the relative loss (lower is better),
[Equation]
between the target matrices and their reconstructions.
We show relative loss, similarity, recall, and runtime of A SSO , G RECOND , S OFA , O R M, B INAPS ,
NMF , P RIMP , and E LBMF , applied to all real-world datasets, targeting a given matrix rank, in Fig.
The cover-based G RECOND and A SSO show comparable loss, similarity, and recall.
Both perform better on smaller matrices ( LGG , GBM , or Cerebellum ) and struggle with complex matrices (e.g.,
TCGA , Genomes ).
On the complex matrices (e.g., on TCGA , String , or Movielens ), Although always outperformed by E LBMF , we see that the rounded NMF reconstructions are surprisingly good (e.g., on TCGA , String , or Movielens ), occasionally surpassing dedicated BMF methods A SSO , G RECOND ,
O R M, and S OFA .
Across the board, G RECOND , A SSO , O R M, S OFA , B INAPS , and NMF almost always result in considerably higher loss than E LBMF .
Compared to the closed competitor P RIMP , our method E LBMF always results in lower reconstruction loss.
We observe the largest gap between the two on the Cerebellum dataset, where P RIMP ’s grid search procedure fails to find suitable thresholds.
This is an impressive result, because, unlike P RIMP E LBMF does not necessitate heavy post-processing.
In Fig.
6b we see that all methods, except B INAPS , result a high similarity, thus are sparsity inducing.
As B INAPS overfits and densely reconstructs sparse inputs, it surpasses sparsity-inducing methods in recall.
Considering non-overfitting methods, however, E LBMF is among the best performing in terms or recall, often outperforming P RIMP , while under significantly stronger regularization.
When P RIMP has a higher recall (e.g., Genomes ), it often does so at a higher loss than E LBMF .
We see in Fig.
6d (log scale) that—except for simple cases—A SSO , G RECOND , O R M, S OFA , and P RIMP are slower than E LBMF .
Although NMF is less constrained than E LBMF , they are on-par when it comes to runtime.
Degraded by post-processing, our closed competitor P RIMP is always much slower than
E LBMF and struggles with Netflix .
Only B INAPS and E LBMF finished Netflix , however, only E LBMF did so at a reasonable loss, considering the given target rank.
4.3 Exploratory Analysis of Gene Expression Data with E LBMF
Knowing that E LBMF performs well quantitatively, we ask whether its outputs are also interpretable.
To this end, we take a closer look at the TCGA data, which contains the expression levels of 20genes from 10 459 patients, who are labeled with 33 cancer types.
Since we are interested in retaining high gene expression levels only, we set expression levels to one if their z -scores fall into the top 5% quantile, and to zero otherwise [23].
We run E LBMF on this dataset, targeting a rank of 33 .
To learn whether our method groups patients meaningfully, we visualize the target matrix and its reconstruction.
As the target matrix is high-dimensional, we embed both the target matrix and the reconstruction into a two-dimensional space using t-SNE [39] as illustrated in Fig.
7, where each color corresponds to one cancer type.
In Fig.
7a, we show that when embedding the target matrix directly, the cancer types are highly overlapping and hard to distinguish without the color coding. In contrast, when embedding our reconstruction, depicted in Fig. 7b, we see a clean segmentation into clusters that predominantly contain a single cancer type.
− 5 −  t-SNEt - S N
E(a) TCGA − 5 −  t-SNEt - S N
E(b) Reconstructed TCGA
Index of Estimated Component
I nd e x o f C a n ce r T yp e 0 .0 .0 .0 .0 .1 .
NMI
[Figure]
While the cancer types are hard to differentiate in the embedding of the original dataset (a), they are separated into easily distinguishable clusters in the embedding of our reconstruction (b).
In (c), we show the normalized mutual information between estimated groups and cancer types, and observe that there is an almost 1 -to- 1 correspondence between our estimated groups and the cancer types.
To better understand these results, we quantify the association between our 33 estimated components and the ground-truth cancer types by computing the normalized mutual information matrix.
This matrix is noticeably sparse (Fig. 7c), leading to a clean segmentation.
Upon closer inspection with
E NRICHR [8], the associations we discover turn out to be biologically meaningful.
For example, we find that E LBMF associates a set of 356 genes to patients with thyroid carcinoma .
This component is associated with thyroid hormone generation and thyroid gland development , and statistically significantly so—even under a strict False Discovery Control, with p -values as low as 2 .
574 × 10 −and 6 .
530 × 10 − 6 .
Conclusion
We introduced E LBMF to efficiently factorize Boolean matrices using an elegant and simple algorithm that, unlike its closest competitors, does not rely on heavy post-processing.
E LBMF considers a novel relaxed BMF problem, which allows intermediate solutions to be real-valued.
It solves this problem by leveraging an efficiently computable proximal operator, derived from the innovative E LB regularizer, and using a regularization rate to obtain Boolean factors upon convergence.
Experimentally, we have shown that E LBMF works well in practice.
It operates reliably on synthetic data, outperforms the discrete state of the art, is at least as good as the best relaxations on real data, and yields interpretable results even in difficult domains—without relying on post-processing.
Limitations Although E LBMF works overall, it has two bottlenecks.
First, by randomly initializing factors, we start with highly dense matrices, thus prohibiting efficient sparse matrix operations.
This is not ideal for sparse datasets that are too large to fit into memory, and future research on sparse initialization of Boolean factors will benefit not only E LBMF but also many other methods.
Second, the larger the datasets, the higher the cost of computing gradients, and future work might adapt stochastic gradient methods for E LBMF to mitigate this problem.
Broader Impact
E LBMF is a method for factorizing Boolean matrices, which permits interpretable, rather than black box data analysis.
As such, it can help to make explicit any biases that may be present in the data.
Just like any other method for interpretable data analysis, the insights they yield us can be used for good (e.g., insight into cancer-causing mutations) or for bad, and the ultimate quality of the outcome also depends on the behavior of the user.
In our experiments we focus on good applications, and only consider anonymized open data.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Sec.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec. 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Sec. 2 and appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Sec. 4, appendix, and under doi.org/10.5281/zenodo.7187021.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sec. 4 and appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Sec. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Sec. 4.
(b) Did you mention the license of the assets?
[Yes] See Sec. 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Sec. 4, appendix, and under doi.org/10.5281/zenodo.7187021.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
us- ing/curating?
[Yes] See Sec. 4.
All datasets are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[Yes] See Sec. 4.
All datasets have been anonymized prior to their publication.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix
We include the following supplementary materials in this Appendix.
In Appendix A, we formally derive our proximal operator from our E LB regularizer, resulting in the proximal operator Eq.
12 as well as an alternative proximal operator Eq.
17, not used in our experiments.
In Appendix B, we provide a long version of Algorithm 1.
In Appendix C, we provide details regarding reproducibility such as hyper-parameters, data-preprocessing, implementation details and datasets.
In Appendix D, we provide additional experiments on synthetic data under varying levels of additive noise, this time allowing for overlapping tiles.
In Appendix E, we provide additional experiments on synthetic data under a low additive noise and varying densities.
A Derivation of the Proximal Operator
In this section, we derive our proximal operator (given in Eq.
(12), repeated below)
[Equation]
from the E LB (Eq.
(4), repeated below)
[Equation]
Starting with the definition [32] of the general proximal operator
[Equation]
we observe that this proximal operator is coordinate-wise solvable.
This allows us to derive a scalar proximal operator, which we then apply to each value in the matrix independently.
Substituting the regularizer R with its scalar version, we obtain a scalar proximal operator (Eq.
[Equation]
which is non-convex, has no unique minima, and, therefore, is not straightforwardly solvable.
We can, however, separate this function into two locally convex V-shaped functions, which are straightforwardly solvable.
By asserting that its least-squares solution is either less than 1 / 2 (if x ≤ 1 / 2 ), or greater than 1 / 2 (if x > 1 / 2 ), we can address each case independently, and merge the outcome into a single piecewise proximal operator.
Case I
In the first case, we address the operator for x ≤ 1 / 2 .
For this, we start by simplifying our scalar proximal operator Eq.
(11), by substituting r ( y ) with its definition, and get
[Equation]
for λ ′ / 2 = λ .
Then, we take its partial derivative for y
[Equation]
which we set to zero, obtaining
[Equation]
By asserting that we can obtain a better least-squares solution if y has the same sign as x , we can substitute the sign of x with sign( y ) , and get
[Equation]
which concludes the first case.
Case II Analogously, we now repeat the steps from above for x > 1 / 2 .
Again, we start by simplifying
Eq.
(11), substituting r ( y − 1)
[Equation]
for λ ′ / 2 = λ .
By taking its partial derivative for y
[Equation]
and setting it to zero, we obtain
[Equation]
Then, asserting that the least-squares solution does not get worse by using the same sign for y − 1 and x − 1 , we can substitute the sign of x − 1 with sign( y − 1) , and get
[Equation]
which concludes the x > 1 / 2 case.
Combining Case I & Case II
Combining the cases above yields our piecewise proximal operator (see
Eq.
prox
[Equation] [Equation]
≡
[Equation]
Alternative Proximal Operator Considering Eq.
(11), we notice that the term y − x is squared, which means that there are multiple solutions to this equation.
We derive the alternative operator analogously to the steps above, however, by switching the positions of y and x in f .
[Equation]
alt.
κλ
[Equation]
Since this operator is denominated by λ ′ − 1 , we need to ensure that λ ′ ̸ = 1 .
Because our original proximal operator in Eq.
(12) is denominated by λ ′ + 1 , and since λ ′ is usually positive, we are not required to take extra precautions.
Since this is more convenient, we select Eq.
(12) as our proximal operator, rather than taking extra precautions when using Eq.
B Extended pseudocode for E LBMF
Extending Alg.
1, we provide the long version of E LBMF as pseudo code in Alg. 2.
Algorithm 2:
Input:
Target Matrix
A ∈ { 0 , 1 } n × m
Rank k ∈ N , l 1 Regularizer Coefficients κ ∈ R , l 2 Regularizer Coefficients λ ∈ R ,
Regularization Rate ν t ∈ N → R , optional Inertial Parameter β ∈ R +
Output: Factors U ∈ { 0 , 1 } n × k , V ∈ { 0 , 1 } k × m
U
= U 1 ← rand( n, k ) V
= V 1 ← rand( k, m ) for t = 1 , 2 , . . .
until convergence do λ t ← λ · ν t
U ← U t − 1 + β (U t − 1 − U t − 2 ) ∇ U f = UVV ⊤ − AV ⊤
L ← ∥
VV ⊤ ∥
U ← prox κL − 1 ,λ t L − 1 (cid:0)
U − L − 1 ∇ U f (cid:1)
U t ← U
V ← V t − 1 + β
(V t − 1 − V t − 2 )
∇ V f =
U ⊤ UV − U ⊤ A
L ← ∥
U ⊤ U ∥
V ← prox κL − 1 ,λ t L − 1 (cid:0) V − L − 1 ∇ V f (cid:1)
V t ← V end if U or V not Boolean (i.e., if the above was aborted early (cf.
Fig. 2)) let λ ′ ∈ R be huge
U ← ⌊ prox 0 .
5 ,λ ′ (U)
V ← ⌊ prox 0 .
5 ,λ ′ (V) ⌉ end return U , V
C Reproducibility
[Table]
Netflix
[Table] [Table]
[Table]
Patents
[Table]
In this addendum, we explain ( 1 ) how to obtain the datasets, ( 2 ) how we binarized them, ( 3 ) how to obtain the source code, and ( 4 ) how we parameterized the algorithms, starting with the dataset descriptions.
Broadly speaking, our datasets cover three domains: the biomedical domain , the entertainment domain , and the innovation domain .
To cover the biomedical domain , we extract the network containing empirical evidence of protein-protein interactions in homo sapiens from the S TRING database.
For this, we remove all interactions in the protein-protein network, for which the empirical-evidence sub-score of the S TRING database is zero, retaining only the protein-protein interactions discovered experimentally.
From the G RAND repository, we take the gene regulatory networks sampled from Glioblastoma (GBM) and Lower Grade Glioma (LGG) brain cancer tissues, as well as from non-cancerous Cerebellum tissue.
These networks all stem from a method, P ANDA , which extracts gene-regulatory networks from data.
The weights are between − 20 and 20 , where − 20 corresponds to a low interaction likelihood, and 20 corresponds to a high interaction likelihood.
To binarize, we set everything to zero except the weights whose z -scores are in the upper 5% quantile, retaining the information about likely interactions.
The TCGA dataset contains gene expressions from cancer patients.
To binarize these logarithmic expression rates ( log 10 ( x + 1) ), we again only set those weights to one whose z -scores lie in the upper 5% quantile [23], retaining high gene expressions.
We further obtain the single nucleotide polymorphism (SNP) mutation data from the 1 k Genomes project, and follow the data retrieval steps from the authors of B INAPS [9], which immediately produces a binary dataset.
In the entertainment domain , we use the user-movie datasets Movielens and Netflix .
Since we are only interested in recommending good movies, we binarize the original 5 -star-scaled ratings, by setting reviews with more than 3 .
5 stars to one, and everything else to zero.
Finally, as data from the innovation domain , we derive a directed citation network between patent groups from patent citation and classification data provided by PatentsView .
We binarize this weighted network simply by setting every non-zero weight to one, retaining all edges in the network.
For each of these publicly 3 available dataset, we give its dimensionality, density, and the matrix rank used in our experiments in Table 1.
In our experiments in Sec. 4, we compare E LBMF against six methods: four dedicated BMF methods (A SSO [24], G RECOND [6], O R M [35], and P RIMP [15]), one streaming Bi-Clustering algorithm S OFA [29], one elasticnet-regularized NMF method leveraging proximal gradient descent (NMF [31, 21, 22]), and one interpretable Boolean autoencoder (B INAPS [9]).
The code for A SSO , G RECOND , P RIMP , S OFA , O R M, and B INAPS was written by their respective authors and is publicly available.
We implement NMF and E LBMF in the Julia programming language and provide their source code for reproducibility.
On TCGA , Genomes , Movielens ,
Netflix , and Patents , we set the l 2 -regularizer λ = 0 .
001 , the l 1 -regularizer κ = 0 .
005 , and the regularization rate to ν t = 1 .
0033 t .
On GBM , LGG , and Cerebellum , we set the l 2 -regularizer λ = 0 .
001 , the l 1 -regularizer κ = 0 .
001 , and the regularization rate to ν t = 1 . 0015 t .
We run NMF , E LBMF , P RIMP for at most 1 500 epochs on each dataset.
In the case that E LBMF reaches its maximum number of iterations without convergence, we bridge the remaining integrality gap simply by applying our proximal operator (see Fig. 2).
To obtain a good reconstruction for P RIMP , we use a grid-width of 0 . 01 .
To obtain a binary solution from NMF , we first clamp and then round its factor matrices upon convergence.
We set A SSO ’s threshold, gain for covering, and penalty for over-covering each to 1 .
To achieve a better performance with A SSO , we parallelize A SSO on 16 CPU cores.
Further, because A SSO ’s runtime scales with the number of columns, we reconstruct the transposed target whenever it has more columns than rows (see Table 1).
For example, transposing GBM , LGG , Cerebellum , and Genomes is particularly beneficial for A SSO , as these datasets have orders-of-magnitude more columns than rows.
D Additional Results on the Performance of E LBMF on Synthetic Data
In this section, we provide additional results on our synthetic experiments in Sec.
4.1 and Fig. In our synthetic experiments in Sec. 4.1, we simulate data by generating non-overlapping tiles using rejection sampling.
That is, before we place the next randomly drawn tile into our matrix, we check for overlap with past placements.
If we detect an overlap, we reject, redraw, and repeat, until we placed the desired number of tiles into our matrix.
In the following we allow overlapping tiles, for which we simply omit the rejection step laid out above. To depict results on harder to separate data, we generate synthetic matrices as described in Sec. 4, however, this time, allowing tiles to overlap arbitrarily.
In Fig.
8 we show similarity, recall, and recall ∗ for the overlapping case , observing a similar behavior to Fig.
3 across the board.
Again noticeable is the surprisingly good performance of rounded NMF reconstructions, outperforming A SSO , G RECOND , S OFA , and B INAPS by a large margin.
Overall, P RIMP and E LBMF are clearly outperforming A SSO , G RECOND , O R M, S OFA , B INAPS , and NMF across varying noise levels in similarity, recall, and recall ∗ .
Fig.
3 and Fig.
8 show that E LBMF , which does not use any post-processing, achieves best-in-class results for overlapping and non-overlapping tiles, on par with the strongest competitor, which relies heavily on post-processing.
[Table]
DE / PRJ / BINAPS
GITHUB
COM /T
AMMO R L OGICAL F ACTORISATION M ACHINES
[Figure]
[Figure]
[Figure] [Figure]
as line and standard deviation as shade of similarity , recall , and relative loss w.r.t.
the target matrix, for B INAPS , A SSO , G RECOND , O R M, S OFA , NMF , P RIMP , and E LBMF .
E Additional Results on the Performance of E LBMF on Synthetic Data with Varying Densities
Continuing our experiments from Sec. 4.1 and Fig.
4, we ask whether our observations carry over to a low-noise scenario, in which A SSO and G RECOND performance improves significantly (see Fig. 3). For this, we study the effects of varying densities under low noise level of only 5% .
As tiny tiles are hard to distinguish from noise, we see an overall improvement with increasing density, regardless of the method. With less noise, A SSO , G RECOND , and NMF improve significantly in comparison to their performance under more noise Fig.
They, however, are still outperformed by P RIMP and E LBMF in recall, and loss.
The similarities of A SSO , G RECOND , NMF , P RIMP , and E LBMF are close to 1 , whereas S OFA , B INAPS , and O R M exhibit lower similarity with increasing density. From Fig.
4 and Fig.
9, we see that E LBMF performs consistently well across varying densities, regardless of the noise level.