Lazy and Fast Greedy MAP Inference for Determinantal Point Process
Abstract
The maximum a posteriori (MAP) inference for determinantal point processes (DPPs) is crucial for selecting diverse items in many machine learning applications.
Although DPP MAP inference is NP-hard, the greedy algorithm often finds highquality solutions, and many researchers have studied its efficient implementation.
One classical and practical method is the lazy greedy algorithm, which is applicable to general submodular function maximization, while a recent fast greedy algorithm based on the Cholesky factorization is more efficient for DPP MAP inference.
This paper presents how to combine the ideas of “lazy” and “fast”, which have been considered incompatible in the literature.
Our lazy and fast greedy algorithm achieves almost the same time complexity as the current best one and runs faster in practice.
The idea of “lazy + fast” is extendable to other greedy-type algorithms.
We also give a fast version of the double greedy algorithm for unconstrained DPP
MAP inference.
Experiments validate the effectiveness of our acceleration ideas.
Introduction
Determinantal point processes (DPPs) offer a popular diversification model in machine learning.
Macchi [34] first used DPPs to represent repulsion in quantum physics, and later, DPPs have been used in various scenarios such as recommendation systems [13], document summarization [21, 31], and diverse molecule selection [39].
An important problem in the DPP applications is the maximum a posteriori (MAP) inference, which asks to find an item subset with the highest probability.
Intuitively, if each item is associated with a vector whose length and direction represent its importance and feature, respectively, then the aim of DPP MAP inference is to select items whose vectors form the largest volume parallelotope, thus selecting important and diverse items.
In DPPs, exact MAP inference is NP-hard [27].
Fortunately, however, the standard greedy algorithm (G REEDY ) for submodular function maximization [41] enjoys a ( 1 − 1 / e )-approximation guarantee in terms of the log-determinant function value under the monotonicity assumption, and it often finds high-quality solutions in practice.
A naive implementation of G REEDY , however, incurs too much computation cost for large instances since evaluating the determinant of a k × k matrix takes O( k ω ) time, where ω ∈ [2 , 3] is the matrix-multiplication exponent (usually ω = 3 ).
To overcome this issue, researchers have studied techniques for implementing efficient greedy algorithms.
One classical and powerful method is the so-called lazy greedy algorithm
(L AZY G REEDY )
[37], which avoids the redundant computation of function values by making good use of the submodularity.
L AZY G REEDY is applicable to submodular function maximization, and it empirically runs much faster than naive
G REEDY , although the worst-case running time is not improved.
Chen et al.
[13] proposed another notable Cholesky-factorization-based method, called the fast greedy algorithm (F AST G REEDY ).
Their algorithm is specialized for DPP MAP inference and provides the fastest O( knd ) -time implementation of G REEDY for selecting k out of n items represented by d -dimensional vectors.
Chen et al.
[13] also experimentally showed that F AST G REEDY can run faster than L AZY G REEDY .
Since the study of [13], although a pre-processing method for customized DPP MAP inference [23] and fast parallel algorithms for submodular function maximization [3, 4, 18, 8, 17, 14, 29] have been studied, no progress has been made that directly accelerates G REEDY for DPP MAP inference.
Since real-world dataset sizes have been growing, a further speed-up of G REEDY is eagerly awaited.
Our main contribution is to combine the two ideas, “lazy” and “fast”, to develop an even faster implementation of G REEDY for DPP MAP inference, which we call L AZY F AST G REEDY .
In the literature, the two ideas have been thought to be incompatible; in fact, experiments in [13] compared
F AST G REEDY with L AZY G REEDY without considering their combination.
The core idea of “lazy + fast” is widely applicable to other greedy-type algorithms.
Below is a summary of our results.
We present L AZY F AST G REEDY for cardinality-constrained DPP MAP inference.
It takes O( kn ( d +log n )) time even in the worst case and is faster than F AST G REEDY in practice.
We also extend the idea of “lazy + fast” to other greedy-type algorithms: R ANDOM G REEDY [10], S TOCHASTIC G REEDY [38, 43], and I NTERLACE G REEDY [28].
We present a “fast” version of D OUBLE G REEDY [11] for unconstrained DPP MAP inference by extending the idea of [13] with Jacobi’s complementary minor formula.
Experiments on synthetic and real-world datasets validate the empirical effectiveness of our acceleration techniques for the greedy-type algorithms.
In particular, our L AZY F ASTG REEDY runs up to about 17 times faster than F AST G REEDY in real-world settings.
Accelerating the greedy variants [10, 38, 28, 43], which enjoy approximation guarantees for nonmonotone submodular function maximization, is worthwhile since the log-determinant function is non-monotone in general.
Also, note that both cardinality-constrained and unconstrained settings are important in DPP MAP inference and require substantially different technical ideas.
Therefore, we study the fast version of [11] separately from the algorithms for the cardinality-constrained setting.
In short, we accelerate various important greedy-type algorithms for DPP MAP inference.
1.1 Related work
The greedy algorithm (G REEDY ) is a popular approach to DPP MAP inference [31].
Han et al.
[24] gave a fast but inexact implementation of G REEDY .
Later, Chen et al.
[13] gave an exact implementation of G REEDY with the same time complexity as that of [24].
Han and Gillenwater [23] studied special cases where kernel matrices of DPPs are generated via customization (or re-weighting) of fixed feature vectors and developed a pre-processing method for accelerating G REEDY .
Note that this pre-processing usually takes longer than the algorithm of [13] (see [23, Section 4]), while their algorithm after pre-processing is much faster.
In summary, the algorithm of Chen et al.
[13] has been the fastest greedy-style algorithm for general DPPs.
A continuous-relaxation-based 1 / 4 -approximation algorithm for general down-closed constraints was also studied [21].
Our implementation of I NTERLACE G REEDY [28] yields a faster 1 / 4 -approximation algorithm for a special case with a cardinality constraint.
Approximation algorithms and inapproximability results of DPP MAP inference (without log) have also been extensively studied [15, 35, 36, 6, 42].
Sampling is another important research subject in DPPs and has been widely studied [2, 33, 16, 22, 1, 12, 32].
Also, we leave extension of our algorithms to non-symmetric DPPs [20], which have recently gained increasing attention, for future work.
Since the log-determinant function is known to be submodular [19], DPP MAP inference has a close connection to submodular function maximization [41].
Besides L AZY G REEDY , S TOCHASTICG REEDY [38, 43] is a popular fast variant of G REEDY , which we will discuss later.
A recent line of work [3, 4, 18, 8, 17, 14, 29] has studied adaptive algorithms for submodular function maximization, where we are allowed to execute polynomially many queries in parallel to reduce the number of sequential rounds.
Those studies consider oracle models of submodular functions, whereas we focus on the log-determinant functions and develop fast algorithms without such parallelization.
Background
Let [ n ] denote the set { 1 , 2 , . . . , n } for any n ∈ N .
For any S ⊆ [ n ] , S denotes its complement [ n ] \ S .
We use 0 and O as all-zero vectors and matrices, respectively, and 1 as all-ones vectors (their sizes will be clear from the context).
Let ⟨· , ·⟩ denote the inner product.
For a matrix L ∈ R n × n and subsets S, T ⊆ [ n ] , L [ S, T ] is the submatrix of L indexed by S in rows and T in columns.
For brevity, we write L i,j = L [ { i } , { j } ] , L [ S ] =
L [ S, S ] , L [ S, i ] =
L [ S, { i } ] , and L [ i, S ]
=
L [ { i } , S ] for any i, j ∈ [ n ] and S ⊆ [ n ] .
The determinant of L is denoted by det L .
Set det L [ ∅ ] = 1 by convention.
For any M ∈ R n × n , we suppose that M ⊤ M is computed in O( n ω ) time, which implies that we can compute det M and M − 1 (if non-singular) in O( n ω ) time (see, e.g., [7, Chapter 2]).
DPP MAP inference.
Let L ∈ R n × n be a positive semi-definite matrix.
A probability measure P on 2 [ n ] is called a determinantal point process (DPP) with a kernel matrix L if P [ X = S ] ∝ det L [ S ] holds for all S ⊆ [ n ] .
1 MAP inference for DPPs is the problem of finding a subset S ⊆ [ n ] with the largest det L [ S ] value.
We suppose that each i ∈ [ n ] is associated with a vector ϕ i ∈ R d and that a kernel matrix L ∈ R n × n is given as L = B ⊤ B , where B = [ ϕ 1 , ϕ 2 , . . . , ϕ n ] ∈ R d × n , i.e.,
L i,j = ⟨ ϕ i , ϕ j ⟩ for i, j ∈ [ n ] .
Note that (cid:112) det L [ S ] represents the volume of the parallelotope spanned by { ϕ i | i ∈ S } .
Hence, if the length and direction of ϕ i indicate i ’s importance and feature, respectively, the larger value of det L [ S ] implies that S contains more important and diverse items.
In many situations, we want to select a limited number of items; let k ∈ N denote the upper bound.
Therefore, MAP inference for DPPs with a cardinality constraint, or k -DPP [30], is often considered.
Note that since rank L ≤ min { n, d } , we can assume k ≤ min { n, d } without loss of generality.
Submodular function maximization.
For a set function f : 2 [ n ] → R ∪ {−∞} , the marginal gain of i ∈ [ n ] with respect to S ⊆ [ n ] is defined by f i ( S ) =
f ( S ∪ { i } ) − f ( S ) .
A set function f : 2 [ n ] → R ∪ {−∞} is called monotone if f i ( S ) ≥ 0 for every S ⊆ [ n ] and i ∈ S .
It is called submodular if it has the diminishing returns property : f i ( S ) ≥ f i ( T ) for every S ⊆ T ⊆ [ n ] and i ∈ T .
Since f ( S ) = log det L
[ S ] is submodular, DPP MAP inference can be written as submodular function maximization: max S ∈X f ( S ) , where X ⊆ 2 [ n ] is a family of feasible subsets.
This paper mostly considers the cardinality-constrained setting, i.e., X = { S ⊆
[ n ] | | S | ≤ k } for given k ∈ N ; in Section 4, we study the unconstrained setting, i.e., X = 2 [ n ] .
The log-determinant function f is monotone if the smallest eigenvalue of L is at least 1 (see, e.g., [44]), but this is not always the case.
It is well-known that the greedy algorithm (G REEDY ) enjoys a (1 − 1 / e) -approximation guarantee for cardinality-constrained monotone submodular function maximization with f ( ∅ ) ≥ {+
+}
This approximation ratio is optimal under the evaluation oracle model [40].
G REEDY works as follows: setting S (0) = ∅ , in each t th step ( t = 1 , . .
. , k ), choose j t ∈ arg max { f i ( S ( t − 1) )
| i ∈ S ( t − 1) } and put S ( t ) =
S ( t − 1) ∪ { j t } .
We call a solution obtained in this way a greedy solution .
Besides G REEDY , many algorithms [11, 10, 28, 43] achieve constant-factor approximations for cardinality-constrained/unconstrained submodular function maximization.
These results motivate us to apply submodular-function-maximization algorithms to DPP MAP inference, although constantfactor approximations of the log-determinant value do not imply those of the determinant value.
2.1 Lazy greedy algorithm for submodular function maximization
L AZY G REEDY [37] is an efficient implementation of G REEDY for submodular function maximization.
As explained above, G REEDY finds j t by computing marginal gains
f i ( S ( t − 1) ) for all i ∈ S ( t − 1) .
L AZY G REEDY attempts to find j t more efficiently by keeping an upper bound ρ i on
f
i ( S ( t − 1) ) for each i ∈ S ( t − 1) , which is an old marginal gain, i.e., ρ i = f i ( S ( u i ) ) for some u i ≤ t − 1 .
In each 1: V ← O, d (0) i ← (cid:112) L i,i ( ∀ i ∈ [ n ]) , S (0) ← ∅ 2: for t = 1 to k do 3: Take j t ∈ arg max i ∈ S ( t − 1) d ( t − 1)
i ▷ Terminate if d ( t − 1) j t ≤ 1 (i.e., f j t ( S ( t − 1) ) ≤ 0 ) 4: S ( t ) ← S ( t − 1) ∪ { j t } ▷ S ( t ) = { j 1 , . . . , j t } 5: for i in S ( t ) do ▷ Skip Lines 5–7 (updates for the next step) if t = k 6: V i,j t ← ( L i,j t − ⟨ V
[ i, S ( t − 1) ] , V [ j t , S ( t − 1) ] ⟩ ) /d ( t − 1) j t ▷ V [ i, ∅ ] = 0 for any i ∈ [ n ] 7: d ( t ) i ← (cid:113)(cid:0) d ( t − 1) i (cid:1)− V i,j t8: return S ( k ) ▷ Return S ( t − 1) if terminates with t < k iteration, L AZY G REEDY picks i ∈ S ( t − 1) with the largest ρ i value as the most promising element.
It then updates the ρ i value to the latest marginal gain
f i (cid:0) S ( t − 1) (cid:1) .
If ρ i is still the largest among ρ i ′ for all i ′ ∈ S ( t − 1) , the diminishing returns property guarantees i ∈ arg max { f i ′ ( S ( t − 1) ) | i ′ ∈ S ( t − 1) } , and hence it adds j t = i to S ( t − 1) .
L AZY G REEDY thus constructs a greedy solution, deferring updates of upper bounds of unpromising elements.
If the upper bounds are managed by a priority queue, every single iteration computes the marginal gain only once and makes O(log n ) comparisons (note that “iteration” is distinguished from “step” of G REEDY ).
With this contrivance, L AZY G REEDY runs much faster than G REEDY in practice, even though it does not improve the worst-case complexity.
2.2 Fast greedy algorithm for DPP MAP inference
We turn to DPP MAP inference with a kernel matrix L = B ⊤ B and B ∈ R d × n .
If we apply G REEDY to f ( S ) = log det L [ S ] , it takes O( k ω +1 n ) time since computing an f value takes O( k ω ) time; in addition, computing L at first takes O(min { n ω − 1 d, n 2 d ω − 2 } ) time.
F AST G REEDY [13] provides an
O( knd ) -time implementation of G REEDY for cardinality-constrained DPP MAP inference.
Algorithm 1 describes the procedure of F AST G REEDY , which is based on the Cholesky factorization of L with maximum pivoting.
The Cholesky decomposition produces a matrix V ∈ R n × n such that
L = V V ⊤ and PV is lower triangular for some permutation matrix P .
We call V a Cholesky factor of L .
In each t th step, Lines 5–7 calculate the t th column of PV via backward substitution (see
Fig. 1a).
The following Proposition 2.1 implies an important fact that once V [ i, S ( t ) ] is filled, we can obtain f i ( S ( t ) ) from d ( t ) i computed in Line 7, which is equal to the diagonal V i,i of the current
Cholesky factor.
Although the proposition is already proved in [13], we present a proof sketch since it would be helpful to understand the subsequent discussion (see [13] for the complete proof).
Proposition 2.1 ([13]) .
For t = 0 , 1 , . . .
, k − 1 , it holds that f i (cid:0) S ( t ) (cid:1) =
2 log d ( t ) i for every i ∈ S ( t ) .
Proof sketch.
The proof is by induction on t .
If t = 0 , it holds that f i (cid:0) S ( t ) (cid:1) =
f ( { i } ) − f ( ∅ ) =
log L i,i = 2 log d (0) i for i ∈ [ n ] .
Given a Cholesky factor V [ S ( t ) ] of L [ S ( t ) ] , for i ∈ S ( t ) , we have
L [ S ( t ) ∪ { i } ] = (cid:20) L (cid:2) S ( t ) (cid:3) L (cid:2) S ( t ) , i (cid:3) L (cid:2) i, S ( i ) (cid:3) L i
[Equation]
i (cid:21)
[Equation]
t [Equation] Hence, we have log det L [ S ( t ) ∪{ i } ]
= log (cid:0) d ( t )
i det V [ S ( t ) ] (cid:1) 2 = 2 log d ( t ) i +
log det L [ S ( t ) ] , which implies log det L [ S ( t ) ∪ { i } ] − log det L [ S ( t ) ]
= 2 log d ( t ) i .
Thus, the statement holds.
Therefore, by iteratively adding j t ∈ arg max i ∈ S ( t − 1) d ( t − 1) i to S ( t − 1) as in Algorithm 1, we can obtain a greedy solution.
Since L i,j t = ⟨ ϕ i , ϕ j t ⟩ is computed in O( d ) time and d ≥ k , Line 6 takes O( d ) time.
This is repeated O( kn ) times, hence the total time complexity of O( knd ) .
n t ( P V )[ S ( t ) ]
P V
[Figure]
n t ( P V )[ S ( t ) ] P V
[Figure]
Lazy and fast algorithms for cardinality-constrained DPP MAP inference 3.1 Lazy and fast greedy algorithm
We combine L AZY G REEDY and F AST G REEDY to obtain even faster L AZY F AST G REEDY .
As described above, L AZY G REEDY is designed for submodular function maximization under the oracle model, where a marginal gain is computed for each i ∈ S ( t − 1) .
Meanwhile, the core idea of F AST G REEDY is to obtain marginal gains of all i ∈ S ( t − 1) efficiently by computing a new column of a Cholesky factor, as in Fig.
1a.
To combine these seemingly incompatible methods, we need to take a closer look at how F AST G REEDY updates the entries in V .
The following observation is obvious but elucidates the essential row-wise independence of the updates of Cholesky-factor entries.
Observation 3.1.
In each t th step in Algorithm 1, for each i ∈ S ( t ) , V i,j t in Line 6 is computed from d ( t − 1) j t , L i,j t , and V [ { i, j t } , S ( t − 1) ] .
Thus, conditioned on S ( t ) , the i th row of V [ S ( t ) , S ( t ) ] can be updated independently for each i ∈ S ( t ) in a sense that computing V
i,j t only requires V i,j t ′ for t ′ <
t and V [ j t , S ( t − 1) ] , which is included in the already fixed Cholesky factor V [ S ( t ) ] .
That is, in Fig.
1b, once ( PV )[ S ( t ) ] is fixed, we can update the i th row independently for each i ∈ S ( t ) .
In the words of submodular function maximization, we can compute f i (cid:0) S ( t ) (cid:1) if f j t ′ (cid:0) S ( t ′ − 1) (cid:1) and f i (cid:0) S ( t ′ − 1) (cid:1) for t ′ ≤ t are available.
This enables us to apply the idea of lazy updates to F AST G REEDY .
Algorithm 2 describes our L AZY F AST G REEDY , and Fig.
1b illustrates how entries in V are updated.
In each iteration, it picks the most promising element i in Line 3, as with L AZY G REEDY .
Then, it calls U PDATE R OW to compute the entries of V [ i, S ] via backward substitution.
Once V [ i, S ] is filled, the resulting d i computed in Line 12 satisfies 2 log d i = f i ( S ) by Proposition 2.1, thus obtaining the latest marginal gain of i .
Lines 5–7 check whether the latest f i ( S ) is still the largest among the (old) marginal gains of i ′ ∈ S ; if so, j | S | +
= i is added to S .
Note that the deferred updates of d i ′ ( i ′
∈
S \ { i } ) do not matter when deciding whether to add i to S due to the submodularity.
Here, d = ( d 1 , . . .
, d n ) plays the role of upper bounds ( ρ 1 , . .
. , ρ n ) of L AZY G REEDY and is maintained by a priority queue.
L AZY F AST G REEDY thus finds a greedy solution by exactly mimicking the behavior of L AZY G REEDY while computing marginal gains efficiently as with F AST G REEDY .
The vector u = ( u 1 , . . . , u n ) keeps track of when the upper bounds are last updated; specifically, each u i ∈ { 0 , 1 , . . .
, k − 1 } indicates that the upper bound
d i is last updated with respect to { j 1 , . . .
, j u i } , i.e., 2 log d i = f i ( { j 1 , . . . , j u i } ) .
Since U PDATE R OW starts to fill V [ i, S ] from the ( u i + 1) st entry, no off-diagonals V
i,j t are computed more than once in Line 11.
Thus, the complexity of Algorithmdepends on the number of computed off-diagonals.
We denote it by u = (cid:80) i ∈ [ n ] u i with u i values at the end of Algorithm 2.
The u value changes in the range of [ k ( k − 1) / 2 , ( k − 1)( n − k/ 2)] depending on how well the lazy update works and affects the overall time complexity as follows.
1: V ← O, d ← (cid:0)(cid:112) L i,i (cid:1) i ∈ [ n ] , u ← 0 , S ← ∅ ▷ d is maintained by a priority queue 2: while | S | < k do 3:
Take i ∈ arg max i
′ ∈ S
d i ′ ▷ Terminate if d i ≤ 1 (i.e., f i ( S ) ≤ 0 ) 4: U PDATE R OW ( V, d , u ; i, S, L ) ▷ Nothing is done if | S | = 0 5: if d i ≥ max i ′
∈ S
d i ′ then
▷ Otherwise insert d i into the priority queue 6: j | S | +1 ← i 7:
S ← S ∪ { j | S | +1 } 8: return S 9: function
U PDATE R OW ( V, d , u ; i, S, L ) ▷ S = (cid:8) j 1 , j 2 , . . .
, j | S | (cid:9) 10: for t = u i + 1 , u i + 2 , . .
. , | S | do 11:
V i,j t ← ( L i,j t − ⟨ V
[ i, S ( t − 1) ] , V [ j t , S ( t − 1) ] ⟩ ) /d
j t ▷ S ( t − 1) =
{ j 1 , . . .
, j t − 1 } 12: d i ← (cid:113) d i 2 − V i,j t13: u i ← | S |
▷ This line is not needed in Algorithm
Theorem 3.2. Algorithm 2 returns a greedy solution in O( nd + u ( d +log n )) time.
If the lazy update works best and worst, it runs in O(( n + k 2 ) d ) and O( kn ( d + log n )) time, respectively.
Proof.
Algorithm 2 returns a greedy solution as explained above.
We below discuss the running time.
At the beginning, we need O( nd ) time to compute L i,i =
⟨ ϕ i , ϕ i ⟩ for i = 1 , . . . , n .
In U PDATE R OW , an access to L i,j t in Line 11 takes O( d ) time, and the inner product takes O( k ) ( ≲ O( d ) ) time.
This computation is done u times, and thus the total computation time caused by U PDATE R OW is O( ud ) .
In Line 5, we need O(log n ) time to update the priority queue if d i < max i
′ ∈ S
d i ′ , which can hold only when at least one off-diagonal is computed in U PDATE R OW .
Therefore, Line 5 takes O( u log n ) time in total.
Thus, the overall time complexity is O( nd + u ( d + log n )) .
Let S be the output of Algorithm 2 and P a permutation matrix such that ( PV )[ S ] is lower triangular.
In the best case, U PDATE R OW is called up to k times and u = k
( k − 1) / 2 off-diagonals of ( PV )
[ S ] are computed.
Moreover, updates of the priority queue are done only up to k times, taking O( k log n ) ( ≲ O( nd ) ) time in total.
Thus, it runs in O(( n + k 2 ) d ) time.
In the worst case, Algorithm 2 calculates the off-diagonals of ( PV )[ S ] and all the entries of V [ S, S \ { j k } ] ; the total number of those entries is u = k
( k − 1) /
+
( k − 1)( n − k )
=
( k − 1)( n − k/ 2) .
Hence, it takes O( kn ( d + log n )) time.
The best-case time complexity is better than O( knd ) of F AST G REEDY if k = o( n ) .
Even in the worse case, it is as fast as F AST G REEDY if d = Ω(log n ) .
Note that both k
= o( n ) and d = Ω(log n ) are true in most practical situations.
Experiments in Section 5 demonstrate that L AZY F AST G REEDY can run much faster than F AST G REEDY in practice.
3.2 Extension to random, stochastic, and interlace greedy algorithms
The core idea of L AZY F AST G REEDY can be used for speeding up other greedy-type algorithms:
R ANDOM G REEDY [10], S TOCHASTIC G REEDY [38, 43], and I NTERLACE G REEDY [28], which enjoy 1 / e -, 1 / 4 -, and 1 / 4 -approximation guarantees, respectively, for non-monotone submodular function maximization with a cardinality constraint.
Note that the guarantees for the non-monotone case are essential in DPP MAP inference since the log-determinant function is non-monotone in general.
Due to the space limitation, we present the details of those extensions in Appendix A.
Fast double greedy algorithm for unconstrained DPP MAP inference
This section discusses unconstrained DPP MAP inference with a kernel matrix L = B ⊤ B , where
B ∈ R d × n .
In this setting, if f ( S ) = log det L
[ S ] is monotone, S =
[ n ] is a trivial optimal solution.
Thus, we suppose f to be non-monotone.
We also assume L to be positive definite since the algorithm of [11] discussed below requires f ( S ) >
−∞ for any S ⊆ [ n ] .
Note that this implies d ≥ n .
−2: V ← O , W ← O , d ← (cid:0)(cid:112) L i,i (cid:1) i ∈ [ n ] , e ← (cid:16)(cid:113)
( L − 1 ) i,i (cid:17)
i ∈ [ n ] , S ← ∅ 3: for do 4: U PDATE R OW V d
0 ; i, S, L ) ▷ S is sorted in
. . , n 5:
U PDATE R OW ( W, e , 0 ; i, [ i ] \ S, L − 1 )
▷ [ i ] \ S is sorted in order
. . , n 6: a i ← max { 2 log d i , 0 } , b i ← max { 2 log e i , 0 } 7:
S ← S ∪ { i }
w p a
i
a
+ b i )
▷ Implicity update
= n
\ S w.p. b ( a i + b i ) 8: return S A famous algorithm for unconstrained submodular function maximization is D OUBLE G REEDY [11], a randomized 1 / 2 -approximation algorithm.
Although it calls an evaluation oracle only O( n ) times, its naive implementation is too costly for large DPP MAP inference instances since computing the logdeterminant function value takes O( n ω ) time, which will lead to the total time of O( n ω − 1 d + n ω +1 ) .
We below extend the idea of F AST G REEDY [13] to D OUBLE G REEDY and obtain its O( n ω − 1 d + n 3 ) - time implementation for unconstrained DPP MAP inference.
D OUBLE G REEDY maintains two subsets S and T , which are initially set to S = ∅ and T =
[ n ] .
For i = 1 , . . . , n , it computes a i = max { f i ( S ) , 0 } and b i = max {− f i ( T \ { i } ) , 0 } , and then either adds i to S with probability a i / ( a
i + b i ) or removes i from T with probability
b i / ( a i + b i ) .
2 Note that
T = [ n ] \ ([ i ] \ S ) =
[ i ] \ S always holds.
Finally, it returns S (or equivalently T =
[ n ] \ S = S ).
As for the growing subset S , we can efficiently compute marginal gains
f i ( S ) by incrementally updating a Cholesky factor, as with F AST G REEDY .
When it comes to the shrinking subset T , however, we cannot directly use the efficient incremental update for computing − f i ( T \ { i } ) .
If we naively compute it in each step, it takes O( n ω ) time, resulting in the same time complexity as the naive implementation.
Our key idea for overcoming this difficulty is to use the following Jacobi’s complementary minor formula (see, e.g., [9]).
Proposition 4.1.
Let L ∈ R n × n be a non-singular matrix and I, J ⊆ [ n ] be subsets with | I | = | J | .
Then, it holds that det L
[ I, J ] =
( − 1)
[Equation]
[-det L det L − 
-]
This formula provides a lemma that enables us to compute − f i ( T \ { i } ) via incremental updates.
Lemma 4.2.
Let L ∈ R n × n be positive definite.
For any S ⊆ [ n ] , define f ( S ) =
log det L
[ S ] and g ( S ) =
log det L − 1 [ S ] .
Then, g ( S ∪ { i } ) − g ( S ) =
f ( S \ { i } ) − f ( S ) holds for any S ⊆
[ n ] and i ∈ [ n ] .
Proof.
By using Proposition 4.1, we can prove the claim as follows:
[Equation]
− 1 [[ n ] \ ( S \ { i } )] − log det L det L − 1 [[ n ] \ S ] = log det L − 1 [ S ∪ { i } ] − log det L − 1 [ S ] = g ( S ∪ { i } ) − g ( S ) . Lemma 4.2 implies g (([ i ] \ S ) ∪ { i } ) − g ([ i ] \ S ) =
f (([ i ] \ S ) \ { i } ) − f ([ i ] \ S ) =
− f i ( T ) .
Therefore, by computing L = B ⊤ B and L − 1 in O( n ω − 1 d ) ( ≳ O( n ω ) ) time in advance, we can compute − f i ( T \ { i } ) in each step by incrementally updating a Cholesky factor of size | [ i ] \ S | .
Algorithm 3 presents our F AST D OUBLE G REEDY based on this idea.
In each i th step, the first and second calls to U PDATE R OW , defined in Algorithm 2, fill V [ i, S ] and W [ i, [ i ] \ S ] , respectively.
Hence, Proposition 2.1 and Lemma 4.2 imply 2 log d i = f i ( S ) and
2 log e i =
g i ([ i ] \ S ) =
− f i ( T ) .
Thus, Algorithm 3 exactly mimics the behavior of D OUBLE G REEDY while efficiently computing marginal gains via incremental updates of Cholesky factors.
In U PDATE R OW defined in Algorithm 2, since L and L − 1 are already computed, each V i,j t is calculated in O( n ) time; therefore, a single call to U PDATE R OW takes O( n 2 ) time.
Since U PDATE R OW is called 2 n times, Lines 2–7 construct a solution in O( n 3 ) time.
In total, Algorithm 3 runs in O( n ω − 1 d + n 3 ) time.
Experiments
We evaluate the effectiveness of our acceleration techniques on synthetic and real-world datasets.
Section 5.1 examines speed-ups of G REEDY for cardinality-constrained DPP MAP inference, and
Section 5.2 focuses on D OUBLE G REEDY for the unconstrained setting.
We present experimental results on R ANDOM G REEDY , S TOCHASTIC G REEDY , and I NTERLACE G REEDY in Appendix A.4.
The algorithms are implemented in C++ with library Eigen 3.4.0 for matrix computations.
Experiments are conducted using a compiler GCC 10.2.0 on a computer with 3 . 8 GHz Intel Xeon Gold
CPU and 800 GB RAM.
We use synthetic and two real-world datasets, Netflix Prize [5] and MovieLens [25].
Each dataset provides a matrix B ∈ R d × n consisting of column vectors ϕ 1 , ϕ 2 , . . . , ϕ n ∈ R d of n items, which defines an n × n kernel matrix L = B ⊤ B .
Below we explain the item vectors of each dataset.
Synthetic datasets.
We use the setting of [21].
Each entry of ϕ i ∈ R d is independently drawn from the standard normal distribution, ϕ ij ∼ N (0 , 1) .
As a result, the kernel matrix L conforms to a
Wishart distribution with n degrees of freedom and an identity covariance matrix, i.e., L ∼ W ( I, n, n ) .
We consider various n values in the experiments below, and we always set the vector length d to n .
Real-world datasets.
Both Netflix Prize and MovieLens datasets contain users’ ratings of movies from one to five, where we regard a movie as an item.
Following [13], we binarize the ratings based on whether it is greater than or equal to four.
After that, we eliminate movies that result in all-zero vectors and users who result in all-zero ratings since those are redundant.
Consequently, the Netflix
Prize dataset has n = 17770 movies and d = 478615 users with 56919190 ratings; the MovieLens dataset has n = 40858 movies and d = 162342 users with 12452811 ratings.
- and L -input settings.
It is important to care about whether L = B ⊤ B is computed in advance or not.
In practice, a matrix B ∈ R d × n of item vectors is often given.
Then, computing L = B ⊤ B in advance takes O(min { n ω − 1 d, n 2 d ω − 2 } ) time, which we should avoid when n is large since the running time of L AZY F AST G REEDY (and F AST G REEDY ) increases only linearly in n .
On the other hand, we are sometimes given a pre-computed kernel matrix L , and we can access L i,j in O(1) time.
We below consider both settings, called B - and L -input settings, respectively.
In the L -input setting, we exclude the time to compute L = B ⊤ B from consideration.
Under this condition, F AST G REEDY takes O( k 2 n ) time and L AZY F AST G REEDY does O( n + u ( k + log n )) , where the first O( n ) term is for constructing a priority queue.
By similar reasoning to that in the proof of Theorem 3.2, it runs in
O( n + k 3 + k log n ) and O( kn ( k + log n )) time if the lazy update works best and worst, respectively.
5.1 Greedy algorithm for cardinality-constrained DPP MAP inference
We compare the running time of G REEDY , L AZY G REEDY , F AST G REEDY , and L AZY F AST G REEDY , which we here call Naive, Lazy, Fast, and LazyFast, respectively, for short.
As regards synthetic datasets, we consider two settings that fix either n or k : (i) n =
and k = 1 , 2 , . . . , n , and (ii) k = 200 and n = 1000 , 2000 , . .
We set the timeout periods of (i) and (ii) to 3600 andseconds, respectively.
Regarding real-world datasets, n is fixed as explained above and we increase k = 1 , 2 , . . .
, n .
We set the timeout period to 3600 seconds.
With the Netflix (MovieLens) dataset, the objective value peaked with k =
( k = 18763 ), and thus we stopped increasing k after that.
Figures 2 and 3 present the results on synthetic and real-world datasets, respectively.
The curves in the runtime figures represent that faster algorithms can return greedy solutions to instances with larger k or n values within the timeout periods.
The rightmost figures present the number of off-diagonals of Cholesky factors computed by Fast and LazyFast.
As explained in Section 3.1, while Fast always computes all the off-diagonals of V [[ n ] , S ] , LazyFast does not due to the lazy update.
LazyFast was the fastest in all the settings.
In particular, in the synthetic (ii) and real-world settings,
LazyFast computed fewer off-diagonals than Fast, thus running the fastest.
In the synthetic setting (i), although LazyFast computed almost all off-diagonals in V [[ n ] , S ] , it was still faster than Fast.
For example, for the L -input setting with n = k = 6000 , Fast and LazyFast took 37 .
4 and 13 .
7 seconds, respectively.
This unexpected speed-up is caused by the cache efficiency of LazyFast.
Specifically, every call to U PDATE R OW computes off-diagonals from V i,j ui +1 to V i,j | S | by accessing entries only in
[Figure]
V [ { i, j u i +1 , . . .
, j | S | } , S ( | S |− 1) ] .
This process virtually creates blocks in a Cholesky factor, enabling the cache-efficient computation of off-diagonals via blocking (see, e.g., [26, Section 2.3]).
In fact, the cache miss rates of Fast and LazyFast for the above example were 71 .
3% and 3 .
3% , respectively.
Another interesting observation in the synthetic (ii) and real-world settings is that while Fast was often faster than Lazy in the L -input setting, the opposite occurred in the B -input setting.
This is because computing an off-diagonal in the B -input setting is costly relative to the L -input setting.
As a result, avoiding the redundant computation of off-diagonals by the lazy update tends to be more effective than computing marginal gains efficiently via the Cholesky factorization.
[-Synthetic 30 .36233 .
-]
[-> 86400 .— 37829 .
-]
5.2 Double greedy algorithm for unconstrained DPP MAP inference
We compare naive D OUBLE G REEDY and our F AST D OUBLE G REEDY by applying them to unconstrained DPP MAP inference on three datasets: synthetic ( n = 6000 ), Netflix Prize, and MovieLens.
Since kernel matrices L in the real-world datasets are singular, we use kernel matrices computed as
L = 0 .
9 B ⊤ B + 0 .
1 I in this section, ensuring that the resulting matrices L are positive definite.
In this section, we set the timeout period to one day ( 86400 seconds).
Table 1 presents the results.
Note that both algorithms require L to be computed in advance.
Therefore, we measured the time of computing B ⊤ B (Product) separately from the time of constructing solutions (Greedy).
Our F AST D OUBLE G REEDY additionally requires L − 1 to be computed in advance for accelerating the solution construction; therefore, we also measured the time of computing L −(Inverse) separately.
As in Table 1, naive D OUBLE G REEDY took so long for solution construction that it failed to return solutions to real-world instances in one day.
By contrast, our F AST D OUBLE G REEDY constructed solutions far faster and succeeded in returning solutions to all the instances.
Also, the computation of B ⊤ B (Product) took a considerably long time relative to the running time of L AZY F AST G REEDY in the previous section.
Therefore, as mentioned above, when a matrix B of item vectors is given and our goal is to obtain a greedy solution for small k
= o( n ) , we should avoid computing the kernel matrix L = B [Figure] s [Figure]
[Table] Product Greedy Total
Product
Inverse
Greedy
Total
Synthetic n =1 .258 .259 .1 .1 .0 .3 .n
{+=8 .
+}
.10 .35 .n
{+=30 .36233 .
+}
.59 .34 .124 .n
=70 .> 86400 .
— 70 .151 .103 .324 .n
=137 .> 86400 .
Netflix Prize 20561 .
> 86400 .
MovieLens 37829 .
{+> 86400 .— 37829 .
 Acknowledgements The authors thank anonymous reviewers for their valuable comments. This work was supported by+}
JST ERATO Grant Number JPMJER1903 and JSPS KAKENHI Grant Number JP22K17853.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Abstract and Section 1.
(b) Did you describe the limitations of your work?
[Yes] The remarkable limitation is that our L AZY F AST G REEDY does not improve the worst-case time complexity, and it is clear from the description in Section 3.1.
(c) Did you discuss any potential negative societal impacts of your work?
[No] This is a purely algorithmic study, and no negative societal impacts are expected.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2.
Additional assumptions required in the unconstrained setting is described in Section 4.
(b) Did you include complete proofs of all theoretical results?
[Yes] All theoretical statements are followed by their proofs.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A]
No training is performed in our experiments.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] Although some algorithms are randomized, it is obvious that the randomness does not largely affect our experimental results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See [5, 25].
(b) Did you mention the license of the assets?
[No] We used publicly available standard datasets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] The codes of the algorithms are provided in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] Since the datasets are publicly available standard ones, no such discussion seems to be necessary.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] Since the datasets are publicly available standard ones, no such discussion seems to be necessary.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]