S CALABLE S AMPLING FOR N ONSYMMETRIC D ETERMINANTAL P OINT P ROCESSES
A BSTRACT
A determinantal point process (DPP) on a collection of M items is a model, parameterized by a symmetric kernel matrix, that assigns a probability to every subset of those items.
Recent work shows that removing the kernel symmetry constraint, yielding nonsymmetric DPPs (NDPPs) can lead to significant predictive performance gains for machine learning applications.
However, existing work leaves open the question of scalable NDPP sampling.
There is only one known
DPP sampling algorithm based on Cholesky decomposition that can directly apply to NDPPs as well
Unfortunately , its runtime is cubic in M , and thus does not scale to large item collections.
In this work , we first note that this algorithm can be transformed into a linear-time one for kernels with low -rank structure.
Furthermore , we develop a scalable sublinear-time rejection sampling algorithm by constructing a novel proposal distribution.
Additionally, we show that imposing certain structural constraints on the NDPP kernel enables us to bound the rejection rate in a way that depends only on the kernel rank.
In our experiments we compare the speed of all of these samplers for a variety of real-world tasks
I NTRODUCTION
A determinantal point process (DPP) on M items is a model, parameterized by a symmetric kernel matrix, that assigns a probability to every subset of those items.
DPPs have been applied to a wide range of machine learning tasks, including stochastic gradient descent (SGD) (Zhang et al., 2017), reinforcement learning (Osogami & Raymond, 2019; Yang et al., 2020), text summarization (Dupuy & Bach, 2018), coresets (Tremblay et al., 2019), and more.
However, a symmetric kernel can only capture negative correlations between items.
Recent works (Brunel, 2018; Gartrell et al., 2019) have shown that using a nonsymmetric DPP (NDPP) allows modeling of positive correlations as well, which can lead to significant predictive performance gains.
Gartrell et al.
(2021) provides scalable
NDPP kernel learning and MAP inference algorithms, but leaves open the question of scalable sampling.
The only known sampling algorithm for NDPPs is the Cholesky-based approach described in Poulson (2019), which has a runtime of O ( M 3 ) and thus does not scale to large item collections.
There is a rich body of work on efficient sampling algorithms for (symmetric) DPPs, including recent works such as Derezinski et al.
; Poulson (2019); Calandriello et al.
Key distinctions between existing sampling algorithms include whether they are for exact or approximate sampling, whether they assume the DPP kernel has some low-rank K ≪ M , and whether they sample from the space of all 2 M subsets or from the restricted space of size- k subsets, so-called k -DPPs. In the context of MAP inference, influential work, including Summa et al. ; Chen et al. Hassani et al.
; Ebrahimi et al. ; Indyk et al. , proposed efficient algorithms that the approximate (sub)determinant maximization problem and provide rigorous guarantees. In this work we focus on exact sampling for low-rank kernels, and provide scalable algorithms for NDPPs.
Our contributions are as follows, with runtime and memory details summarized in Table 1:
• Linear-time sampling (Section 3): We show how to transform the O ( M 3 )
Choleskydecomposition-based sampler from Poulson (2019) into an O ( MK 2 ) sampler for rank- K kernels.
[Table] • Sublinear-time sampling (Section 4): Using rejection sampling, we show how to leverage existing sublinear-time samplers for symmetric DPPs to implement a sublinear-time sampler for a subclass of NDPPs that we call orthogonal NDPPs (ONDPPs).
Learning with orthogonality constraints (Section 5): We show that the scalable NDPP kernel learning of Gartrell et al.
(2021) can be slightly modified to impose an orthogonality constraint, yielding the ONDPP subclass.
The constraint allows us to control the rejection sampling algorithm’s rejection rate, ensuring its scalability.
Experiments suggest that the predictive performance of the kernels is not degraded by this change.
For a common large-scale setting where M is 1 million our sublinear-time sampler results in runtime that is hundreds of times faster than the linear-time sampler.
In the same setting our linear-time sampler provides runtime that is millions of times faster than the only previously known NDPP sampling algorithm which has cubic time complexity and is thus impractical in this scenario
B ACKGROUND
Notation.
We use [ M ] := { 1 , . . .
, M } to denote the set of items 1 through M .
We use I K to denote the K -by- K identity matrix, and often write I := I M when the dimensionality should be clear from context.
Given L ∈ R M × M , we use L i,j to denote the entry in the i -th row and j -th column, and
L A,B ∈ R | A |×| B | for the submatrix formed by taking rows A and columns B .
We also slightly abuse notation to denote principal submatrices with a single subscript, L A :=
L A,A .
Kernels.
As discussed earlier, both (symmetric) DPPs and NDPPs define a probability distribution over all 2 M subsets of a ground set [ M ] .
The distribution is parameterized by a kernel matrix
L ∈ R M × M and the probability of a subset Y ⊆
[ M ] is defined to be Pr( Y ) ∝ det( L Y ) .
For this to define a valid distribution, it must be the case that det( L Y ) ≥ 0 for all Y .
For symmetric DPPs, the non-negativity requirement is identical to a requirement that L be positive semi-definite (PSD).
For nonsymmetric DPPs, there is no such simple correspondence, but prior work such as Gartrell et al.
(2019; 2021) has focused on PSD matrices for simplicity.
Normalizing and marginalizing.
The normalizer of a DPP or NDPP distribution can also be written as a single determinant: (cid:80) Y ⊆
[ M ] det( L Y )
= det( L + I ) (Kulesza & Taskar, 2012, Theorem 2.1).
Additionally, the marginal probability of a subset can be written as a determinant: Pr( A ⊆ Y )
= det( K A ) , for K :
= I − ( L + I ) − 1 (Kulesza & Taskar, 2012, Theorem 2.2) * , where K is typically called the marginal kernel.
Intuition.
The diagonal element K i,i is the probability that item i is included in a set sampled from the model.
The 2 -by- 2 determinant det( K { i,j } ) =
K i,i K j,j − K i,j K j,j is the probability that both i and j are included in the sample.
A symmetric DPP has a symmetric marginal kernel, meaning
K i,j = K j,i , and hence K i,i K j,j − K i,j K j,i ≤ K i,i K j,j .
This implies that the probability of including both i and j in the sampled set cannot be greater than the product of their individual inclusion probabilities.
Hence, symmetric DPPs can only encode negative correlations.
In contrast, NDPPs can have K i,j and K j,i with differing signs, allowing them to also capture positive correlations.
2. R ELATED W ORK
Learning.
Gartrell et al.
(2021) proposes a low-rank kernel decomposition for NDPPs that admits linear-time learning.
The decomposition takes the form L := V V ⊤ + B ( D − D ⊤ )
B ⊤ for
Algorithm
1: procedure S AMPLE C HOLESKY ( K ) ▷ marginal kernel factorization Z , W 2:
Y ← ∅
Q ← W 3: for i = 1 to M do 4: p i ← K i,i p i ← z ⊤ i
Qz i 5: u ← uniform (0 , 1) 6: if u ≤ p i then Y ← Y ∪ { i } 7: else p i ← p i −8:
K A ← K A − K A,i K i,A p i for A :
= { i + 1 , . . .
, M }
Q ← Q − Qz i z ⊤ i Q p i 9: return Y
V , B ∈ R M × K , and D ∈ R K × K . The V V ⊤ component is a rank- K symmetric matrix, which can model negative correlations between items. The B ( D − D ⊤ ) B ⊤ component is a rank- K skewsymmetric matrix, which can model positive correlations between items.
For compactness of notation, we will write L = ZXZ ⊤ , where Z = (cid:2) V B (cid:3) ∈ R M × 2 K , and X = (cid:104) I K {+0 
 D − D ⊤ (cid:105) ∈+}
R 2 K × 2 K .
The marginal kernel in this case also has a rank- 2 K decomposition, as can be shown via application of the Woodbury matrix identity:
[Equation]
Note that the matrix to be inverted can be computed from Z and X in O ( MK 2 ) time, and the inverse itself takes O ( K 3 ) time.
Thus, K can be computed from L in time O ( MK 2 ) .
We will develop sampling algorithms for this decomposition, as well as an orthogonality-constrained version of it.
We use W :=
X (cid:0)
I 2 K + Z ⊤ ZX (cid:1) − 1 in what follows so that we can compactly write K = ZW Z ⊤ .
Sampling.
While there are a number of exact sampling algorithms for DPPs with symmetric kernels, the only published algorithm that clearly can directly apply to NDPPs is from Poulson (2019) (see
Theorem 2 therein).
This algorithm begins with an empty set Y = ∅ and iterates through the M items, deciding for each whether or not to include it in Y based on all of the previous inclusion/exclusion decisions.
Poulson (2019) shows, via the Cholesky decomposition, that the necessary conditional probabilities can be computed as follows:
[Equation]
Algorithm 1 (left-hand side) gives pseudocode for this Cholesky-based sampling algorithm † .
There has also been some recent work on approximate sampling for fixed-size k -NDPPs: Alimohammadi et al.
(2021) provide a Markov chain Monte Carlo (MCMC) algorithm and prove that the overall runtime to approximate ε -close total variation distance is bounded by O ( M 2 k 3 log(1 / ( ε Pr( Y 0 ))) , where Pr( Y 0 ) is probability of an initial state Y 0 .
Improving this runtime is an interesting avenue for future work, but for this paper we focus on exact sampling.
L INEAR - TIME C HOLESKY -B ASED S AMPLING
In this section, we show that the O ( M 3 ) runtime of the Cholesky-based sampler from Poulson (2019) can be significantly improved when using the low-rank kernel decomposition of Gartrell et al.
First, note that Line 8 of Algorithm 1, where all marginal probabilities are updated via an ( M − i ) -by-
( M − i ) matrix subtraction, is the most costly part of the algorithm, making overall time and memory complexities O ( M 3 ) and O ( M 2 ) , respectively.
However, when the DPP kernel is given by a low-rank decomposition, we observe that marginal probabilities can be updated by matrix-vector (Tree-based sampling) 1: procedure P REPROCESS ( V , B , D ) 2: { ( σ j , y 2 j − 1 , {+y 2 j ) } K/ 
 j =1 ← Y OULA D ECOMPOSE ( B , D ) ‡ 3: ˆ X ← diag (cid:0)+}
I K , σ 1 , σ 1 , . . .
, σ K/ 2 , σ K/ 2 (cid:1) 4: Z {+← [ V , y 1 , . . , y K ] { ( λ i , z i ) } 2 Ki =
 ← E IGEN D ECOMPOSE (+}
Z ˆ X 1 / 2 )
T ← C ONSTRUCT T REE ( M, [ z 1 , . . .
, z 2 K ] ⊤ ) 5: return Z , ˆ X return T , { ( λ i , z i ) }
2 Ki = 6: procedure S AMPLE R
EJECT ( V , B , D , Z , ˆ X )
▷ tree T , eigen pair { ( λ i , z i ) } 2 Ki =1 of Z
ˆ XZ 7: while true do 8:
Y ← S AMPLE DPP ( Z ˆ XZ ⊤ )
Y ← S AMPLE DPP ( T , { ( λ i , z i ) } 2 Ki =1 ) 9: u ← uniform(0 , 1) 10: p ← det([ V V ⊤ + B
D − D ⊤ ) B ⊤ ] Y ) det([ Z ˆ XZ ⊤ ] Y ) 11: if u ≤ p then break 12: return Y multiplications of dimension 2 K , regardless of M .
In more detail, suppose we have the marginal kernel K = ZW Z ⊤ as in Eq.
(1) and let z j be the j -th row vector in Z .
Then, for i ̸ = j : Pr ( j ∈ Y | i ∈ Y ) =
K j,j − ( K j,i K i,j ) / K i,i = z ⊤ j (cid:18) W − ( W z i )( z ⊤ i W )
z ⊤ i W z i (cid:19) [Equation]
The conditional probabilities in Eqs.
(4) and (5) are of bilinear form, and the z j do not change during sampling.
Hence, it is enough to update the 2 K -by- 2 K inner matrix at each iteration, and obtain the marginal probability by multiplying this matrix by z i .
The details are shown on the right-hand side of
Algorithm 1.
The overall time and memory complexities are O ( MK 2 ) and O ( MK ) , respectively.
S UBLINEAR - TIME R EJECTION S AMPLING
Although the Cholesky-based sampler runs in time linear in M , even this is too expensive for the large M that are often encountered in real-world datasets.
To improve runtime, we consider rejection sampling (Von Neumann, 1963).
Let p be the target distribution that we aim to sample, and let q be any distribution whose support corresponds to that of p ; we call q the proposal distribution.
Assume that there is a universal constant U
such that p ( x ) ≤
Uq ( x ) for all x .
In this setting, rejection sampling draws a sample x from q and accepts it with probability p ( x ) /
( Uq ( x )) , repeating until an acceptance occurs.
The distribution of the resulting samples is p .
It is important to choose a good proposal distribution q so that sampling is efficient and the number of rejections is small.
[-y 2 j − 1 , y 2 j ) } K/ 
-]
[-, y K ] { ( λ i , z i ) } 2 Ki =
-]
4. P ROPOSAL DPP C ONSTRUCTION
Our first goal is to find a proposal DPP with symmetric kernel (cid:98) L that can upper-bound all probabilities of samples from the NDPP with kernel L within a constant factor.
To this end, we expand the determinant of a principal submatrix, det( L Y ) , using the spectral decomposition of the NDPP kernel.
Such a decomposition essentially amounts to combining the eigendecomposition of the symmetric part of L with the Youla decomposition (Youla, 1961) of the skew-symmetric part.
Specifically, suppose { ( σ j , y 2 j − 1 , y 2 j ) } K/ 2 j =1 is the Youla decomposition of B ( D − D ⊤ ) B ⊤ (see
Appendix D for more details), that is,
[Equation]
Then we can simply write L = ZXZ ⊤ , for Z :=
[ V , y 1 , . . .
, y K ] ∈ R M × 2 K , and
[Equation]
Now, consider defining a related but symmetric PSD kernel (cid:98) L :=
Z ˆ XZ ⊤ with ˆ X := diag (cid:0)
I K , σ 1 , σ 1 , . . .
, σ K/ 2 , σ K/ 2 (cid:1) .
All determinants of the principal submatrices of (cid:98) L =
Z ˆ XZ ⊤ upper-bound those of L , as stated below.
Theorem 1.
For every subset Y ⊆
[ M ] , it holds that det( L Y ) ≤ det( (cid:98) L Y ) .
Moreover, equality holds when the size of Y is equal to the rank of L .
Proof sketch :
From the Cauchy-Binet formula, the determinants of L Y and (cid:98) L Y for all Y ⊆
[ M ] , | Y | ≤ 2 K can be represented as
[Equation]
Many of the terms in Eq.
(8) are actually zero due to the block-diagonal structure of X .
For example, note that if 1 ∈ I but 1 / ∈ J , then there is an all-zeros row in X I,J , making det( X I,J )
= 0 .
We show that each X I,J with nonzero determinant is a block-diagonal matrix with diagonal entries among ± σ j , or (cid:104)
σ j − σ j 0 (cid:105) .
With this observation, we can prove that det( X I,J ) is upper-bounded by det(ˆ X I ) or det(ˆ X J ) .
Then, through application of the rearrangement inequality, we can upper-bound the sum of the det( X I,J ) det( Z Y,I ) det( Z Y,J ) in Eq.
(8) with a sum over det(ˆ X I ) det( Z Y,I ) 2 .
Finally, we show that the number of non-zero terms in Eq.
(8) is identical to the number of non-zero terms in
Eq.
Combining these gives us the desired inequality det( L Y ) ≤ det( (cid:98) L Y ) .
The full proof of
Theorem 1 is in Appendix E.1.
Now, recall that the normalizer of a DPP (or NDPP) with kernel L is det( L + I ) .
The ratio of probability of the NDPP with kernel L to that of a DPP with kernel (cid:98) L is thus:
[Equation]
where the inequality follows from Theorem 1.
This gives us the necessary universal constant U upper-bounding the ratio of the target distribution to the proposal distribution.
Hence, given a sample
Y drawn from the DPP with kernel (cid:98) L , we can use acceptance probability Pr L ( Y ) /
( U Pr (cid:98) L ( Y ))
=
det( L Y ) / det( (cid:98) L Y ) .
Pseudo-codes for proposal construction and rejection sampling are given in
Algorithm 2.
Note that to derive (cid:98) L from L it suffices to run the Youla decomposition of B ( D −
D ⊤ ) B ⊤ , because the difference is only in the skew-symmetric part.
This decomposition can run in
O ( MK 2 ) time; more details are provided in Appendix D. Since (cid:98) L is a symmetric PSD matrix, we can apply existing fast DPP sampling algorithms to sample from it.
In particular, in the next section we combine a fast tree-based method with rejection sampling.
4. S UBLINEAR - TIME T REE - BASED S AMPLING
There are several DPP sampling algorithms that run in sublinear time, such as tree-based (Gillenwater et al., 2019) and intermediate (Derezinski et al., 2019) sampling algorithms.
Here, we consider applying the former, a tree-based approach, to sample from the proposal distribution defined by (cid:98) L .
We give some details of the sampling procedure, as in the course of applying it we discovered an optimization that slightly improves on the runtime of prior work.
Formally, let { ( λ i , z i ) } 2 K i =1 be the eigendecomposition of (cid:98) L and Z :=
[ z 1 , . . .
, z 2 K ] ∈ R M × 2 K .
As shown in Kulesza & Taskar (2012,
Lemma 2.6), for every Y ⊆
[ M ] , | Y | ≤ 2 K , the probability of Y under DPP with (cid:98) L can be written:
[Equation]
A matrix of the form Z : ,E Z ⊤ : ,E can be a valid marginal kernel for a special type of DPP, called an elementary DPP.
Hence, Eq.
(10) can be thought of as DPP probabilities expressed as a mixture of elementary DPPs.
Based on this mixture view, DPP sampling can be done in two steps: (1) choose an elementary DPP according to its mixture weight, and then (2) sample a subset from the selected elementary DPP.
Step (1) can be performed by 2 K independent random coin tossings, while step (2) involves computational overhead.
The key idea of tree-based sampling is that step (2) can be accelerated by traversing a binary tree structure, which can be done in time logarithmic in M .
More specifically, given the marginal kernel K =
Z : ,E Z ⊤ : ,E , where E is obtained from step (1), we start from the empty set Y = ∅ and repeatedly add an item j to Y with probability:
[Equation]
where S is some final selected subset, and Q Y :
= I | E | − Z ⊤ Y,E (cid:0) Z Y,E Z ⊤ Y,E (cid:1) − 1 Z Y,E .
Consider a binary tree whose root includes a ground set [ M ] .
Every non-leaf node contains a subset A ⊆
[ M ] and stores a 2 K -by- 2 K matrix (cid:80) j ∈
A Z ⊤ j, : Z j, : .
A partition A ℓ and A r , such that A ℓ ∪ A r = A, A ℓ ∩ A r = ∅ , are passed to its left and right subtree, respectively.
The resulting tree has M leaves and each has exactly a single item.
Then, one can sample a single item by recursively moving down to the left node with probability:
[Equation]
12) or to the right node with probability 1 − p ℓ , until reaching a leaf node.
An item in the leaf node is chosen with probability according to Eq.
Since every subset in the support of an elementary DPP with a rank- k kernel has exactly k items, this process is repeated for | E | iterations.
Full descriptions of tree construction and sampling are provided in Algorithm 3.
The proposed tree-based rejection sampling for an NDPP is outlined on the right-side of Algorithm 2.
The one-time pre-processing step of constructing the tree (C ONSTRUCT T REE ) requires O ( MK 2 ) time.
After pre-processing, the procedure S AMPLE DPP involves | E | traversals of a tree of depth O (log M ) , where in each node a
O ( | E | 2 ) operation is required.
The overall runtime is summarized in Proposition 1 and the proof can be found in Appendix E.2.
Proposition 1.
The tree-based sampling procedure S AMPLE DPP in Algorithm 3 runs in time
O ( K + k 3 log M + k 4 ) , where k is the size of the sampled set § .
4. A VERAGE N UMBER OF R EJECTIONS
We now return to rejection sampling and focus on the expected number of rejections.
The number of rejections of Algorithm 2 is known to be a geometric random variable with mean equal to the constant U used to upper-bound the ratio of the target distribution to the proposal distribution: det( (cid:98) L + I ) / det( L + I ) .
If all columns in V and B are orthogonal, which we denote V ⊥ B , then the expected number of rejections depends only on the eigenvalues of the skew-symmetric part of the
NDPP kernel.
Theorem 2.
Given an NDPP kernel L = V V ⊤ + B ( D − D ⊤ ) B ⊤ for V , B ∈ R M × K , D ∈
R K × K , consider the proposal kernel (cid:98) L as proposed in Section 4.1.
Let { σ j } K/ 2 j =1 be the positive eigenvalues obtained from the Youla decomposition of B ( D − D ⊤ ) B ⊤ .
If V ⊥ B , then det( (cid:98) L + I ) det( L + I ) = (cid:81) K/
j =1 (cid:16) 1 + 2 σ j σ 2 j +1 (cid:17) ≤ (1 + ω ) K/ 2 , where ω = 2 K (cid:80) K/
j =1 2 σ j σ 2 j +1 ∈ (0 , 1] .
Proof sketch :
Orthogonality between V and B allows det( L + I ) to be expressed just in terms of the eigenvalues of V V ⊤ and B ( D − D ⊤ ) B ⊤ .
Since both L and (cid:98) L share the symmetric part V V ⊤ , the ratio of determinants only depends on the skew-symmetric part.
A more formal proof appears in
Appendix E.3.
Assuming we have a kernel where V ⊥ B , we can combine Theorem 2 with the tree-based rejection sampling algorithm (right-side in Algorithm 2) to sample in time
O (( K + k 3 log M + k 4 )(1+ ω ) K/ 2 ) .
Hence, we have a sampling algorithm that is sublinear in M , and can be much faster than the Choleskybased algorithm when (1 + ω ) K/ 2 ≪ M .
In the next section, we introduce a learning scheme with the V ⊥ B constraint, as well as regularization to ensure that ω is small.
L EARNING WITH O RTHOGONALITY C ONSTRAINTS
We aim to learn a NDPP that provides both good predictive performance and a low rejection rate.
We parameterize our NDPP kernel matrix L = V V ⊤ + B ( D − D ⊤ ) B ⊤ by
[Equation]
for σ j ≥ 0 , B ⊤ B = I , and, motivated by Theorem 2, require V ⊤ B = 0 ¶ .
We call such orthogonality-constrained NDPPs “ONDPPs”.
Notice that if V ⊥ B , then L has the full rank of 2 K , since the intersection of the column spaces spanned by V and by B is empty, and thus the full rank available for modeling can be used.
Thus, this constraint can also be thought of as simply ensuring that ONDPPs use the full rank available to them.
Given example subsets { Y 1 , . . .
, Y n } as training data, learning is done by minimizing the regularized negative log-likelihood:
[Equation]
where α, β, γ > 0 are hyperparameters, µ i is the frequency of item i in the training data, and v i and
b i represent the rows of V and B , respectively.
This objective is very similar to that of Gartrell et al.
(2021), except for the orthogonality constraint and the final regularization term.
Note that this regularization term corresponds exactly to the logarithm of the average rejection rate, and therefore should help to control the number of rejections.
E XPERIMENTS
We first show that the orthogonality constraint from Section 5 does not degrade the predictive performance of learned kernels.
We then compare the speed of our proposed sampling algorithms.
[Table]
6. P REDICTIVE P ERFORMANCE R ESULTS FOR NDPP L EARNING
We benchmark various DPP models, including symmetric (Gartrell et al., 2017), nonsymmetric for scalable learning (Gartrell et al., 2021)
[Table]
as well as our ONDPP kernels with and without rejection rate regularization.
We use the scalable NDPP models (Gartrell et al., 2021) as a baseline || .
The kernel components of each model are learned using five real-world recommendation datasets, which have ground set sizes that range from 3 , 941 to 1 , 059 , 437 items (see Appendix A for more details) [Table]
Our experimental setup and metrics mirror those of Gartrell et al.
We report the mean percentile rank (MPR) metric for a next-item prediction task, the AUC metric for subset discrimination, and the log-likelihood of the test set; see Appendix B for more details on the experiments and metrics.
For all metrics, higher numbers are better.
For NDPP models, we additionally report the average rejection rates when they apply to rejection sampling.
In Table
[Table]
we observe that the predictive performance of our ONDPP models generally match or sometimes exceed the baseline.
This is likely because the orthogonality constraint enables more effective use of the full rank- 2 K feature space.
Moreover, imposing the regularization on rejection rate, as shown in Eq.
(14), often leads to dramatically smaller rejection rates, while the impact on predictive performance is generally marginal.
These results justify the ONDPP and regularization for fast sampling.
Finally, we observe that the learning time of our ONDPP models is typically a bit longer than that of the NDPP models, but still quite reasonable (e.g., the time per iteration for the
NDPP takes 27 seconds for the Book dataset, while our ONDPP takes 49.7 seconds).
Fig.
1 shows how the regularizer γ affects the test log-likelihood and the average number of rejections.
We see that γ degrades predictive performance and reduces the rejection rate when set above a certain threshold; this behavior is seen for many datasets.
However, for the Recipe dataset we observed that the test log-likelihood is not very sensitive to γ , likely because all models in our experiments achieve very high performance on this dataset.
In general, we observe that γ can be set to a value that results in a small rejection rate, while having minimal impact on predictive performance.
6. S AMPLING T IME C OMPARISON
We benchmark the Cholesky-based sampling algorithm
(Algorithm 1) and
tree-based rejection sampling algorithm
(Algorithm 2) on ONDPPs with both synthetic and real-world data.
|| We use the code from https://github.com/cgartrel/scalable-nonsymmetric-DPPs for the NDPP baseline, which is made available under the MIT license.
To simplify learning and MAP inference, Gartrell et al. set B = V in their experiments. However, since we have the V ⊥ B constraint in our ONDPP approach, we cannot set B = V . Hence, for a fair comparison, we do not set B = V for the NDPP baseline in our experiments, thus the results in Table slightly those published
Gartrell et al.
[Figure]
[Figure]
[Figure]
[Table]
Spectral decomposition
[Table]
[Figure]
[Table]
Synthetic datasets.
We generate non-uniform random features for V , B as done by (Han & Gillenwater, 2020).
In particular, we first sample x 1 , . . .
, x 100 from N ( 0 , I 2 K / (2 K )) , and integers t 1 , . . .
, t 100 from Poisson distribution with mean 5 , rescaling the integers such that (cid:80) i t i = M .
Next, we draw t i random vectors from N ( x i , I 2 K ) , and assign the first K -dimensional vectors as the row vectors of V and the latter vectors as those of B .
Each entry of D is sampled from N (0 , 1) .
We choose K = 100 and vary M from 2 12 to 2 20 .
Fig. 2(a) illustrates the runtimes of Algorithms 1 and 2.
We verify that the rejection sampling time tends to increase sub-linearly with the ground set size M , while the Cholesky-based sampler runs in linear time.
In Fig. 2(b), the runtimes of the preprocessing steps for Algorithm 2 (i.e., spectral decomposition and tree construction) are reported.
Although the rejection sampler requires these additional processes, they are one-time steps and run much faster than a single run of the
Choleksy-based method for M = 2 20 .
Real-world datasets.
In Table 3, we report the runtimes and speedup of NDPP sampling algorithms for real-world datasets.
All NDPP kernels are obtained using learning with orthogonality constraints, with rejection rate regularization as reported in Section 6.1.
We observe that the tree-based rejection sampling runs up to 246 times faster than the Cholesky-based algorithm.
For larger datasets, we expect that this gap would significantly increase.
As with the synthetic experiments, we see that the tree construction pre-processing time is comparable to the time required to draw a single sample via the other methods, and thus the tree-based method is often the best choice for repeated sampling ** .
C ONCLUSION
In this work we developed scalable sampling methods for NDPPs.
One limitation of our rejection sampler is its practical restriction to the ONDPP subclass.
Other opportunities for future work include the extension of our rejection sampling approach to the generation of fixed-size samples (from k -NDPPs), the development of approximate sampling techniques, and the extension of DPP samplers along the lines of Derezinski et al.
; Calandriello et al.
(2020) to NDPPs.
Scalable sampling also opens the door to using NDPPs as building blocks in probabilistic models.
E THICS S TATEMENT
In general, our work moves in a positive direction by substantially decreasing the computational costs of NDPP sampling.
When using our constrained learning method to learn kernels from user data, we recommend employing a technique such as differentially-private SGD (Abadi et al., 2016) to help prevent user data leaks, and adjusting the weights on training examples to balance the impact of sub-groups of users so as to make the final kernel as fair as possible.
As far as we are aware, the datasets used in this work do not contain personally identifiable information or offensive content. We were not able to determine if user consent was explicitly obtained by the organizations that constructed these datasets.  R EPRODUCIBILITY S TATEMENT We have made extensive effort to ensure that all algorithmic, theoretical, and experimental contributions described in this work are reproducible.
All of the code implementing our constrained learning and sampling algorithms is publicly available †† .
The proofs for our theoretical contributions are available in Appendix E.
For our experiments, all dataset processing steps, experimental procedures, and hyperparameter settings are described in Appendices A, B, and C, respectively.
A CKNOWLEDGEMENTS
Amin Karbasi acknowledges funding in direct support of this work from NSF (IIS-1845032) and
ONR (N00014-19-1-2406).
A F ULL DETAILS ON DATASETS
We perform experiments on several real-world public datasets composed of subsets:
• UK Retail: This dataset (Chen et al., 2012) contains baskets representing transactions from an online retail company that sells all-occasion gifts.
We omit baskets with more than 100 items, leaving us with a dataset containing 19 , 762 baskets drawn from a catalog of M = 3 , 941 products.
Baskets containing more than 100 items are in the long tail of the basket-size distribution, so omitting these is reasonable, and allows us to use a low-rank factorization of the NDPP with K = 100 .
• Recipe: This dataset (Majumder et al., 2019) contains recipes and food reviews from Food.com (formerly Genius Kitchen) ‡‡ .
Each recipe (“basket”) is composed of a collection of ingredients, resulting in 178 , 265 recipes and a catalog of 7 , 993 ingredients.
• Instacart: This dataset (Instacart, 2017) contains baskets purchased by Instacart users §§ .
We omit baskets with more than 100 items, resulting in 3.2 million baskets and a catalog of 49 , 677 products.
• Million Song: This dataset (McFee et al., 2012) contains playlists (“baskets”) of songs from Echo Nest users ¶¶ .
We trim playlists with more than 100 items, leaving 968 , 674 playlists and a catalog of 371 , 410 songs.
• Book: This dataset (Wan & McAuley, 2018) contains reviews from the Goodreads book review website, including a variety of attributes describing the items *** .
For each user we build a subset (“basket”) containing the books reviewed by that user.
We trim subsets with more than 100 books, resulting in 430 , 563 subsets and a catalog of 1 , 059 , 437 books.
As far as we are aware, these datasets do not contain personally identifiable information or offensive content.
While the UK Retail dataset is publicly available, we were unable to find a license for it.
Also, we were not able to determine if user consent was explicitly obtained by the organizations that constructed these datasets.
B F ULL DETAILS ON EXPERIMENTAL SETUP AND METRICS
We use 300 randomly-selected baskets as a held-out validation set, for tracking convergence during training and for tuning hyperparameters.
Another 2000 random baskets are used for testing, and the rest are used for training.
Convergence is reached during training when the relative change in validation log-likelihood is below a predetermined threshold.
We use PyTorch with Adam (Kingma &
Ba, 2015) for optimization.
We initialize D from the standard Gaussian distribution N (0 , 1) , while
V and B are initialized from the uniform (0 , 1) distribution.
Subset expansion task.
We use greedy conditioning to do next-item prediction (Gartrell et al., 2021,
Section 4.2).
We compare methods using a standard recommender system metric: mean percentile rank (MPR)
(Hu et al., 2008; Li et al., 2010).
MPR of 50 is equivalent to random selection; MPR of 100 means that the model perfectly predicts the next item.
See Appendix B.1 for a complete description of the MPR metric.
Subset discrimination task.
We also test the ability of a model to discriminate observed subsets from randomly generated ones.
For each subset in the test set, we generate a subset of the same length by drawing items uniformly at random (and we ensure that the same item is not drawn more than once for a subset).
We compute the AUC for the model on these observed and random subsets, where the score for each subset is the log-likelihood that the model assigns to the subset. the
B. M EAN P ERCENTILE R ANK
We begin our definition of MPR by defining percentile rank (PR).
First, given a set J , let p i,J =
Pr( J ∪ { i } | J ) .
The percentile rank of an item i given a set J is defined as
[Equation]
where Y\ J indicates those elements in the ground set Y that are not found in J .
For our evaluation, given a test set Y , we select a random element i ∈ Y and compute PR i,Y \{ i } .
We then average over the set of all test instances T to compute the mean percentile rank (MPR):
[Equation]
C H YPERPARAMETERS FOR EXPERIMENTS
Preventing numerical instabilities : The det( L Y i ) in Eq.
(14) will be zero whenever | Y i | > K , where Y i is an observed subset.
To address this in practice we set K to the size of the largest subset observed in the data, K ′ , as in Gartrell et al.
However, this does not entirely fix the issue, as there is still a chance that the term will be zero even when | Y i | ≤ K .
In this case though, we know that we are not at a maximum, since the value of the objective function is −∞ .
Numerically, to prevent such singularities, in our implementation we add a small ϵ I correction to each L Y i when optimizing Eq.
(14) ( ϵ = 10 − 5 in our experiments).
We perform a grid search using a held-out validation set to select the best-performing hyperparameters for each model and dataset.
The hyperparameter settings used for each model and dataset are described below.
Symmetric low-rank DPP (Gartrell et al., 2017).
For this model, we use K for the number of item feature dimensions for the symmetric component V , and α for the regularization hyperparameter for
V .
We use the following hyperparameter settings:
• UK Retail dataset: K = 100 , α = 1 .
• Recipe dataset: K = 100 , α = 0 .•
Instacart dataset: K = 100 , α = 0 .
• Million Song dataset: K = 100 , α = 0 . 0001 .
• Book dataset: K = 100 , α = 0 .
Scalable NDPP (Gartrell et al., 2021).
As described in Section 2.1, we use K to denote the number of item feature dimensions for the symmetric component V and the dimensionality of the nonsymmetric component D .
α and β are the regularization hyperparameters.
We use the following hyperparameter settings:
• UK dataset: K = 100 , α = 0 .
• Recipe dataset: K = 100 , α = β
=
• Instacart dataset: K = 100 , α = 0 .
• Million Song dataset: K = 100 , α = 0 .
• Book dataset: K = 100 , α = β = 0 .
ONDPP .
As described in Section 5, we use K to denote the number of item feature dimensions for the symmetric component V and the dimensionality of the nonsymmetric component C .
α , β , and γ are the regularization hyperparameters.
We use the following hyperparameter settings:
• UK dataset: K = 100 , α = β = 0 .
01 , γ = 0 .
• Recipe dataset: K = 100 , α = β
= 0 .
01 , γ = 0 .
• Instacart dataset: K = 100 , α
=
β
=
001 , γ = 0 .
• Million Song dataset: K = 100 , α = β = 0 .
01 , γ = 0 .
• Book dataset: K = 100 , α = β = 0 .
01 , γ = 0 .
For all of the above model configurations and datasets, we use a batch size of 800 during training.
D Y OULA D ECOMPOSITION : S PECTRAL D ECOMPOSITION FOR S KEW - SYMMETRIC M ATRIX
We provide some basic facts on the spectral decomposition of a skew-symmetric matrix, and introduce an efficient algorithm for this decomposition when it is given by a low-rank factorization.
We write i :=
√− 1 and v H as the conjugate transpose of v ∈ C M , and denote Re( z ) and Im( z ) by the real and imaginary parts of a complex number z , respectively.
Given B ∈ R M × K and D ∈
R K × K , consider a rank- K skew-symmetric matrix B
( D − D ⊤ ) B ⊤ .
Note that all nonzero eigenvalues of a real-valued skew-symmetric matrix are purely imaginary.
Denote i σ 1 , − i σ 1 , . .
, i σ K/ 2 , − i σ K/ 2 by its nonzero eigenvalues where each of σ j is real, and a 1 + i b 1 , a 1 − i b 1 , . . .
a K/ 2 + i b K/ 2 , a K/ 2 − i b K/ 2 by the corresponding eigenvectors for a j , b j ∈ R M , which come in conjugate pairs.
Then, we can write
[Equation]
[Equation]
[Equation]
Note that a 1 ± b 1 , . .
. , a K/ 2 ± b K/ 2 are real-valued orthonormal vectors, because a 1 , b 1 , . .
a K/ 2 , b K/ 2 are orthogonal to each other and ∥ a j ± b j ∥ 22 = [-(cid:107) a j (cid:107) 
-]∥
a j ∥ 22 + ∥ b j [-(cid:107) 
-]∥ 22 = 1 for all j .
The pair { ( σ j , a j − b j , a j + b j ) } K/ 2 j =1 is often called the Youla decomposition (Youla, 1961) of
B ( D − D ⊤ ) B ⊤ .
To efficiently compute the Youla decomposition of a rank- K matrix, we use the following result.
Proposition 2 (Proposition 1, Nakatsukasa (2019)) .
Given A , B ∈ C M × K , the nonzero eigenvalues of AB ⊤ ∈ C M × M and B ⊤ A ∈ C K × K are identical.
In addition, if ( λ, v ) is an eigenpair of B ⊤ A with λ ̸ = 0 , then ( λ, Av / ∥ Av ∥ 2 ) is an eigenpair of AB ⊤ .
From the above proposition, one can first compute ( D − D ⊤ ) B ⊤ B and then apply the eigendecomposition to that K -by- K matrix.
Taking the imaginary part of the obtained eigenvalues gives us the σ j ’s, and multiplying B by the eigenvectors gives us the eigenvectors of B ( D − D ⊤ ) B ⊤ .
In addition, this can be done in O ( MK 2 + K 3 ) time; when M > K it runs much faster than the eigendecomposition of B ( D − D ⊤ ) B ⊤ , which requires O ( M 3 ) time.
The pseudo-code of the
Youla decomposition is provided in Algorithm 4.
1: procedure Y OULA D ECOMPOSITION ( B , D ) 2: { ( η j , z j ) , ( η j , z j ) } K/ 2 j =1 ← eigendecomposition of ( D − D ⊤ ) B ⊤ B 3: for j = 1 , . .
. , K/ 2 do 4:
σ j ← Im( η j ) for j = 1 , . . .
, K/5:
y
j − 1 ← B (Re( z j ) − Im( z j )) 6: y 2 j ← B (Re( z j ) + Im( z j )) 7: y j ← y j / ∥ y j ∥ for j = 1 , . . .
, K 8: return { ( σ j , y
j − 1 , y 2 j ) } K/ 2 j =
E P ROOFS E. P ROOF OF T HEOREM
Theorem 1.
For every subset Y ⊆
[ M ] , it holds that det( L Y ) ≤ det( (cid:98) L Y ) .
Moreover, equality holds when the size of Y is equal to the rank of L .
Proof of Theorem 1.
It is enough to fix Y ⊆
[ M ] such that 1 ≤ | Y | ≤ 2 K , because the rank of both
L and (cid:98) L is up to 2 K .
Denote k :=
| Y | and (cid:0) [2 K ] k (cid:1) :
= { I ⊆
[2 K ]; | I | = k } for k ≤ 2 K .
We recall the definition of (cid:98) L : given V , B , D such that L = V V ⊤ + B ( D − D ⊤ ) B ⊤ , let { ( ρ i , v i ) }
Ki = be the eigendecomposition of V V ⊤ and { ( σ j , y 2 j − 1 , y 2 j ) } K/
j =1 be the Youla decomposition of
B ( D − D ⊤ ) B ⊤ .
Denote Z := [ v 1 , . .
. , v K , y 1 , . . .
, y K ] ∈ R M × 2 K and
X := diag (cid:18) ρ, . . .
, ρ K , (cid:20)σ− σ (cid:21) , . . .
, (cid:20) σ K/− σ K/ (cid:21)(cid:19) , ˆ X :=
diag (cid:18) ρ 1 , . .
. , ρ K , (cid:20) σ  σ(cid:21) , . . .
, (cid:20) σ K/  σ K/(cid:21)(cid:19) , so that L = ZXZ ⊤ and (cid:98) L =
Z ˆ XZ ⊤ .
Applying the Cauchy-Binet formula twice, we can write the determinant of the principal submatrices of both L and (cid:98) L :
[Equation]
where Eq.
(19) follows from the fact that ˆ X is diagonal, which means that det(ˆ X I,J ) = {+
+} for I ̸
= J .
When the size of Y is equal to the rank of L (i.e., k = 2 K ), the summations in Eqs.
(18) and (19) simplify to single terms: det( L Y )
= det( X ) det( Z Y, : ) 2 and det( (cid:98) L Y ) =
det(ˆ X ) det( Z Y, : ) 2 .
Now, observe that the determinants of the full X and ˆ X matrices are identical: det( X ) =
det(ˆ X ) =
(cid:81)
Ki =
1 ρ
i (cid:81) K/ 2 j =1 σ 2 j .
Hence, it holds that det( L Y ) =
det( (cid:98) L Y ) .
This proves the second statement of the theorem.
To prove that det( L Y ) ≤ det( (cid:98) L Y ) for smaller subsets Y , we will use the following:
Claim 1.
For every I, J ∈ (cid:0) [2 K ] k (cid:1) such that det( X I,J ) ̸ = 0 , there exists a (nonempty) collection of subset pairs S ( I, J ) ⊆
(cid:0) [2 K ] k (cid:1) × (cid:0) [2 K ] k (cid:1) such that
[Equation]
Claim 2.
The number of nonzero terms in Eq.
(18) is identical to that in Eq.
Combining Claim 1 with Claim 2 yields
[Equation]
We conclude the proof of Theorem 1.
Below we provide proofs for Claim 1 and Claim 2.
Proof of Claim 1.
Recall that X is a block-diagonal matrix, where each block is of size either 1 -by- 1 , containing ρ i , or 2 -by- 2 , containing both σ j and − σ j in the form (cid:104)
σ j − σ j 0 (cid:105) .
A submatrix
X I,J ∈ R k × k with rows I and columns J will only have a nonzero determinant if it contains no all-zero row or column.
Hence, any X I,J with nonzero determinant will have the following form (or some permutation of this block-diagonal):
[Equation]
 
[Equation] [Equation]
and we denote P
I,J :=
{ p 1 , . .
. , p | P I,J | } , Q I,J :=
{ q 1 , . . . , q | Q I,J | } , and R I,J :=
{ r 1 , . . .
, r | R I,J | } .
Indices p ∈ P
I,J yield a diagonal matrix with entries ρ p .
For such p , both I and J must contain index p .
Indices r ∈ R
I,J yield a block-diagonal matrix of the form (cid:2)
σ r − σ r 0 (cid:3) .
For such r , both I and J must contain a pair of indices, ( K + 2 r − 1 , K + 2 r ) .
Finally, indices q ∈ Q I,J yield a diagonal matrix with entries of ± σ q (the sign can be + or − ).
For such q , I contains K + 2 q − 1 or K + 2 q , and J must contain the other.
Note that there is no intersection between Q I,J and R I,J .
If Q I,J is an empty set (i.e., I = J ), then det( X I,J ) =
det(ˆ X
I,J ) and [Equation]
Thus, the terms in Eq.
(18) in this case appear in Eq.
Now assume that Q I,J ̸ = ∅ and consider the following set of pairs:
[Equation]
In other words, for ( I ′ , J ′ ) ∈ S
( I, J ) , the diagonal X I ′ ,J ′ contains ρ p , (cid:2)
σ r − σ r 0 (cid:3) exactly as in X I,J .
However, the signs of the σ r ’s may differ from X
I,J .
Combining this observation with the definition of ˆ X ,
[Equation]
Therefore,
[Equation]
[Equation] [Equation] [Equation]
[Equation]
(28) where the third line comes from Eq.
(23) and the fourth line follows from the rearrangement inequality.
Note that application of this inequality does not change the number of terms in the sum.
This completes the proof of Claim 1.
Proof of Claim 2.
In Eq.
(19), observe that det(ˆ X I ) det( Z Y,I ) 2 {+̸ = 
+}
[-= 
-] if and only if det(ˆ X I ) ̸ = 0 .
Since all ρ i ’s and σ j ’s are positive, the number of I ⊆
[2 K ] , | I | =
k such that det(ˆ X I ) ̸ = 0 is equal to (cid:0) 2 Kk (cid:1) .
Similarly, the number of nonzero terms in Eq.
(18) equals the number of possible choices of I, J ∈ (cid:0) [2 K ] k (cid:1) such that det( X I,J ) ̸ = 0 .
This can be counted as follows: first choose i items in { ρ 1 , . .
. , ρ K } for i = 0 , . . .
, k ; then, choose j items in (cid:110)(cid:2)[-
-]
[Table] [Table]
[Table]
for j = 0 , . . .
, ⌊ k − i 2 ⌋ ; lastly, choose k − i − 2 j of {± σ q ; q / ∈ R
I,J } , then choose the sign for each of these ( σ q or − σ q ).
Combining all of these choices, the total number of nonzero terms is:
[Table]
[Equation]
[Table]
[Equation]
31) where the second line comes from the fact that (cid:0) 2 nm (cid:1) =
(cid:80)
⌊ m 2 ⌋ j =0 (cid:0) nj (cid:1)(cid:0) n − j m − 2 j (cid:1) 2 m − 2 j for any integers n, m ∈ N such that m ≤ 2 n (see (1.69) in Quaintance (2010)), and the third line follows from the fact that (cid:80) ri =0 (cid:0) mi (cid:1)(cid:0) nr − i (cid:1) =
(cid:0) n + mr (cid:1) for n, m, r ∈ N (Vandermonde’s identity).
Hence, both the number of nonzero terms in Eqs.
(18) and (19) is equal to (cid:0) 2 Kk (cid:1) .
This completes the proof of Claim 2.
E. P ROOF OF P ROPOSITION
Proposition 1.
The tree-based sampling procedure S AMPLE DPP in Algorithm 3 runs in time
O ( K + k 3 log M + k 4 ) , where k is the size of the sampled set ††† .
Proof of Proposition 1.
Since computing p ℓ takes O ( k 2 ) from Eq.
(12), and since the binary tree has depth O (log M ) , S AMPLE
I TEM in Algorithm 3 runs in O ( k 2 log M ) time.
Moreover, the query matrix Q Y can be updated in O ( k 3 ) time as it only requires a k -by- k matrix inversion.
Therefore, the overall runtime of the tree-based elementary DPP sampling algorithm (after pre-processing) is
O ( k 3 log M + k 4 ) .
This improves the previous O ( k 4 log M ) runtime studied in Gillenwater et al.
Combining this with elementary DPP selection (Line 15 in Algorithm 3), we can sample a set in O ( K + k 3 log M + k 4 ) time.
This completes the proof of Proposition 1.
E. P ROOF OF T HEOREM
Theorem 2.
Given an NDPP kernel L = V V ⊤ + B ( D − D ⊤ ) B ⊤ for V , B ∈ R M × K , D ∈
R K × K , consider the proposal kernel (cid:98) L as proposed in Section 4.1.
Let { σ j } K/ 2 j =1 be the positive eigenvalues obtained from the Youla decomposition of B ( D − D ⊤ ) B ⊤ .
If V ⊥ B , then det( (cid:98) L + I ) det( L + I ) = (cid:81) K/
j =1 (cid:16) 1 + 2 σ j σ 2 j +1 (cid:17) ≤ (1 + ω ) K/ 2 , where ω = 2 K (cid:80) K/
j =1 2 σ j σ 2 j +1 ∈ (0 , 1] .
Proof of Theorem 2.
Since the column spaces of V and B are orthogonal, the corresponding eigenvectors are also orthogonal, i.e., Z ⊤ Z
= I 2 K .
Then,
[Equation] (32) = [Equation]
(cid:89)
[Equation] [Equation]
det (cid:18)(cid:20) [Equation] (cid:21)(cid:19) (33) = [Equation]
(cid:89) i
[Equation]
[Equation]
[Equation]
Computing p ℓ via Eq.
(12) improves on Gillenwater et al.
(2019)’s
O ( k 4 log M ) runtime for this step.
and similarly
[Equation]
Combining Eqs.
(34) and (35), we have that
[Equation]
where the inequality holds from the Jensen’s inequality.
This completes the proof of Theorem 2.