D PP N ET : A PPROXIMATING D ETERMINANTAL P OINT P ROCESSES WITH D EEP N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Determinantal Point Processes (D PP s) provide an elegant and versatile way to sample sets of items that balance the point-wise quality with the set-wise diversity of selected items.
For this reason, they have gained prominence in many machine learning applications that rely on subset selection.
However, sampling from a
D PP over a ground set of size N is a costly operation, requiring in general
an
O ( N 3 ) preprocessing cost and an O ( Nk 3 ) sampling cost for subsets of size k .
We approach this problem by introducing D PP N ET s: generative deep models that produce D PP -like samples for arbitrary ground sets.
We develop an inhibitive attention mechanism based on transformer networks that captures a notion of dissimilarity between feature vectors.
We show theoretically that such an approximation is sensible as it maintains the guarantees of inhibition or dissimilarity that makes
D PP s so powerful and unique.
Empirically, we demonstrate that samples from our model receive high likelihood under the more expensive D PP alternative.
I NTRODUCTION
Selecting a representative sample of data from a large pool of available candidates is an essential step of a large class of machine learning problems: noteworthy examples include automatic summarization, matrix approximation, and minibatch selection.
Such problems require sampling schemes that calibrate the tradeoff between the point-wise quality – e.g. the relevance of a sentence to a document summary – of selected elements and the set-wise diversity 1 of the sampled set as a whole.
Determinantal Point Processes (D PP s) are probabilistic models over subsets of a ground set that elegantly model the tradeoff between these often competing notions of quality and diversity.
Given a ground set of size N , D PP s allow for O ( N 3 ) sampling over all 2 N possible subsets of elements, assigning to any subset S of a ground set Y of elements the probability
[Equation]
where L ∈ R N × N is the D PP kernel and L S =
[ L ij ] i,j ∈ S denotes the principal submatrix of L indexed by items in S .
Intuitively, D PP s measure the volume spanned by the feature embedding of the items in feature space (Figure 1).
First introduced by Macchi (1975) to model the distribution of possible states of fermions obeying the
Pauli exclusion principle, the properties of D PP s have since then been studied in depth (Hough et al., 2006; Borodin, 2009, see e.g. ).
As D PP s capture repulsive forces between similar elements, they arise in many natural processes, such as the distribution of non-intersecting random walks (Johansson, 2004), spectra of random matrix ensembles (Mehta & Gaudin, 1960; Ginibre, 1965), and zerocrossings of polynomials with Gaussian coefﬁcients (Hough et al., 2009).
More recently, D PP s have become a prominent tool in machine learning due to their elegance and tractability: recent applications include video recommendation (Chen et al., 2018), minibatch selection (Zhang et al., 2017), and kernel approximation (Li et al., 2016b; Mariet et al., 2018).
However, the O ( N 3 ) sampling cost makes the practical application of D PP s intractable for large datasets, requiring additional work such as subsampling from Y , structured kernels (Gartrell et al.,
[Figure]
2017; Mariet & Sra, 2016b), or approximate sampling methods (Anari et al., 2016a; Li et al., 2016a;
Affandi et al., 2013).
Nonetheless, even such methods require signiﬁcant pre-processing time, and scale poorly with the size of the dataset.
Furthermore, when dealing with ground sets with variable components, pre-processing costs cannot be amortized, signiﬁcantly impeding the application of
D PP s in practice.
These setbacks motivate us to investigate the use of more scalable models to generate high-quality, diverse samples from datasets to obtain highly-scalable methods with ﬂexibility to adapt to constantly changing datasets.
Speciﬁcally, we use generative deep models to approximate the D PP distribution over a ground set of items with both ﬁxed and variable feature representations.
We show that a simple, carefully constructed neural network, D PP N ET , can generate D PP -like samples with very little overhead, while maintaining fundamental theoretical properties of D PP measures.
Furthermore, we show that D PP N ET s can be trivially employed to sample from a conditional D PP (i.e. sampling S such that A ⊆ S is predeﬁned) and for greedy mode approximation.
Contributions.
• We introduce D PP N ET , a deep network trained to generate D PP -like
samples based on static and variable ground sets of possible items.
• We derive theoretical conditions under which the D PP N ET s inherit the D PP ’s log-submodularity.
• We show empirically that D PP N ET s provide an accurate approximation to D PP s and drastically speed
up D PP sampling.
R ELATED WORK
D PP s belong to the class of Strongly Rayleigh (SR) measures; these measures beneﬁt from the strongest characterization of negative association between similar items; as such, SR measures have beneﬁted from signiﬁcant interest in the mathematics community (Pemantle, 2000;
Borcea et al., 2009; Borcea & Br¨and´en, 2008; Marcus et al., 2015) and more recently in machine learning (Anari et al., 2016b; Li et al., 2016a; Mariet et al., 2018).
This, combined with their tractability, makes
D PP s a particularly attractive tool for subset selection in machine learning, and is one of the key motivations for our work.
The application of D PP s to machine learning problems spans ﬁelds from document and video summarization (Gong et al., 2014;
Chao et al., 2015), recommender systems (Zhou et al., 2010; Chen et al., 2018) and object retrieval (Affandi et al., 2014) to kernel approximation (Li et al., 2016b), neural network pruning (Mariet & Sra, 2016a), and minibatch selection (Zhang et al., 2017).
Zou &
Adams (2012) developed DPP priors for encouraging diversity in generative models and Snoek et al.
(2013) showed that DPPs accurately model inhibition in neural spiking data.
In the general case, sampling exactly from a D PP requires an initial eigendecomposition of the kernel matrix L , incurring a O ( N 3 ) cost.
In order to avoid this time-consuming step, several approximate sampling methods have been derived; Affandi et al.
(2013) approximate the D PP kernel during sampling; more recently, results by Anari et al.
(2016a) followed by Li et al.
(2016a) showed that
DPPs are amenable to efﬁcient MCMC-based sampling methods.
estimation from a D
1: function S AMPLE ( k , A ) 2:
S ← A 3: while | S | < k do 4: v ← D PP N ET ( S ) 5: if sampling then 6: i ∼ Multinomial ( v/ (cid:107) v (cid:107) )
7: else if greedy mode then 8: i ← argmax v 9:
S ← S ∪ { i } 10: return S
Exact methods that signiﬁcantly speed up sampling by leveraging speciﬁc structure in the D PP kernel have also been developed (Mariet & Sra, 2016b; Gartrell et al., 2017).
Of particular interest is the dual sampling method introduced in Kulesza & Taskar (2012): if the DPP kernel can be composed as an inner product over a ﬁnite basis, i.e. there exists a feature matrix Φ ∈ R N × D such that the D PP kernel is given by L = ΦΦ (cid:62) , exact sampling can be done in
O ( ND 2 + NDk 2 + D 2 k 3 ) .
However, MCMC sampling requires variable amounts of sampling rounds, which is unfavorable for parallelization; dual D PP sampling requires an explicit feature matrix Φ .
Motivated by recent work on modeling set functions with neural networks (Zaheer et al., 2017; Cotter et al., 2018), we propose instead to generate approximate samples via a generative network; this allows for simple parallelization while simultaneously beneﬁting from recent improvements in specialized architectures for neural network models ( e.g. parallelized matrix multiplications).
We furthermore show that, extending the abilities of dual D PP sampling, neural networks may take as input variable feature matrices Φ and sample from non-linear kernels L .
G ENERATING D PP SAMPLES WITH DEEP MODELS
In this section, we build up a framework that allows the O ( N 3 ) computational cost associated with
D PP sampling to be addressed via approximate sampling with a neural network.
Given a positive semi-deﬁnite matrix L ∈ R N × N , we take P L to represent the distribution modeled by a D PP with kernel L over the power set of [ N ] := { 1 , . . .
, N } .
Given A ⊆ [ N ] , we write ¯
A =
[ N ] \ A and L A =
[ L ij ] i,
j ∈
A the | A | × | A | submatrix of L indexed by items in A .
M OTIVATING THE MODEL CHOICE
Although the elegant quality/diversity tradeoff modeled by D PP s is a key reason for their recent success in many different applications, they beneﬁt from other properties that make them particularly well-suited to machine learning problems.
We now focus on how these properties can be maintained with a deep generative model with the right architecture.
Closure under conditioning.
D PP s are closed under conditioning: given A ⊆ Y , the conditional distribution over Y \
A given by P L ( S =
A ∪ B | A ⊆ S )
(for B ∩ A = ∅ ) is also D PP with kernel
L A =
([( L + I ¯ A ) − 1 ] ¯ A ) − 1 − I (Borodin & Rains, 2005).
Although conditioning comes at the cost of an expensive matrix inversion, this property make D PP s well-suited to applications requiring diversity in conditioned sets, such as basket completion for recommender systems.
Standard deep generative models such as (Variational) Auto-Encoders (Kingma & Welling, 2014) (VAEs) and Generative Adversarial Networks (Goodfellow et al., 2014) (GANs) would not enable simple conditioning operations during sampling.
Instead, we develop a model that given an input set
S , returns a prediction vector v ∈ R N such that v i =
Pr( S ∪ { i } ⊆
Y | S ⊆ Y ) where Y ∼ P L : in other words, v i is the marginal probability of item i being included in the ﬁnal set, given that S is a subset of the ﬁnal set.
Mathematically, we can compute v i as
[Equation]
) for i (cid:54)∈ S (Kulesza & Taskar, 2012); for i ∈ S , we simply set v i = 0 .
With this architecture, we sample a set via Algorithm 1, which allows for trivial basket-completion type conditioning operations.
Furthermore, Algorithm 1 can be modiﬁed to implement a greedy sampling algorithm without any additional cost.
Log-submodularity.
As mentioned above, D PP s are included in the larger class of Strongly
Rayleigh (SR) measures over subsets.
Although being SR is a delicate property, which is maintained by only few operations (Borcea et al., 2009), log-submodularity 3 (which is implied by SR-ness) is more robust, as well as a fundamental property in discrete optimization (Zagoruyko & Komodakis, 2016;
Buchbinder et al., 2012; Gr¨otschel et al., 1981) and machine learning (Iyer & Bilmes, 2015;
Wei et al., 2015).
Crucially, we show in the following that (log)-submodularity can be inherited by a generative model trained on a log-submodular distribution:
T HEOREM 1.
Let P be a strictly submodular function over subsets of Y , and Q be a function over the same space such that
[Equation]
where D TV indicates the total variational distance.
Then Q is also submodular.
C OROLLARY 1.1.
Let P L be a strictly log-submodular D PP over Y and D PP N ET be a network trained on the D PP probabilities p ( S ) , with a loss function of the form (cid:107) p − q (cid:107) where (cid:107) · (cid:107) is a norm and p ∈ R 2 N (resp.
q ) is the probability vector assigned by the D PP (resp.
the D PP N ET ) to each subset of Y .
Let α = max (cid:107) x (cid:107) ∞
=1 1 (cid:107) x (cid:107) .
If D PP N ET converges to a loss smaller than min
[Equation]
,T
[Equation]
its generative distribution is log-submodular.
The result follows directly from Thm.
3 and the equivalence of norms in ﬁnite dimensional spaces.
R EMARK 1.
Cor.
1.1 is generalizable to the KL divergence loss D KL ( P(cid:107)Q ) via Pinsker’s inequality.
For this reason, we train our models by minimizing the distance between the predicted and target probabilities, rather than optimizing the log-likelihood of generative samples under the true D PP .
Leveraging the sampling path.
When drawing samples from a D PP , the standard D PP sampling algorithm (Kulesza & Taskar, 2012, Alg. 1) generates the sample as a sequence, adding items one after the other until reaching a pre-determined size 4 , similarly to Alg.
We take advantage of this by recording all intermediary subsets generated by the D PP when sampling training data: in practice, instead of training on n subsets of size k , we train on kn subsets of size 0 , . . .
, k − 1 .
Thus, our model is very much like an unrolled recurrent neural network.
3. S AMPLING OVER FIXED AND VARYING GROUND SETS
In the simplest setting, we may wish to draw many samples over a ground set with a ﬁxed feature embedding.
In this case, we wish to model a D PP with a ﬁxed kernel via a generative neural network.
Speciﬁcally, we consider a ﬁxed D PP with kernel L and wish to obtain a generative model such that its distribution G : 2 Y → [0 , 1] is close (under a chosen metric) to the D PP distribution P L .
More generally, in many cases we may care about sampling from a D PP over a ground set of items that varies: this may be the case for example with a pool of products that are available for sale at a given time, or social media posts with a relevance that varies based on context.
To leverage the speed-up provided by dual D PP sampling, we can only sample from the D PP with kernel given by
L = ΦΦ (cid:62) ; for more complex kernels, we once again incur the O ( N 3 ) cost.
Furthermore, training a static neural network for each new feature embedding may be too costly.
Instead, we augment the static D PP N ET to include the feature matrix Φ representing the ground set of all items as input to the network.
Speciﬁcally, we draw inspiration for the dot-product attention introduced in (Vaswani et al., 2017).
In the original paper, the attention mechanism takes as input 3 matrices: the keys K , the values V , and the query Q .
Attention is computed as
A = softmax ( QK (cid:62) / √ d ) where d is the dimension of each query/key: the inner product acts as a proxy to the similarity between each query and each key.
Finally, the reweighted value matrix AV is fed to the trainable neural network.
[Figure]
S ⊆
Y
[Figure]
p l
[Figure]
Here, the feature representation of items in the input set S acts as the query Q ∈ R k × d ; the feature representation Φ ∈ R N × d of our ground set is both the keys and the values.
In order for the attention mechanism to make sense in the framework of D PP modeling, we make two modiﬁcations to the attention in (Vaswani et al., 2017):
• We want our network to attend to items that are dissimilar to the query (input subset): for each item i in the input subset S , we compute its pairwise dissimilarity to each item in Y as the vector d i = 1 − softmax ( Q i Φ (cid:62) / √ d ) .
• Instead of returning this k × N matrix D of dissimilarities
d i , we return a vector a ∈ R N in the probability simplex such that a j ∝ (cid:81) i ∈ S D ij .
This allows us to have a ﬁxed-size input to the neural network, and simultaneously enforces the desirable property that similarity to a single item is enough to disqualify an element from the ground set.
Note that we could also return D in the form of a N × N matrix, but this would be counterproductive to speeding up D PP sampling.
Putting everything together, our attention vector a is computed via the inhibitive attention mechanism
[Equation]
where (cid:12) represents the row-wise multiplication operator; this vector can be computed in O ( kDN ) time.
The attention component of the neural network ﬁnally feeds the element-wise multiplication of each row of V with a to the feed-forward component.
Given Φ and a subset S , the network is trained as in the static case to learn the marginal probabilities of adding any item in Y to S under a D PP with a kernel L dependent on Φ .
In practice, we set L to be an exponentiated quadratic kernel L ij = exp( − β (cid:107) φ i − φ j (cid:107) 2 ) constructed with the features φ i .
R EMARK 2.
Dual sampling for D PP s as introduced in (Kulesza & Taskar, 2012) is efﬁcient only when sampling from a D PP with kernel L = ΦΦ (cid:62) ; for non-linear kernels, a low-rank decomposition of L ( Φ ) must ﬁrst be obtained, which in the worst case requires O ( N 3 ) operations.
In comparison, the dynamic D PP N ET can be trained on any D PP kernel, while only requiring Φ as input.
[Table]
E XPERIMENTAL RESULTS
To evaluate D PP N ET , we look at its performance both as a proxy for a static D PP (Section 4.1) and as a tool for generating diverse subsets of varying ground sets (Section 4.2).
Our models are trained with TensorFlow, using the Adam optimizer.
Hyperparameters are tuned to maximize the normalized log-likelihood of generated subsets.
We compare D PP N ET to D PP performance as well as two additional baselines:
• U NIF :
Uniform sampling over the ground set.
• k -M EDOIDS :
The k -medoids clustering algorithm (Hastie et al., 2001, 14.3.10), applied to items in the ground set, with distance between points computed as the same distance metric used by the D PP .
Conversely to k -means, k -M ED uses data points as centers for each cluster.
We use the negative log-likelihood (NLL) of a subset under a D PP constructed over the ground set to evaluate the subsets obtained by all methods.
This choice is motivated by the following considerations: D PP s have become a standard way of measuring and enforcing diversity over subsets of data in machine learning, and b) to the extent of our knowledge, there is no other standard method to benchmark the diversity of a selected subset that depends on speciﬁc dataset encodings.
4. S AMPLING OVER THE UNIT SQUARE
We begin by analyzing the performance of a D PP N ET trained on a D PP with ﬁxed kernel over the unit square.
This is motivated by the need for diverse sampling methods on the unit hypercube, motivated by e.g. quasi-Monte Carlo methods, latin hypercube sampling (McKay et al., 1979) and low discrepancy sequences.
The ground set consists of the 100 points lying at the intersections of the 10 × 10 grid on the unit square.
The D PP is deﬁned by setting its kernel L to L ij = exp( −(cid:107) x
i − x j (cid:107) 22 / 2) .
As the D PP kernel is ﬁxed, these experiments exclude the inhibitive attention mechanism.
We report the performance of the different sampling methods in Figure 3.
Visually (Figure 3a) and quantitively (Figure 3b), D PP N ET improves signiﬁcantly over all other baselines.
The NLL of
D PP N ET samples is almost identical to that of true D PP samples.
Furthermore, greedily sampling the mode from the D PP N ET achieves a better NLL than D PP samples themselves.
Numerical results are reported in Table 1.
4. S AMPLING OVER VARIABLE GROUND SETS
We evaluate the performance of D PP N ET s on varying ground set sizes through the MNIST (LeCun & Cortes, 2010), CelebA (Liu et al., 2015), and MovieLens (Harper & Konstan, 2015) datasets.
For MNIST and CelebA, we generate feature representations of length 32 by training a Variational
Auto-Encoder (Kingma & Welling, 2014) on the dataset 5 ; for MovieLens, we obtain a feature vector for each movie by applying nonnegative matrix factorization the rating matrix, obtaining features of length 10.
Experimental results presented below were obtained using feature representations obtained via the test instances of each dataset.
The D PP N ET is trained based on samples from D PP s with a linear kernel for MovieLens and with an exponentiated quadratic kernel for the image datasets.
Bandwidths were set to β = 0 .
0025 for
MNIST and β = 0 .
1 for CelebA in order to obtain a D PP average sample size ≈ 20: recall that for a
D PP with kernel L , the expected sample size is given by the formula
[Equation]
[Figure]
NLL
[Figure] [Figure]
[Figure]
For MNIST, Figure 4 shows images selected by the baselines and the D PP N ET , chosen amongdigits with either random labels or all identical labels; visually, D PP N ET and D PP samples provide a wider coverage of writing styles.
However, the NLL of samples from D PP N ET decay signiﬁcantly, whereas the D PP N ET mode continues to maintain competitive performance with D PP samples.
Numerical results for MNIST are reported in Table 2; additionally to the previous baselines, we also consider two further ways of generating subsets.
I NHIB
A TTN samples items from the multinomial distribution generated by the inhibitive attention mechanism only (without the subsequent neural network).
N O A TTN is a pure feed-forward neural network without attention; after hyper-parameter tuning, we found that the best architecture for this model consisted in 6 layers of 585 neurons each.
Table 2 reveals that both the attention mechanism and the subsequent neural network are crucial to modeling D PP samples.
Strikingly, D PP N ET performs signiﬁcantly better than other baselines even on feature matrices drawn from a single class of digits (Table 2), despite the training distribution over feature matrices being much less specialized.
This implies that D PP N ET sampling for dataset summarization may be leveraged to focus on sub-areas of datasets that are identiﬁed as areas of interest.
Numerical results for CelebA and MovieLens are reported in Table 3, conﬁrming the modeling ability of D PP N ET s.
Finally, we verify that D PP N ET allows for signiﬁcantly faster sampling by running D PP and D PP N ET sampling for subsets of size 20 drawn from a ground set of size 100 with both a standard D PP
[Table] [Table]
[Figure]
and D ET ( architecture). Both methods were implemented in graph-mode
TensorFlow. Sampling batches of size 32, standard D PP sampling costs 2.74 ± 0.02 seconds;
4. D PP N ET FOR KERNEL RECONSTRUCTION
As a ﬁnal experiment, we evaluate D PP N ET ’s performance on a downstream task for which D PP s have been shown to be useful: kernel reconstruction using the Nystr¨om method (Nystr¨om, 1930; Williams & Seeger, 2001).
Given a positive semideﬁnite matrix K ∈ R N × N , the Nystr¨om approximates K by ˆ
K = K · ,S K † S,S K S, · where K † denotes the pseudoinverse of K and K · ,S (resp.
K S, · ) is the submatrix of K formed by its rows (resp. columns) indexed by S .
The Nystr¨om method is a popular method to scale up kernel methods and has found many applications in machine learning (see e.g. (Bac; She; Fow; Tal)).
Importantly, the approximation quality directly depends on the choice of subset S .
Recently, D PP s have been shown to be a competitive approach for selecting S (Mariet et al., 2018;
Li et al., 2016b).
Following the approach of Li et al.
(2016b), we evaluate the quality of the kernel reconstruction by learning a regression kernel K on a training set, and reporting the prediction error on the test set using the Nystr ¨ om reconstructed kernel ˆ K .
Additionally to the full D PP , we also
Figure 5 reports our results on the Ailerons dataset 6 ; Figure 5b reports the runtimes for each method.
C ONCLUSION AND DISCUSSION
We introduced D PP N ET s, generative networks trained on D PP s over static and varying ground sets which enable fast and modular sampling in a wide variety of scenarios.
We showed experimentally on several datasets and standard D PP applications that D PP N ET s obtain competitive performance as evaluated in terms of NLLs, while being amenable to the extensive recent advances in speeding up computation for neural network architectures.
Although we trained our models on D PP s on exponentiated quadratic and linear kernels; we can train on any kernel type built from a feature representations of the dataset.
This is not the case for dual
D PP exact sampling, which requires that the D PP kernel be L = ΦΦ (cid:62) for faster sampling.
D PP N ET s are not exchangeable: that is, two sequences i 1 , . . .
, i k and σ ( i 1 ) , . . .
, σ ( i k ) where σ is a permutation of [ k ] , which represent the same set of items, will not in general have the same probability under a D PP N ET .
Exchangeability can be enforced by leveraging previous work (Zaheer et al., 2017); however, non-exchangeability can be an asset when sampling a ranking of items.
Our models are trained to take as input a ﬁxed-size subset representation; we aim to investigate the ability to take a variable-length encoding as input as future work.
The scaling of the D PP N ET ’s complexity with the ground set size also remains an open question.
However, standard tricks to enforce ﬁxed-size ground sets such as sub-sampling from the dataset may be applied to D PP N ET s.
Similarly, if further speedups are necessary, sub-sampling from the ground set – a standard approach for D PP sampling over very large set sizes – can be combined with D PP N ET sampling.
In light of our results on dataset sampling, the question of whether encoders can be trained to produce encodings conducive to dataset summarization via D PP N ET s seems of particular interest.
Assuming knowledge of the (encoding-independent) relative diversity of a large quantity of subsets, an end-to-end training of the encoder and the D PP N ET simultaneously may yield interesting results.
Finally, although Corollary 1.1 shows the log-submodularity of the D PP can be transferred to a generative model, understanding which additional properties of training distributions may be conserved through careful training remains an open question which we believe to be of high signiﬁcance to the machine learning community in general.
A M AINTAINING LOG - SUBMODULARITY IN THE GENERATIVE MODEL
T HEOREM 2.
Let p be a strictly submodular distribution over subsets of a ground set Y , and q be a distribution over the same space such that
[Equation]
Then q is also submodular.
Proof.
In all the following, we assume that S, T are subsets of a ground set Y such that S (cid:54) =
T and
S, T (cid:54)∈ {∅ , Y} (the inequalities being immediate in these corner cases).
Let
[Equation]
By the strict submodularity hypothesis, we know (cid:15) > 0 .
Let S, T ⊆
Y such that S (cid:54) =
T and S, T (cid:54) = ∅ , Y .
To show the log-submodularity of q , it sufﬁces to show that
[Equation]
By deﬁnition of (cid:15) ,
[Equation]
From equation 5, we know that
[Equation]
It follows that
[Equation]
which proves the submodularity of q .
B E NCODER DETAILS
For the MNIST encodings, the VAE encoder consists of a 2d-convolutional layer with 64 ﬁlters of height and width 4 and strides of 2, followed by a 2d convolution layer with 128 ﬁlters (same height, width and strides), then by a dense layer of 1024 neurons.
The encodings are of length