Sequence-to-Set Generative Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In this paper, we propose a sequence-to-set method that can transform any sequencegenerative model based on maximum likelihood to a set generative model wherewe can evaluate the utility/probability of any set.
An efﬁcient importance samplingalgorithm is devised to tackle the computational challenge of learning our sequenceto-set model.
We present GRU2Set, which is an instance of our sequence-to-setmethod and employs the famous GRU model as the sequence generative model.
Tofurther obtain permutation invariant representation of sets, we devise the SetNNmodel which is also an instance of the sequence-to-set model.
A direct applicationof our models is to learn an order/set distribution from a collection of e-commerceorders, which is an essential step in many important operational decisions such asinventory arrangement for fast delivery.
Based on the intuition that small-sizedsets are usually easier to learn than large sets, we propose a size-bias trick that canhelp learn better set distributions with respect to the ` 1 -distance evaluation metric.
Two e-commerce order datasets, TMALL and HKTVMALL, are used to conductextensive experiments to show the effectiveness of our models.
The experimentalresults demonstrate that our models can learn better set/order distributions fromorder data than the baselines.
Moreover, no matter what model we use, applyingthe size-bias trick can always improve the quality of the set distribution learnedfrom data.
Introduction
Learning a generative model [Goodfellow et al., 2014, Kingma and Welling, 2014] from data togenerate random samples is a fundamental unsupervised learning task.
In many real applications, thedata, such as customer orders and drug packages, can be represented by sets.
Learning a set generativemodel can boost many important applications such as simulating users’ demand of e-commerceorders for inventory arrangement [Wu et al., 2021].
However, in literature, little attention has beenpaid to generative models for sets.
In this paper we study the problem of learning a set generative model from data.
Speciﬁcally, westudy sets containing some categorical items from a ﬁnite ground set.
Some previous studies [Bensonet al., 2018, Wu et al., 2021] proposed traditional statistical models to learn the set generative model,while how to employ deep learning to design set generative models remains largely untouched.
Meanwhile, deep generative models for sequence data [Hochreiter and Schmidhuber, 1997, Schäfer and Zimmermann, 2006, Cho et al., 2014, Devlin et al., 2018] have been extensively studied.
Sincethe only difference between a sequence and a set is that the order of items does not matter in a set, wepropose to leverage a sequence generative model to build a set generative model.
A natural idea of converting a random sequence to a random set is to just ignore the order of items.
However, for a set, the number of possible sequences that can be regarded as equivalent to the setis exponential to the number of items.
This brings us a serious computational challenge where we need to enumerate all the equivalent sequences for a set when learning the parameters of the model.
Traditional statistical models [Benson et al., 2018, Wu et al., 2021] along this line either makeunrealistic independent assumptions on items or run a learning algorithm of complexity exponentialto set sizes to only deal with small sets.
These constrained treatments undermine the model’s abilityto capture the complex correlations among items in sets.
To tackle the computational challenge while still well capturing the correlations among items, in thispaper, we make the following technical contributions.
In Section 3, we ﬁrst show that we can convert any sequence generative model based on maximumlikelihood, such as RNN [Schäfer and Zimmermann, 2006] and DeepDiffuse [Islam et al., 2018], to aset generative model.
We call our method Sequence-to-Set.
To address the computational challenge,we propose an efﬁcient importance sampling algorithm to estimate the gradient of the log-likelihoodfunction of our Sequence-to-Set model.
In Section 4, to capture the complex item correlations, we propose two deep models GRU2Set and
SetNN, which are instances of our Sequence-to-Set model.
GRU2Set directly uses the GRU as anessential building block, while SetNN is designed for obtaining permutation invariant set embeddingswhen learning a set generative model.
We discuss that GRU2Set and SetNN can express any setdistributions as long as we increase the model capacity.
To learn a better set distribution from data, in Section 5, we reveal that we often need to modify theempirical size distribution of the training data and propose a heuristic to do so.
This size-bias trickcan be applied to all set generative models aiming at learning a set distribution.
We conduct extensive experiments using two e-commerce datasets TMALL and HKTVMALL, andreport the experimental results in Section 6.
The experimental results clearly show the superiority ofour models to the baselines and the effectiveness of our size-bias trick.
Related Work
Learning a set generative model is often regarded as learning a choice model for subset selection.
Along this line, Benson et al.
[2018] proposed a discrete choice model (DCM) to model the utility/probability that users select a speciﬁc set.
Wu et al.
[2021] utilized representation learning todesign a parameterized Markov Random Walk to model how users select random sets from an itemgraph.
Both [Benson et al., 2018] and [Wu et al., 2021] are set generative models, though they arenot deep models.
To achieve practical computational efﬁciency, both [Benson et al., 2018] and [Wuet al., 2021] only handle small sets and compromise on modeling high-order item correlations.
To model the permutation invariant nature of sets, Ruiz et al.
[2017] used the sum or max of itemembedding as the set embedding.
Zaheer et al.
[2017] systematically investigated the theory ofpermutation invariant functions.
Stelzner et al.
[2020] proposed a generative model for point cloudsbased on GAN [Goodfellow et al., 2014] and Set Transformer [Lee et al., 2019].
Although a pointcloud is a set of 2D-coordinates, we cannot use the point cloud generative model [Stelzner et al.
,2020] to solve our problem as the model outputs continuous random variables instead of categoricaldata.
Kosiorek et al.
[2020] presented another generative model for sets of real-valued vectors basedon Set Transformer [Lee et al., 2019] and their generating process is based on VAE [Kingma and
Welling, 2014].
A disadvantage of using GAN or VAE to train a generative model is that it would bedifﬁcult to evaluate the probability density of a random sample.
Normalizing Flows are powerfulgenerative models based on solid statistical theories.
However, the study of Discrete Flows is still inits infancy
[Hoogeboom et al., 2019, Tran et al., 2019] and no existing works have investigated using
Discrete Flows to learn set distributions.
Multi-label classiﬁcation also needs to model the permutation invariance of sets of labels.
Yang et al.[2019] applied reinforcement learning with a permutation-invariant reward function to remove theimpact of label orders.
Yazici et al.
[2020] proposed to align the labels of a sample with its predictedlabels before calculating the loss of the sample.
However, our task in this paper is an unsupervisedlearning task.
We cannot apply the loss functions in [Yang et al., 2019, Yazici et al., 2020] since wedo not have “labels” (or feature vectors if one regards a set as a set of labels) for training samples.
To sum up, existing works on set generative models either deal with sets of real-valued vectors whichare different from categorical sets, or adopt non-deep models which need to compromise on capturingitem correlations to make the computation efﬁcient.
It is still urgent to design deep generative modelsfor categorical sets and propose efﬁcient algorithms to deal with computational challenges.
Sequence-to-Set Generative Models
In this section, we introduce how to convert a sequence generative model to a set generative modeland how to optimize the model parameters.
3.1 Converting Sequences to Sets
We ﬁrst characterize the sequence generative model that we want to convert to a set generative model.
Deﬁnition 1 (Sequence Generative Model)
A sequence generative model is a model with a sequential generating process, which can be represented by a sequence of states and actions[ s 0 , a 0 , s 1 , a 1 , ..., s T  1 , a T  1 , s T ] , where s i is the state at time i , a i is the action taken at timei and s T is the end state.
We call [ s 0 , a 0 , s 1 , a 1 , ..., s T  1 , a T  1 , s T ] a generating path .
Like aﬁnite-state machine, the sequence generative model uses each action a i to deterministically updatethe current state s i to s i +1 .
For each state s , the sequence generative model has a correspondingprobability distribution p ( a | s ; ✓ ) indicating the probability of choosing action a when the currentstate is s , where ✓ is the parameters of the sequence generative model.
Our deﬁnition of the sequence generative model can cover a wide range of models that can generateitems in a sequential manner.
For example, the commonly used sequence model RNN is an instance,where the state is the history vector H and an action is to choose the next word/item.
Moreover,inﬂuence diffusion models such as the Independent Cascade (IC) model [Goldenberg et al., 2001] andthe DeepDiffuse model [Islam et al., 2018] are also covered by Deﬁnition 1, since in these modelsthe state at time t is the set of activated nodes until time t and the action a t is to select the group ofnodes to be activated at time t + 1 .
To convert a sequence generative model in Deﬁnition 1 to a set generative model, we need to associateeach state s with a state-associated set S s .
For example, in an RNN, we can deﬁne S s t as the set ofitems generated in the ﬁrst t steps.
In the IC model, we can deﬁne S s t as the set of nodes activated ator before time t .
We say that a generating path l
= [ s 0 , a 0 , . . .
, s T  1 , a T  1 , s T ] can induce a set S if S = S s T .
Let L ( S ) =
{ l | l can induce S } be the path set of S .
As the sequence generative model has theprobability distribution p ( a | s ; ✓ ) indicating the probability of taking a as the next action given thecurrent state s , we can derive the probability of having a path l
= [ s 0 , a 0 , . . .
, s T  1 , a T  1 , s T ] as
[Equation]
Therefore, given a sequence generative model with parameters ✓ , the probability to induce a set S is
[Equation]
3.2 Parameter Learning
Suppose we have a collection (multiset) of observed sets S = { S 1 , S 2 , ..., S N } .
We want to use S asthe training data to learn a sequence model which can generate the observed sets.
We adopt maximumlikelihood estimation to learn the parameters ✓ .
Speciﬁcally, the loss function is
[Equation]
We apply stochastic gradient descent based algorithms to minimize the loss function.
The stochasticgradient w.r.t.
an observation S is r log p ( S ; ✓ ) =
r log ⇣ P l [Equation] L ( [Equation]
p ( l [Equation] ✓ ) ⌘ .
As in the log-sum termof p ( S ; ✓ ) , the number of paths that can induce S is often exponential to | S | (in some sequencegenerative model the number of such paths is even inﬁnite), it is very challenging to evaluater log p ( S ; ✓ ) exactly.
To tackle the computational challenging in computing r log p ( S ; ✓ ) , we ﬁrst ﬁnd that r log p ( S ; ✓ )can be regarded as an expectation as follows.
[Equation]
where p ( l | S ; ✓ ) = p ( l ; ✓ ) p ( S ; ✓ ) is a posterior distribution indicating the probability that a given set S isgenerated by the path l .
Therefore, r log p ( S ; ✓ ) can be seen as the expected value of p ( l ; ✓ ) wherethe random variable l follows the posterior distribution p ( l | S ; ✓ ) .
We can apply Monte Carlo method to estimate r log p ( S ; ✓ ) =
E l ⇠ p ( l | S ; ✓ ) [ r log p ( l ; ✓ )] , where thekey point is to sample generating paths from p ( l | S ; ✓ ) .
A naive idea is to use rejection sampling,where we use the model with parameters ✓ to generate random paths and reject those that cannotinduce S .
Obviously such a naive implementation would be extremely inefﬁcient as the rejection ratewould be very high.
To sample l from p ( l | S ) more efﬁciently, we devise an important sampling method as follows.
Let ⌧ ( a, s, S ) be an indicator function, where ⌧ ( a, s, S )
=
if 9 l = [ s 0 , a 0 , . .
, s T  1 , a T  1 , s T ] such that l induces S and 9 i 2 { 0 , 1 , . .
, T  1 } , s i
= s ^ a i = a .
In our sampling algorithm, for a state s t , we only consider actions a t such that ⌧ ( a t , s, S ) = 1 .
We take such an action a t with probability
[Equation]
Therefore, the proposal distribution q ( l ; ✓ , S ) for l = [ s 0 , a 0 , . . .
, s T  1 , a T  1 , s T ] in our importancesampling is
[Equation]
1 ; ✓ , S )
The (unnormalized) importance weight of l isr ( l ; ✓ ) =
[Equation]
✓ [Equation] A Clearly, we haver ( l ; ✓ ) q ( l ; ✓ , S ) / p ( a 0 | s 0 ; ✓
[Equation]
p ( a
[Equation]
| s
[Equation]
; ✓ [Equation] ...p [Equation]
[Equation] s T  1 ; ✓ [Equation] = p ( l
[Equation]
✓ ) .
This shows the correctness of the sampling method.
An EM perspective Using our importance sampling algorithm to estimate r log p ( S ; ✓ ) in minimizing the loss function L ( ✓ ) can be interpreted from an Expectation-Maximization algorithm’sperspective.
Our method is actually a Monte Carlo EM algorithm if we treat a generating path las a latent variable.
In the E-step, we sample the latent variable l from the posterior distributionp ( l | S ; ✓ old ) .
Our importance sampling can boost the efﬁciency of the E-step.
In the M-step, wereconstruct an object function which is
P Mi r ( l i ) R  1 log p ( l i ; ✓ old ) , where M is the number ofsamples and R = P Mi =1 r ( l i ) .
To maximize this function, we update ✓ by gradient based methods, such as ✓ new = ✓ old + ⌘
P Mi =1 r ( l i ) R  1 r ✓ log p ( l i ; ✓ old ) , where ⌘ is the learning rate and
P
Mi =1 r ( l i ) R  1 r ✓ log p ( l i ; ✓ old ) is estimated by our importance sampling algorithm.
MRW as an instance of Sequence2Set We show that the Markov Random Walk(MRW)
model[Wu et al., 2021] is also an instance of our Sequence2Set model.
The sequence model in MRW isa random walk on the item graph [Wu et al., 2021] where a node is an item and we have a specialstop item  .
The state s t at time t is associated with a set S s t which includes all the items visiteduntil time t .
An action a t taken at time t given the state s t is to jump to an item i t from i t  1 , the itemvisited at time t  1 .
Once the stop item  is visited, the random walk terminates.
The probabilityof jumping from i t  1 to i t
= i is parameterized as p ( i t
= i | s t  1 ; ✓ ) , where ✓ is the embeddingsof items.
Speciﬁcally, in MRW, p ( i t = i | s t  1 ; ✓ ) =
exp( e > i e it  1 ) P j exp( e > j e it  1 ) , where e i is the embeddingvector of item i .
When an action a is jumping to an item i 6 =  , the indicator ⌧ ( a, s, S )
= {+
+} if i 2 S .
When an action a is jumping to the stop item  , ⌧ ( a, s, S )
=
if S s = S .
Wu et al.
[2021] proposean algorithm of complexity exponential to | S | to calculate the gradient r log p ( S ; ✓ ) for learning theitem embeddings and thus, only small sets of size at most 4 are considered in [Wu et al., 2021].
If weadopt our important sampling algorithm, we can deal with large sets.
GRU2Set and SetNN
In this section, we propose two deep Sequence-to-Set models GRU2Set and Set Neural Networks(SetNN).
We ﬁrst introduce some general settings that will be used in both GRU2Set and SetNN.
Item embedding and stop item We also adopt item embeddings, which should be learned from thetraining dataset, to parameterize the probability p ( a | s ; ✓ ) in the sequence generative model.
Weassign an embedding vector e i 2 R d to each item i .
Similar to MRW [Wu et al., 2021], we also add astop item  where once we add  to the set associated to the current state, we stop the generatingprocess.
The stop item  also has an embedding vector e  2 R d .
Sparse item graph In our Sequence-to-Set model, the indicator function⌧ ( a, s, S ) is a key for us to efﬁciently sample generating paths fromthe posterior distribution p ( l | S ; ✓ ) .
Intuitively, ⌧ ( a, s, S ) indicateswhat actions are valid given the current state s and the induced set S .
To further boost the efﬁciency of our importance sampling algorithm,we make extra constraints on the items that can be added in the nextstep when we are given the current state s .
To do so, we build an itemgraph G = h V, E i , where V is the set of items plus the stop item and ( i, j ) 2 E if there is a set S in the training dataset S such thati 2 S and j 2 S .
The stop item is connected to all the other items.
When using our importance sampling algorithm, given the current states associated with a set S s ✓ S , ⌧ ( a, s, S )
= 1 if action a is adding anitem i 2 S \ Neighbor ( S s ) , where Neighbor ( S s ) is the set of neighborsto items in S s .
When starting from the ﬁrst state s 0 associated with a set
S s 0 = ; , we set Neighbor ( S s ) =
V \{  } .
In real set datasets such as TMALL and HKTVMALL used 189 in our experiments, such an item graph built from training datasets is often very sparse.
Therefore,we call the item graph the sparse item graph .
Clearly, the sparsity of the item graph can help boostthe efﬁciency of generating random sets from a learned model.
Choice vector
In our model design, the conditional probability p ( a | s ; ✓ ) is the key to control thegenerating process.
Exploiting the idea of representation learning, we represent each state s with anembedding vector c s 2 R d .
We call c
s the choice vector of s .
Given s , we control the probability ofadding i as the next item as
[Equation]
The sparsity of the item graph can make computing the normalization term in p ( i | s ; ✓ ) efﬁcient.
Note that in MRW [Wu et al., 2021], the choice vector c s is the embedding of the last item visited sofar.
We will see that in GRU2Set and SetNN, we use deep neural nets to aggregate the embeddings ofall the items visited so far to obtain the choice vector c s .
The initial choice vector c s 0 is trainable.
4.1 GRU2Set
We leverage GRU [Cho et al., 2014] to build our ﬁrst sequence-to-set model, since GRU is one of themost popular sequential model based on RNN and its computational cost is low.
We set the historyvector H of GRU as a d -dimensional vector and H is regarded as the choice vector of the currentstate.
The initial history vector H 0 is trainable and H is updated according to the design of GRU.
The state-associated set S s t for the state s t at time t is just the set of items added in the ﬁrst t steps ofthe sequence generating process of GRU.
When the stop item is added, we terminate the generatingprocess.
Due to limited space, the overview of GRU2Set is put in Appendix.
4.2 SetNN
The choice vector c s can be regarded as the embedding of the set S s .
In GRU2Set, the choice vector,which is the history vector H , is sensitive to the order of items added to S s .
As sets are permutationinvariant, we hope to have a sequence-to-set model which can also produce permutation invariantrepresentations of sets.
Therefore, we develop the Set Neural Networks (SetNN) where the key idea isto design a permutation invariant aggregation of embeddings of items in a set.
The following theoremby Zaheer et al.
[2017] guides our model design.
Theorem 1 ([Zaheer et al., 2017])
A function f ( X ) operating on a set X having elements from acountable universe, is a set function, i.e., invariant to the permutation of instances in X , iff it can bedecomposed in the form ⇢ ( P x 2 X  ( x )) for suitable transformations  and ⇢ .
( x ) can be viewed as the embedding of x and the sum over  ( x ) can be replaced by other aggregateoperators such as mean or pooling.
In SetNN, we use an MLP to express the function ⇢ in Theorem 1to obtain the embedding (choice vector)
c s of S s for any state s .
Fig 2 shows the overview of SetNN.
4.3 Expressive Power of GRU2Set and SetNN
Compared to SetNN, our GRU2Set model uses the history vector H .
By the expressive power of
RNN [Schäfer and Zimmermann, 2006], as we increase the embedding dimension d , hidden neuralunits and layers in GRU module, the history vector H of GRU2Set can approximate any functionon a ﬁnite history path arbitrarily close.
Then, of course, H can express any permutation invariantfunction on sets which implies that the expressive power of GRU2Set is more than SetNN.
To explore the expressive power of SetNN, we ﬁrst give a recursive deﬁnition of the probability that aset S is generated by SetNN, denoted by p ( S ) , as follows.
[Equation]
) where p ( x | S \ { x } ) is the conditional probability of adding item x to the set S \ { x } and  ( S ) is 230 the probability of reaching a state s such that S s = S in SetNN.
The following theorem shows that
Eq.
(1) is general enough to cover all possible set distributions.
Theorem 2 For any distribution Q on all subsets of a ground set of items, there exists a group oftransition probabilities p ( x | S ) and p (  | S ) , such that q ( S )
= p ( S ) hold for any S , where p ( S ) isdeﬁned in Eq.
The proof of Theorem 2 can be found in Appendix B. As we increase the capacity of the MLPin SetNN, SetNN can represent any conditional probability p ( x | S ) and p (  | S ) , and as a result of
Theorem 2, SetNN and GRU2Set can express any set distributions.
Size-Bias Trick for Improving Learned Set Distributions
In this section, we propose a size-bias trick that can help reduce the distance between the ground truth set distribution p ⇤ ( S ) and the learned distribution p ( S ) , where p ( S ) can be learned by any generative model not limited to our Sequence-to-Set models.
We ﬁrst decompose the ground truth distribution p ⇤ ( S ) when | S | =
k as
[Equation]
where p ⇤ k = P A : | A | = k p ( A ) is the probability of generating a set of size k , and p ⇤ ( k ) ( S )
=
p ( S ) p ⇤ kis the probability that a random size- k set is S .
Suppose we know the real size distribution p ⇤ kand we have q ( k ) ( S ) as an estimation of p ⇤ ( k ) ( S ) for each k .
How can we construct a distributionq ( S ) close to p ⇤ ( S ) ?
An intuitive way is to combine q ( k ) ( S ) and p ⇤ k such that q ( S ) =
q ( k ) ( S ) p ⇤ k
if| S
| =
k
[Stelzner et al., 2020, Benson et al., 2018].
However, we argue that this intuitive way is notalways the optimal plan as follows.
Let K be the largest possible set size.
To ﬁnd the best size distribution q k for our estimation q ( k ) , weconsider the following optimization aiming at minimizing the KL-divergence between the distributionq ( S ) =
q ( k ) ( S ) q k and the ground truth p ⇤ ( S ) .
[Equation]
Since we do not know p ⇤ ( S ) , we cannot solve Eq.
(2) exactly.
However, we still can tell if settingq k = p ⇤ k is the optimal solution to some extend.
Let f k ( q k ) =
P S : | S | = k q ( S ) log q ( S ) p ⇤ ( S ) .
We have KL ( q || p ⇤ ) =
P k f k ( q k ) .
Note that f k ( q k ) onlydepends on q k .
We can interpret Eq.
(2) as a portfolio optimization problem.
There are K projects, q kis the investment for project k and  f k ( q k ) is the revenue of project k .
To decide the best portfolioq k , we check the derivative of f k ( q k ) , which indicates the marginal revenue.
We have
[Equation]
By taking q k = p ⇤ k , we have
[Equation]
As the estimation quality of q ( k ) ( S ) often varies for different k , the derivative f 0 k ( p ⇤ k ) probably alsovaries for different k , making q k = p ⇤ k not a stationary point.
Therefore, we probably need to set q kdifferent from p ⇤ k for solving Eq.
We ﬁrst make the following reasonable assumption.
Assumption 1 Small-sized sets are easier to learn than large sets.
In other words, q ( k ) ( S ) estimatesp ⇤ ( k ) ( S ) better as the set size k decreases.
Based on Assumption 1, compared to p ⇤ k , we should “invest” more on small set sizes to construct the“portfolio” (size distribution) q k .
Therefore, we set q k slightly bigger than p ⇤ k for small k , and set q ksmaller than p ⇤ k for big k .
To address the issue of not knowing p ⇤ k , we can resort to the empirical sizedistribution of the training dataset, which is a good estimation of the real size distribution p ⇤ .
Weconstruct a biased size distribution q k by adopting the following heurisitc.
Constructing Biased Size Distribution Suppose the empirical size distribution of the training data
S is [ p 1 , p 2 , ..., p K ] .
We ﬁrst calculate the rest proportion by r k = p k / ( p k + p k +
+ ... + p K ) ,which means p i =
(1  r 1 ) ...
(1  r i  1 ) r i .
Then we calculate r 0 k = max { r k + k p | V | / |S| , 1 } , 268 where V is the set of all items.
We construct a biased size distribution [ q 1 , q 2 , . . .
, q K ] by settingq i = (1  r 0 1 ) ...
(1  r 0 i  1 ) r 0 i .
Experiments
Datasets We did our empirical study on two rea-world datasets which are both the orders formonline e-commerce platforms.
This ﬁrst dataset is TMALL ( https://www.tmall.com ) dataset
[Table] [Table]
which contains 1363 items and orders from August 2018 to March 2019.
We treated the collection oforders in each month as an instance and in total we have 8 instances for the TMALL dataset.
Thesecond dataset is HKTVMALL ( https://www.hktvmall.com ) which contains 728 items form thesupermarket sector and orders from February 2020 to September 2020.
Similar to TMALL, we alsotreated HKTVMALL as 8 instances where each instance is the collection of orders in each month.
For both dataset, We split all the orders in a month to a training dataset S train and a testing dataset
S test , where the size of S train is 100,000 for Tmall and 200,000 for HKTVmall.
The datasets canbe found in the source code.
We report the sparsity of item graphs and statistics of the datasets in
Table 1.
The set size distribution of each dataset is shown in Table 2.
Experiment Setting and Evaluation We treat the method of directly using the histogram of S trainas the benchmark method, as it is a natural and unbiased method to estimate the set distribution.
Notethat this benchmark method has a major disadvantage that its cannot model probabilities of any setsnot showing in S train .
We also choose two methods to compare which are DCM (Discrete Choice
Model)
[Benson et al., 2018] and MRW [Wu et al., 2021].
Following [Wu et al., 2021], for each method (except the baseline), we generate a collection S predof 10,000,000 random sets and use the empirical set distribution of S pred to approximate the setdistribution learned by the model.
We regard the empirical distribution of the testing data S test asthe pseudo ground truth.
Since the bigger S test is, the more accurate the pseudo ground truth is,we set the size of S test bigger than S train as shown in Table 1.
Denote by N test ( S ) the numberof sets that are S in S test .
Let N pred ( S ) be the number of sets that are S in S pred .
We use the` 1 -distance between a model’s (approximate) distribution and the pseudo ground truth to evaluate theeffectiveness of the model.
Speciﬁcally, the ` 1 -distance can be calculated as follows.
[Equation]
We set the embedding dimension of all methods as 10.
For each training and testing dataset, the sizebias distribution is the same for all methods and is obtained by the heuristic introduced at the endof Section 5.
For all experiments, the MLP of SetNN only has one hidden layer which contains 50neural units.
All the experiments are ran on a 10 cores CPU.
The optimizer we use is RMSProp withdefault parameter in PyTorch.
The source code and data used in our experiments can be found athttps://github.com/AnonymousNeurIPS2022/SetLearning .
Experiment Results Due to limited space, we only report the main experimental results.
Moredetailed experimental results can be found in Appendix D and E.
Table 3 and Table 4 show the experimental results.
We place the results of using size-bias trick belowthe original results.
We ﬁnd that both DCM and MRW never beat the benchmark method, no matterthe size-bias trick is played or not.
A possible reason is that in our experiments we do not makesize constraints on sets, while Wu et al.
[2021] constrained the set size to be at most 4 and Bensonet al.
[2018] set the maximum size of sets to be 5.
Big sets in our data may have negative effects onlearning both DCM and MRW.
We can see that applying the size-bias trick can always reduce the ` 1 -distance signiﬁcantly, whichdemonstrates the effectiveness of this trick.
Before applying the size-bias trick, the benchmarkmethod has the best performance which is slightly better than our models GRU2Set and SetNN.
[Table] [Table]
However, after applying the size-bias trick
[Table]
our GRU2Set model becomes the best and it outperformsthe benchmark method by 4% on TMALL and 2.5% on HKTVMALL.
SetNN also outperforms thebenchmark method after using the size-bias trick.
This suggests that GRU2Set and SetNN may learnprobabilities of small sets better than the benchmark method.
The SetNN model performs slightly worse than GRU2Set
but the gap between them is small.
Apossible reason is that GRU2Set has stronger expressive power than SetNN as illustrated in Section 4.3.
GRU2Set records the order of items added to the set, while SetNN ignores such order to achievepermutation invariant set embeddings.
This shows that GRU2Set includes more information in thegenerating process than SetNN.
However, we want to emphasize that SetNN can produce permutationinvariant set embeddings while GRU2Set cannot.
So if we have a downstream task that needs setembeddings, SetNN may have more advantages in this case.
Conclusion
In this paper, we present a Sequence-to-Set method to build generative models for set data andthe SOTA method MRW is an instance of our method.
To utilize deep learning in learning setdistributions, we further design two models, GRU2Set and SetNN, which are two instances of our
Sequence-to-Set model.
To learn better set distributions from data, we also propose a size-bias trick.
Experimental results on two e-commerce order datasets clearly show that our models outperform thebaselines including two SOTA methods in this line of research.
For future work, we will explore using more sophisticated deep learning modules in more, such asreplacing the MLP in our SetNN model with Set Transformer and substituting the RNN in GRU withmore advanced sequence models.
For potential negative societal impacts, learning the embeddings ofitems and sets might cause user privacy leakage as many other embeddings might do.
Research onhow to hide personal information on set data could help protect users’ privacy.
Checklist
For all authors...
(a) Do the main claims made in the abstract
and
introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] It was described in Conclusion.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss
the privacy risk.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] All the proofs can befound in the supplemental material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
[Table] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The standard deviation among 8 groups and detail resultscan be ﬁnd in Table 3 and Table 4 in the supplemental material. [Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] All experiments were ran on aCPU of 10 cores.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use the MRW’s
code to compare their results and we cite it.
[Table]
(b) Did you mention the license of the assets?
[Yes] the code license can be ﬁnd in Github
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]In the source code
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[Yes] The platforms (Tmall and HKTVMall) where we collect our datahave already obtained such consent. [Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[Yes] The datasets we use are already anonymized.5.
If you used crowdsourcing or conducted research with human subjects....
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
[-Let V = V ′ ∪ { v m +
-]
[Table]
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]
[-o 1 + o 
-]
[-+ o 3 + o 
-]