Sequence-to-Set Generative Models
Abstract
In this paper, we propose a sequence-to-set method that can transform any sequence generative model based on maximum likelihood to a set generative model where we can evaluate the utility/probability of any set.
An efﬁcient importance sampling algorithm is devised to tackle the computational challenge of learning our sequenceto-set model.
We present GRU2Set, which is an instance of our sequence-to-set method and employs the famous GRU model as the sequence generative model.
To further obtain permutation invariant representation of sets, we devise the SetNN model which is also an instance of the sequence-to-set model.
A direct application of our models is to learn an order/set distribution from a collection of e-commerce orders, which is an essential step in many important operational decisions such as inventory arrangement for fast delivery.
Based on the intuition that small-sized sets are usually easier to learn than large sets, we propose a size-bias trick that can help learn better set distributions with respect to the ` 1 -distance evaluation metric.
Two e-commerce order datasets, TMALL and HKTVMALL, are used to conduct extensive experiments to show the effectiveness of our models.
The experimental results demonstrate that our models can learn better set/order distributions from order data than the baselines.
Moreover, no matter what model we use, applying the size-bias trick can always improve the quality of the set distribution learned from data.
Introduction
Learning a generative model [Goodfellow et al., 2014, Kingma and Welling, 2014] from data to generate random samples is a fundamental unsupervised learning task.
In many real applications, the data, such as customer orders and drug packages, can be represented by sets.
Learning a set generative model can boost many important applications such as simulating users’ demand of e-commerce orders for inventory arrangement [Wu et al., 2021].
However, in literature, little attention has been paid to generative models for sets.
In this paper, we study the problem of learning a set generative model from data.
Speciﬁcally, we study sets containing some categorical items from a ﬁnite ground set.
Some previous studies [Benson et al., 2018, Wu et al., 2021] proposed traditional statistical models to learn the set generative model, while how to employ deep learning to design set generative models remains largely untouched.
Meanwhile, deep generative models for sequence data [Hochreiter and Schmidhuber, 1997, Schäfer and Zimmermann, 2006, Cho et al., 2014, Devlin et al., 2018] have been extensively studied.
Since the only difference between a sequence and a set is that the order of items does not matter in a set, we propose to leverage a sequence generative model to build a set generative model.
A natural idea of converting a random sequence to a random set is to just ignore the order of items.
However, for a set, the number of possible sequences that can be regarded as equivalent to the set is exponential to the number of items.
This brings us a serious computational challenge where we need to enumerate all the equivalent sequences for a set when learning the parameters of the model.
Traditional statistical models [Benson et al., 2018, Wu et al., 2021] along this line either make unrealistic independent assumptions on items or run a learning algorithm of complexity exponential to set sizes to only deal with small sets.
These constrained treatments undermine the model’s ability to capture the complex correlations among items in sets.
To tackle the computational challenge while still well capturing the correlations among items, in this paper, we make the following technical contributions.
In Section 3, we ﬁrst show that we can convert any sequence generative model based on maximum likelihood, such as RNN [Schäfer and Zimmermann, 2006] and DeepDiffuse [Islam et al., 2018], to a set generative model.
We call our method Sequence-to-Set.
To address the computational challenge, we propose an efﬁcient importance sampling algorithm to estimate the gradient of the log-likelihood function of our Sequence-to-Set model.
In Section 4, to capture the complex item correlations, we propose two deep models GRU2Set and
SetNN, which are instances of our Sequence-to-Set model.
GRU2Set directly uses the GRU as an essential building block, while SetNN is designed for obtaining permutation invariant set embeddings when learning a set generative model.
We discuss that GRU2Set and SetNN can express any set distributions as long as we increase the model capacity.
To learn a better set distribution from data, in Section 5, we reveal that we often need to modify the empirical size distribution of the training data and propose a heuristic to do so.
This size-bias trick can be applied to all set generative models aiming at learning a set distribution.
We conduct extensive experiments using two e-commerce datasets TMALL and HKTVMALL, and report the experimental results in Section 6.
The experimental results clearly show the superiority of our models to the baselines and the effectiveness of our size-bias trick.
Related Work
Learning a set generative model is often regarded as learning a choice model for subset selection.
Along this line, Benson et al.
[2018] proposed a discrete choice model (DCM) to model the utility/probability that users select a speciﬁc set.
Wu et al.
[2021] utilized representation learning to design a parameterized Markov Random Walk to model how users select random sets from an item graph.
Both [Benson et al., 2018] and [Wu et al., 2021] are set generative models, though they are not deep models.
To achieve practical computational efﬁciency, both [Benson et al., 2018] and [Wu et al., 2021] only handle small sets and compromise on modeling high-order item correlations.
To model the permutation invariant nature of sets, Ruiz et al.
[2017] used the sum or max of item embedding as the set embedding.
Zaheer et al.
[2017] systematically investigated the theory of permutation invariant functions.
Stelzner et al.
[2020] proposed a generative model for point clouds based on GAN [Goodfellow et al., 2014] and Set Transformer [Lee et al., 2019].
Although a point cloud is a set of 2D-coordinates, we cannot use the point cloud generative model [Stelzner et al., 2020] to solve our problem as the model outputs continuous random variables instead of categorical data.
Kosiorek et al.
[2020] presented another generative model for sets of real-valued vectors based on Set Transformer [Lee et al., 2019] and their generating process is based on VAE [Kingma and
Welling, 2014].
A disadvantage of using GAN or VAE to train a generative model is that it would be difﬁcult to evaluate the probability density of a random sample.
Normalizing Flows are powerful generative models based on solid statistical theories.
However, the study of Discrete Flows is still in its infancy [Hoogeboom et al., 2019, Tran et al., 2019] and no existing works have investigated using
Discrete Flows to learn set distributions.
Multi-label classiﬁcation also needs to model the permutation invariance of sets of labels.
Yang et al. [2019] applied reinforcement learning with a permutation-invariant reward function to remove the impact of label orders.
Yazici et al.
[2020] proposed to align the labels of a sample with its predicted labels before calculating the loss of the sample.
However, our task in this paper is an unsupervised learning task.
We cannot apply the loss functions in [Yang et al., 2019, Yazici et al., 2020] since we do not have “labels” (or feature vectors if one regards a set as a set of labels) for training samples.
To sum up, existing works on set generative models either deal with sets of real-valued vectors which are different from categorical sets, or adopt non-deep models which need to compromise on capturing item correlations to make the computation efﬁcient.
It is still urgent to design deep generative models for categorical sets and propose efﬁcient algorithms to deal with computational challenges.
Sequence-to-Set Generative Models
In this section, we introduce how to convert a sequence generative model to a set generative model and how to optimize the model parameters.
3.1 Converting Sequences to Sets
We ﬁrst characterize the sequence generative model that we want to convert to a set generative model.
Deﬁnition 1 (Sequence Generative Model)
A sequence generative model is a model with a sequential generating process, which can be represented by a sequence of states and actions [ s 0 , a 0 , s 1 , a 1 , ..., s T  1 , a T  1 , s T ] , where s i is the state at time
i , a i is the action taken at time i and s T is the end state.
We call [ s 0 , a 0 , s 1 , a 1 , ..., s T  1 , a T  1 , s T ] a generating path .
Like a ﬁnite-state machine, the sequence generative model uses each action a i to deterministically update the current state s i to s i +1 .
For each state s , the sequence generative model has a corresponding probability distribution p ( a | s ; ✓ ) indicating the probability of choosing action a when the current state is s , where ✓ is the parameters of the sequence generative model.
Our deﬁnition of the sequence generative model can cover a wide range of models that can generate items in a sequential manner.
For example, the commonly used sequence model RNN is an instance, where the state is the history vector H and an action is to choose the next word/item.
Moreover, inﬂuence diffusion models such as the Independent Cascade (IC) model [Goldenberg et al., 2001] and the DeepDiffuse model [Islam et al., 2018] are also covered by Deﬁnition 1, since in these models the state at time t is the set of activated nodes until time t and the action a t is to select the group of nodes to be activated at time t + 1 .
To convert a sequence generative model in Deﬁnition 1 to a set generative model, we need to associate each state s with a state-associated set S s .
For example, in an RNN, we can deﬁne S s t as the set of items generated in the ﬁrst t steps.
In the IC model, we can deﬁne S s t as the set of nodes activated at or before time t .
We say that a generating path l
= [ s 0 , a 0 , . . .
, s T  1 , a T  1 , s T ] can induce a set S if S = S s T .
Let L ( S ) =
{ l | l can induce S } be the path set of S .
As the sequence generative model has the probability distribution p ( a | s ; ✓ ) indicating the probability of taking a as the next action given the current state s , we can derive the probability of having a path l
= [ s 0 , a 0 , . . .
, s T  1 , a T  1 , s T ] as p
[Equation]
Therefore, given a sequence generative model with parameters ✓ , the probability to induce a set S is
[Equation]
.2 Parameter Learning
Suppose we have a collection (multiset) of observed sets S = { S 1 , S 2 , ..., S N } .
We want to use S as the training data to learn a sequence model which can generate the observed sets.
We adopt maximum likelihood estimation to learn the parameters ✓ .
Speciﬁcally, the loss function is
[Equation]
A We apply stochastic gradient descent based algorithms to minimize the loss function.
The stochastic gradient w.r.t.
an observation S is r log p ( S ; ✓ ) =
r log ⇣P l 2 L ( S ) p ( l ; ✓ ) ⌘ .
As in the log-sum term of r log p ( S ; ✓ ) , the number of paths that can induce S is often exponential to | S | (in some sequence generative model the number of such paths is even inﬁnite), it is very challenging to evaluate r log p ( S ; ✓ ) exactly.
To tackle the computational challenge in computing r log p ( S ; ✓ ) , we ﬁrst ﬁnd that r log p ( S ; ✓ ) can be regarded as an expectation as follows.
[Equation]
where p ( l | S ; ✓ )
= p ( l ; ✓ ) p ( S ; ✓ ) is a posterior distribution indicating the probability that a given set S is generated by the path l .
Therefore, r log p ( S ; ✓ ) can be seen as the expected value of r p ( l ; ✓ ) where the random variable l follows the posterior distribution p ( l | S ; ✓ ) .
We can apply Monte Carlo method to estimate r log p ( S ; ✓ ) =
E l ⇠ p ( l | S ; ✓ ) [ r log p ( l ; ✓ )] , where the key point is to sample generating paths from p ( l | S ; ✓ ) .
A naive idea is to use rejection sampling, where we use the model with parameters ✓ to generate random paths and reject those that cannot induce S .
Obviously such a naive implementation would be extremely inefﬁcient as the rejection rate would be very high.
To sample l from p ( l | S ; ✓ ) more efﬁciently, we devise an important sampling method as follows.
Let ⌧ ( a, s, S ) be an indicator function, where ⌧ ( a, s, S )
=
if 9 l = [ s 0 , a 0 , . .
, s T  1 , a T  1 , s T ] such that l induces S and 9 i 2 { 0 , 1 , . .
, T  1 } , s i
= s ^ a i = a .
In our sampling algorithm, for a state s t , we only consider actions a t such that ⌧ ( a t , s, S ) = 1 .
We take such an action a t with probability
[Equation]
Therefore, the proposal distribution q ( l ; ✓ , S ) for l = [ s 0 , a 0 , . . .
, s T  1 , a T  1 , s T ] in our importance sampling is
[Equation]
The (unnormalized) importance weight of l is
[Equation]
A Clearly, we have
[Equation]
1 | s T  1 ; ✓ )
= p ( l ; ✓ ) .
This shows the correctness of the sampling method.
An EM perspective Using our importance sampling algorithm to estimate r log p ( S ; ✓ ) in minimizing the loss function L ( ✓ ) can be interpreted from an Expectation-Maximization algorithm’s perspective.
Our method is actually a Monte Carlo EM algorithm if we treat a generating path l as a latent variable.
In the E-step, we sample the latent variable l from the posterior distribution p ( l | S ; ✓ old ) .
Our importance sampling can boost the efﬁciency of the E-step.
In the M-step, we reconstruct an object function which is
P
Mi r ( l i ) R  1 log p ( l i ; ✓ old ) , where M is the number of samples and R =
P
Mi =1 r ( l i ) .
To maximize this function, we update ✓ by gradient based methods, such as ✓ new = ✓ old + ⌘
P
Mi =1 r ( l i ) R  1 r ✓ log p ( l i ; ✓ old ) , where ⌘ is the learning rate and
P
Mi =1 r ( l i ) R  1 r ✓ log p ( l i ; ✓ old ) is estimated by our importance sampling algorithm.
MRW as an instance of Sequence2Set We show that the Markov Random Walk (MRW) model
[Wu et al., 2021] is also an instance of our Sequence2Set model.
The sequence model in MRW is a random walk on the item graph [Wu et al., 2021] where a node is an item and we have a special stop item  .
The state s t at time t is associated with a set S s t which includes all the items visited until time t .
An action a t taken at time t given the state s t is to jump to an item i t from i t  1 , the item visited at time t  1 .
Once the stop item  is visited, the random walk terminates.
The probability of jumping from i t  1 to i t
= i is parameterized as p ( i t
= i | s t  1 ; ✓ ) , where ✓ is the embeddings of items.
Speciﬁcally, in MRW, p ( i t = i | s t  1 ; ✓ ) =
exp( e > i e it  1 )
P j exp( e > j e it  1 ) , where e i is the embedding vector of item i .
When an action a is jumping to an item i 6 =  , the indicator ⌧ ( a, s, S )
=
if i 2 S .
When an action a is jumping to the stop item  , ⌧ ( a, s, S )
=
if S s = S .
Wu et al.
[2021] propose an algorithm of complexity exponential to | S | to calculate the gradient r log p ( S ; ✓ ) for learning the item embeddings and thus, only small sets of size at most 4 are considered in [Wu et al., 2021].
If we adopt our important sampling algorithm, we can deal with large sets.
GRU2Set and SetNN
In this section, we propose two deep Sequence-to-Set models GRU2Set and Set Neural Networks (SetNN).
We ﬁrst introduce some general settings that will be used in both GRU2Set and SetNN.
Item embedding and stop item We also adopt item embeddings, which should be learned from the training dataset, to parameterize the probability p ( a | s ; ✓ ) in the sequence generative model.
We assign an embedding vector e i 2 R d to each item i .
Similar to MRW [Wu et al., 2021], we also add a stop item  where once we add  to the set associated to the current state, we stop the generating process.
The stop item  also has an embedding vector e  2 R d .
Sparse item graph In our Sequence-to-Set model, the indicator function ⌧ ( a, s, S ) is a key for us to efﬁciently sample generating paths from the posterior distribution p ( l | S ; ✓ ) .
Intuitively, ⌧ ( a, s, S ) indicates what actions are valid given the current state s and the induced set S .
To further boost the efﬁciency of our importance sampling algorithm, we make extra constraints on the items that can be added in the next step when we are given the current state s .
To do so, we build an item graph G = h V, E i , where V is the set of items plus the stop item  and ( i, j ) 2 E if there is a set S in the training dataset S such that i 2 S and j 2 S .
The stop item is connected to all the other items.
When using our importance sampling algorithm, given the current state s associated with a set S s ✓ S , ⌧ ( a, s, S )
= 1 if action a is adding an item i 2 S \ Neighbor ( S s ) , where Neighbor ( S s ) is the set of neighbors to items in S s .
When starting from the ﬁrst state s 0 associated with a set
S s 0 = ; , we set Neighbor ( S s 0 ) =
V \ {  } .
In real set datasets such as TMALL and HKTVMALL used in our experiments, such an item graph built from training datasets is often very sparse.
Therefore, we call the item graph the sparse item graph .
Clearly, the sparsity of the item graph can help boost the efﬁciency of generating random sets from a learned model.
Choice vector
In our model design, the conditional probability p ( a | s ; ✓ ) is the key to control the generating process.
Exploiting the idea of representation learning, we represent each state s with an embedding vector c s 2 R d .
We call c
s the choice vector of s .
Given s , we control the probability of adding i as the next item as
[Equation]
The sparsity of the item graph can make computing the normalization term in p ( i | s ; ✓ ) efﬁcient.
Note that in MRW [Wu et al., 2021], the choice vector c s is the embedding of the last item visited so far.
We will see that in GRU2Set and SetNN, we use deep neural nets to aggregate the embeddings of all the items visited so far to obtain the choice vector c s .
The initial choice vector c s 0 is trainable.
4.1 GRU2Set
We leverage GRU [Cho et al., 2014] to build our ﬁrst sequence-to-set model, since GRU is one of the most popular sequential model based on RNN and its computational cost is low.
We set the history vector H of GRU as a d -dimensional vector and H is regarded as the choice vector of the current state.
The initial history vector H 0 is trainable and H is updated according to the design of GRU.
The state-associated set S s t for the state s t at time t is just the set of items added in the ﬁrst t steps of the sequence generating process of GRU.
When the stop item is added, we terminate the generating process.
Due to limited space, the overview of GRU2Set is put in Appendix.
4.2 SetNN
The choice vector c s can be regarded as the embedding of the set S s .
In GRU2Set, the choice vector, which is the history vector H , is sensitive to the order of items added to S s .
As sets are permutation invariant, we hope to have a sequence-to-set model which can also produce permutation invariant representations of sets.
Therefore, we develop the Set Neural Networks (SetNN) where the key idea is to design a permutation invariant aggregation of embeddings of items in a set.
The following theorem by Zaheer et al.
[2017] guides our model design.
Theorem 1 ([Zaheer et al., 2017])
A function f ( X ) operating on a set X having elements from a countable universe, is a set function, i.e., invariant to the permutation of instances in X , iff it can be decomposed in the form ⇢ (
P x 2 X  ( x )) for suitable transformations  and ⇢ .
( x ) can be viewed as the embedding of x and the sum over  ( x ) can be replaced by other aggregate operators such as mean or pooling.
In SetNN, we use an MLP to express the function ⇢ in Theoremto obtain the embedding (choice vector)
c s of S s for any state s .
Fig 2 shows the overview of SetNN.
4.3 Expressive Power of GRU2Set and SetNN
Compared to SetNN, our GRU2Set model uses the history vector H .
By the expressive power of
RNN [Schäfer and Zimmermann, 2006], as we increase the embedding dimension d , hidden neural units and layers in GRU module, the history vector H of GRU2Set can approximate any function on a ﬁnite history path arbitrarily close.
Then, of course, H can express any permutation invariant function on sets which implies that the expressive power of GRU2Set is more than SetNN.
To explore the expressive power of SetNN, we ﬁrst give a recursive deﬁnition of the probability that a set S is generated by SetNN, denoted by p ( S ) , as follows.
[Equation]
where p ( x | S \ { x } ) is the conditional probability of adding item x to the set S \ { x } and  ( S ) is the probability of reaching a state s such that S s = S in SetNN.
The following theorem shows that
Eq.
(1) is general enough to cover all possible set distributions.
Theorem 2 For any distribution q ( S ) on all subsets of a ground set of items, there exists a group of transition probabilities p ( x | S ) and p (  | S ) , such that q ( S ) =
p ( S ) holds for any S , where p ( S ) is deﬁned in Eq.
The proof of Theorem 2 can be found in Appendix B. As we increase the capacity of the MLP in SetNN, SetNN can represent any conditional probability p ( x | S ) and p (  | S ) , and as a result of
Theorem 2, SetNN and GRU2Set can express any set distributions.
Size-Bias Trick for Improving Learned Set Distributions
In this section, we propose a size-bias trick that can help reduce the distance between the ground truth set distribution p
⇤ ( S ) and the learned distribution p ( S ) , where p ( S ) can be learned by any generative model not limited to our Sequence-to-Set models.
We ﬁrst decompose the ground truth distribution p ⇤ ( S ) when | S | =
k as
[Equation]
where p ⇤ k
=
P
A : | A | = k p ( A ) is the probability of generating a set of size k , and p ⇤ ( k ) ( S )
=
p ( S ) p
⇤ k is the probability that a random size- k set is S .
Suppose we know the real size distribution p ⇤ k and we have q ( k ) ( S ) as an estimation of p ⇤ ( k ) ( S ) for each k .
How can we construct a distribution q ( S ) close to p
⇤ ( S ) ?
An intuitive way is to combine q ( k ) ( S ) and p ⇤ k
such that q ( S ) =
q ( k ) ( S ) p
⇤ k
if | S | =
k
[Stelzner et al., 2020, Benson et al., 2018].
However, we argue that this intuitive way is not always the optimal plan as follows.
Let K be the largest possible set size.
To ﬁnd the best size distribution q k to collaborate with our estimation q ( k ) ( S ) , we consider the following optimization aiming at minimizing the KL-divergence between the distribution q ( S ) =
q ( k ) ( S ) q k and the ground truth p
⇤ ( S ) .
[Equation]
Since we do not know p ⇤ ( S ) , we cannot solve Eq.
(2) exactly.
However, we still can tell if setting q k = p
⇤ k is the optimal solution to some extend.
Let f k ( q k ) =
P
S : | S | = k q ( S ) log q ( S ) p
⇤ ( S ) .
We have KL ( q || p ⇤ ) =
P k f k ( q k ) .
Note that f k ( q k ) only depends on q k .
We can interpret Eq.
(2) as a portfolio optimization problem.
There are K projects. q k is the investment for project k and  f k ( q k ) is the revenue of project k .
To decide the best portfolio q k , we check the derivative of f k ( q k ) , which indicates the marginal revenue.
We have f 0 k ( q k ) = 1 + X
[Equation]
[Table]
[Table]
By taking q k = p
⇤ k , we have
[Equation]
⌘ As the estimation quality of q ( k ) ( S ) often varies for different k , the derivative f 0 k ( p ⇤ k ) probably also varies for different k , making q k = p
⇤ k not a stationary point.
Therefore, we probably need to set q k different from p ⇤ k for solving Eq.
We ﬁrst make the following reasonable assumption.
Assumption 1 Small-sized sets are easier to learn than large sets.
In other words, q ( k ) ( S ) estimates p ⇤ ( k ) ( S ) better as the set size k decreases.
Based on Assumption 1, compared to p ⇤ k , we should “invest” more on small set sizes to construct the “portfolio” (size distribution) q k .
Therefore, we set q k slightly bigger than p ⇤ k for small k , and set q k smaller than p ⇤ k for big k .
To address the issue of not knowing p ⇤ k , we can resort to the empirical size distribution of the training dataset, which is a good estimation of the real size distribution p ⇤ k .
We construct a biased size distribution q k by adopting the following heuristic.
Constructing Biased Size Distribution Suppose the empirical size distribution of the training data
S is [ p 1 , p 2 , ..., p K ] .
We ﬁrst calculate the rest proportion by r k = p k / ( p k + p k +
+ ... + p K ) , which means p i =
(1  r 1 ) ...
(1  r i  1 ) r i .
Then we calculate r 0 k = max { r k + k p | V | / |S| , 1 } , where V is the set of all items.
We construct a biased size distribution [ q 1 , q 2 , . . .
, q K ] by setting q i = (1  r 0 1 ) ...
(1  r 0 i  1 ) r 0 i .
Experiments
Datasets We did our empirical study on two real-world datasets of customer orders from online e-commerce platforms.
This ﬁrst dataset is TMALL ( https://www.tmall.com ) which contains 1363 items and orders from August 2018 to March 2019.
We treated the collection of orders in each month as an instance and in total we have 8 instances for the TMALL dataset.
The second dataset is HKTVMALL ( https://www.hktvmall.com ) which contains 728 items form the supermarket sector and orders from February 2020 to September 2020.
Similar to TMALL, we also treated HKTVMALL as 8 instances where each instance is the collection of orders in each month.
For both datasets, We split all the orders in a month to a training dataset S train and a testing dataset S test , where the size of S train is 100,000 for Tmall and 200,000 for HKTVmall.
The datasets can be found in the source code.
We report the sparsity of item graphs and statistics of the datasets in Table 1.
The set size distribution of each dataset is shown in Table 2.
Experiment Setting and Evaluation We treat the method of directly using the histogram of S train as the benchmark method, as it is a natural and unbiased method to estimate the set distribution.
Note that this benchmark method has a major disadvantage that its cannot model probabilities of any sets not showing in S train .
We also compare our models with two baselines which are DCM (Discrete Choice Model)
[Benson et al., 2018] and MRW [Wu et al., 2021].
Following [Wu et al., 2021], for each method (except the benchmark), we generated a collection S pred of 10,000,000 random sets and used the empirical set distribution of S pred to approximate the set distribution learned by the model.
We regarded the empirical distribution of the testing data S test as the pseudo ground truth.
Since the bigger S test is, the more accurate the pseudo ground truth is, we set the size of S test bigger than S train as shown in Table 1.
Denote by N test ( S ) the number of sets that are S in S test .
Let N pred ( S ) be the number of sets that are S in S pred .
We used the ` 1 -distance between a model’s (approximate) distribution and the pseudo ground truth to evaluate the effectiveness of the model.
Speciﬁcally, the ` 1 -distance can be calculated as follows.
[Equation]
We set the embedding dimension of all methods as 10.
For each training and testing dataset, the biased size distribution was the same for all methods and was obtained by the heuristic introduced at the end of Section 5.
For all experiments, the MLP of SetNN only has one hidden layer which contains 50 neural units.
All the experiments were ran on a CPU with 10 cores.
The optimizer used by us is RMSProp with default parameter in PyTorch.
The source code and data used in our experiments can be found at https://github.com/LongtaoTang/SetLearning . [Table] [Table]
Experiment Results Due to limited space, we only report the main experimental results.
More detailed experimental results can be found in Appendix D and E.
Table 3 and Table 4 show the experimental results.
We place the results of using the size-bias trick below the original results.
We ﬁnd that both DCM and MRW never beat the benchmark method, no matter the size-bias trick is played or not.
A possible reason is that in our experiments we did not make size constraints on sets, while Wu et al.
[2021] constrained the set size to be at most 4 and Benson et al.
[2018] set the maximum size of sets to be 5.
Big sets in our data may have negative effects on learning both DCM and MRW.
We can see that applying the size-bias trick can always reduce the ` 1 -distance signiﬁcantly, which demonstrates the effectiveness of this trick.
Before applying the size-bias trick, the benchmark method has the best performance which is slightly better than our models GRU2Set and SetNN.
However, after applying the size-bias trick, our GRU2Set model becomes the best and it outperforms the benchmark method by 4% on TMALL and 2.5% on HKTVMALL.
SetNN also outperforms the benchmark method after using the size-bias trick.
This suggests that GRU2Set and SetNN may learn probabilities of small sets better than the benchmark method.
The SetNN model performs slightly worse than GRU2Set
but the gap between them is small.
A possible reason is that GRU2Set has stronger expressive power than SetNN as illustrated in Section 4.3.
GRU2Set records the order of items added to the set, while SetNN ignores such order to achieve permutation invariant set embeddings.
This shows that GRU2Set includes more information in the generating process than SetNN.
However, we want to emphasize that SetNN can produce permutation invariant set embeddings while GRU2Set cannot.
If we have a downstream task that needs set embeddings, SetNN may have more advantages in this case.
Conclusion
In this paper, we present a Sequence-to-Set method to build generative models for set data and the SOTA method MRW is an instance of our method.
To utilize deep learning in learning set distributions, we further design two models, GRU2Set and SetNN, which are two instances of our
Sequence-to-Set model.
To learn better set distributions from data, we also propose a size-bias trick.
Experimental results on two e-commerce order datasets clearly show that our models outperform the baselines including two SOTA methods in this line of research.
For future work, we will explore using more sophisticated deep learning modules in our model, such as replacing the MLP in SetNN with Set Transformer and substituting the RNN in GRU with more advanced sequence models.
For potential negative societal impacts, learning the embeddings of items and sets might cause user privacy leakage as many other embeddings might do.
Research on how to hide personal information on set data could help protect users’ privacy.
Acknowledgments and Disclosure of Funding
Tang and Yang’s research is supported in part by the Hong Kong Research Grants Council under
ECS grant 21214720, City University of Hong Kong under Project 9610465, and Alibaba Group through Alibaba Innovative Research (AIR) Program.
Zhou’s research is supported in part by City
University of Hong Kong under Project 7200694.
The authors thank the HKTVmall Open Databank for providing the HKTVMALL dataset.
All opinions, ﬁndings, conclusions, and recommendations in this paper are those of the authors and do not necessarily reﬂect the views of the funding agencies.